https://betolink.github.io/blog/tags/data-systems/Recent content in Data-Systems on betolinkHugoenThu, 05 Nov 2026 09:15:00 -0600https://betolink.github.io/blog/posts/earth-observation-is-software/Thu, 05 Nov 2026 09:15:00 -0600https://betolink.github.io/blog/posts/earth-observation-is-software/<p>Earth observation archives are often described as data problems. In practice, most of the friction is software: metadata that does not agree with itself, formats that require whole-file downloads, and access patterns that assume a local filesystem.</p> <h2 id="what-cloud-native-changes">What cloud-native changes</h2> <p>Chunked, self-describing arrays make partial reads practical. A workflow can open a dataset, inspect coordinates, and request one time slice without downloading the archive.</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#282a36;-moz-tab-size:2;-o-tab-size:2;tab-size:2;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#ff79c6">import</span> xarray <span style="color:#ff79c6">as</span> xr </span></span><span style="display:flex;"><span> </span></span><span style="display:flex;"><span>ds <span style="color:#ff79c6">=</span> xr<span style="color:#ff79c6">.</span>open_zarr( </span></span><span style="display:flex;"><span> <span style="color:#f1fa8c">&#34;https://example.com/store.zarr&#34;</span>, </span></span><span style="display:flex;"><span> consolidated<span style="color:#ff79c6">=</span><span style="color:#ff79c6">True</span>, </span></span><span style="display:flex;"><span> chunks<span style="color:#ff79c6">=</span>{}, </span></span><span style="display:flex;"><span>) </span></span><span style="display:flex;"><span><span style="color:#8be9fd;font-style:italic">print</span>(ds) </span></span></code></pre></div><p>The hard part is not the API. The hard part is consistency across collections, clear provenance, and clients that fail gracefully.</p>