--- title: "DMR++ as a virtual chunk manifest" description: "Parse DMR++ references, map them to Zarr-style chunk manifests, and see what the sidecar does not prove about the files it points to." date: 2026-11-12 author: "Luis López" categories: ["earth-observation", "data-systems"] tags: ["dmrpp", "zarr", "icechunk", "virtualizarr", "metadata", "notebook"] notebook_source: "notebooks-src/virtualization/dmrpp-to-zarr.qmd" format: hugo-md math: true jupyter: python3 --- DMR++ is a metadata sidecar used by OPeNDAP/Hyrax. For an HDF5 granule, it describes the dataset structure and, importantly, the byte ranges of each chunk inside the original file. A client can therefore build a virtual view of the dataset without parsing the granule from scratch. This notebook makes the mapping explicit with a small synthetic DMR++ document. It then builds a VirtualiZarr `ChunkManifest` from the same references and compares the result with the shape of an Icechunk-style manifest. ## A minimal DMR++ document ```{python} DMRPP_XML = """\ degC 1 3 """ ``` The document names the source file and gives each chunk a position in the array, a byte offset, and a byte length. That is already almost a chunk manifest. ## Parse the references ```{python} import xml.etree.ElementTree as ET DAP = "{http://xml.opendap.org/ns/DAP/4.0#}" DMRPP = "{http://xml.opendap.org/dap/dmrpp/1.0.0#}" root = ET.fromstring(DMRPP_XML) source = root.get("name") dimensions = { element.get("name"): int(element.get("size")) for element in root.findall(f"{DAP}Dimension") } variable = root.find(f"{DAP}Float32") chunks = variable.find(f"{DMRPP}chunks") chunk_shape = tuple( int(value) for value in chunks.findtext(f"{DMRPP}chunkDimensionSizes").split() ) references = [] for chunk in chunks.findall(f"{DMRPP}chunk"): position = tuple( int(value) for value in chunk.get("chunkPositionInArray").strip("[]").split(",") ) key = ".".join( str(position[index] // chunk_shape[index]) for index in range(len(chunk_shape)) ) references.append( { "key": key, "path": source, "offset": int(chunk.get("offset")), "length": int(chunk.get("nBytes")), } ) print("dimensions:", dimensions) print("chunk shape:", chunk_shape) print("references:", references) ``` ## A table of chunk references ```{python} #| echo: false from IPython.display import Markdown, display def markdown_table(rows): columns = list(rows[0]) lines = [ "| " + " | ".join(columns) + " |", "| " + " | ".join(["---"] * len(columns)) + " |", ] lines.extend( "| " + " | ".join(str(row[column]) for column in columns) + " |" for row in rows ) return "\n".join(lines) display(Markdown(markdown_table(references))) ``` ## Build a VirtualiZarr chunk manifest VirtualiZarr uses the same model: array chunk coordinates point at byte ranges in an object store. The manifest does not copy the data; it delegates reads to the original object. ```{python} from virtualizarr.manifests import ChunkManifest grid_shape = ( dimensions["lat"] // chunk_shape[0], dimensions["lon"] // chunk_shape[1], ) manifest = ChunkManifest( entries={ reference["key"]: { "path": reference["path"], "offset": reference["offset"], "length": reference["length"], } for reference in references }, shape=grid_shape, ) manifest ``` The entries are exactly the offsets and lengths from the DMR++ document: ```{python} manifest.dict() ``` ## The Icechunk comparison An Icechunk manifest is also a mapping from chunk coordinates to storage references, but it is part of a versioned store snapshot rather than a static file beside the granule. Conceptually, the same two references look like this: ```{python} icechunk_like = { "array": variable.get("name"), "array_shape": tuple(dimensions.values()), "chunk_shape": chunk_shape, "snapshot": "example", "manifests": [ { "chunk_key": reference["key"], "object": reference["path"], "byte_range": [reference["offset"], reference["length"]], "checksum": None, } for reference in references ], } import json print(json.dumps(icechunk_like, indent=2)) ``` The semantic equivalence is real: both are chunk manifests over byte ranges. The difference is governance. DMR++ is a sidecar generated for a file. An Icechunk manifest belongs to a snapshot with transactional consistency and store-level versioning. ## What DMR++ does not prove The reference above contains no digest binding it to the current contents of `granule.h5`. Offsets and lengths say where the reader should look, not that the bytes at that location are still the bytes the sidecar was built from. ```{python} assert all("checksum" not in reference for reference in references) assert sum(reference["length"] for reference in references) == 24 print("references contain offsets and lengths, but no content hash") ``` If a granule is regenerated, replaced, or rewritten in place, a stale DMR++ file can still point into a perfectly valid object and return the wrong chunk. The reader cannot verify that the reference is up to date before reading. Production systems need one of: - object versioning or immutable object names; - a checksum/ETag binding at the storage layer; - regeneration of the sidecar whenever the object changes; or - a transactional store that records both the reference and the object version. DMR++ speeds up discovery and access. It is not a content-addressed integrity mechanism. ## Where this comes from The idea of exposing a scientific file's internal structure so that it can be read and understood later is old. The [HDF4MapReader](https://hdf4mapreader.sourceforge.net/index.html) project was an early effort to surface internal metadata from scientific files in a form that humans could inspect years after the original tooling disappeared. Modern DMR++ sidecars and virtual chunk manifests are operational descendants of that idea: make the internal map explicit, portable, and useful to software other than the original reader.