DMR++ as a virtual chunk manifest

DMR++ is a metadata sidecar used by OPeNDAP/Hyrax. For an HDF5 granule, it describes the dataset structure and, importantly, the byte ranges of each chunk inside the original file. A client can therefore build a virtual view of the dataset without parsing the granule from scratch.

This notebook makes the mapping explicit with a small synthetic DMR++ document. It then builds a VirtualiZarr ChunkManifest from the same references and compares the result with the shape of an Icechunk-style manifest.

A minimal DMR++ document

DMRPP_XML = """\
<Dataset xmlns="http://xml.opendap.org/ns/DAP/4.0#"
         xmlns:dmrpp="http://xml.opendap.org/dap/dmrpp/1.0.0#"
         name="/data/granule.h5" dapVersion="4.0" dmrVersion="1.0">
  <Dimension name="lat" size="2"/>
  <Dimension name="lon" size="3"/>
  <Float32 name="sst">
    <Dim name="/lat"/>
    <Dim name="/lon"/>
    <Attribute name="units" type="String"><Value>degC</Value></Attribute>
    <dmrpp:chunks fillValue="-999" byteOrder="LE">
      <dmrpp:chunkDimensionSizes>1 3</dmrpp:chunkDimensionSizes>
      <dmrpp:chunk chunkPositionInArray="[0,0]" offset="4096" nBytes="12"/>
      <dmrpp:chunk chunkPositionInArray="[1,0]" offset="8192" nBytes="12"/>
    </dmrpp:chunks>
  </Float32>
</Dataset>
"""

The document names the source file and gives each chunk a position in the array, a byte offset, and a byte length. That is already almost a chunk manifest.

Parse the references

import xml.etree.ElementTree as ET

DAP = "{http://xml.opendap.org/ns/DAP/4.0#}"
DMRPP = "{http://xml.opendap.org/dap/dmrpp/1.0.0#}"

root = ET.fromstring(DMRPP_XML)
source = root.get("name")
dimensions = {
    element.get("name"): int(element.get("size"))
    for element in root.findall(f"{DAP}Dimension")
}
variable = root.find(f"{DAP}Float32")
chunks = variable.find(f"{DMRPP}chunks")
chunk_shape = tuple(
    int(value)
    for value in chunks.findtext(f"{DMRPP}chunkDimensionSizes").split()
)

references = []
for chunk in chunks.findall(f"{DMRPP}chunk"):
    position = tuple(
        int(value)
        for value in chunk.get("chunkPositionInArray").strip("[]").split(",")
    )
    key = ".".join(
        str(position[index] // chunk_shape[index])
        for index in range(len(chunk_shape))
    )
    references.append(
        {
            "key": key,
            "path": source,
            "offset": int(chunk.get("offset")),
            "length": int(chunk.get("nBytes")),
        }
    )

print("dimensions:", dimensions)
print("chunk shape:", chunk_shape)
print("references:", references)

dimensions: {'lat': 2, 'lon': 3}
chunk shape: (1, 3)
references: [{'key': '0.0', 'path': '/data/granule.h5', 'offset': 4096, 'length': 12}, {'key': '1.0', 'path': '/data/granule.h5', 'offset': 8192, 'length': 12}]

A table of chunk references

Build a VirtualiZarr chunk manifest

VirtualiZarr uses the same model: array chunk coordinates point at byte ranges in an object store. The manifest does not copy the data; it delegates reads to the original object.

from virtualizarr.manifests import ChunkManifest

grid_shape = (
    dimensions["lat"] // chunk_shape[0],
    dimensions["lon"] // chunk_shape[1],
)
manifest = ChunkManifest(
    entries={
        reference["key"]: {
            "path": reference["path"],
            "offset": reference["offset"],
            "length": reference["length"],
        }
        for reference in references
    },
    shape=grid_shape,
)

manifest

ChunkManifest<shape=(2, 1)>

The entries are exactly the offsets and lengths from the DMR++ document:

manifest.dict()

{'0.0': {'path': 'file:///data/granule.h5', 'offset': 4096, 'length': 12},
 '1.0': {'path': 'file:///data/granule.h5', 'offset': 8192, 'length': 12}}

The Icechunk comparison

An Icechunk manifest is also a mapping from chunk coordinates to storage references, but it is part of a versioned store snapshot rather than a static file beside the granule. Conceptually, the same two references look like this:

icechunk_like = {
    "array": variable.get("name"),
    "array_shape": tuple(dimensions.values()),
    "chunk_shape": chunk_shape,
    "snapshot": "example",
    "manifests": [
        {
            "chunk_key": reference["key"],
            "object": reference["path"],
            "byte_range": [reference["offset"], reference["length"]],
            "checksum": None,
        }
        for reference in references
    ],
}

import json

print(json.dumps(icechunk_like, indent=2))

{
  "array": "sst",
  "array_shape": [
    2,
    3
  ],
  "chunk_shape": [
    1,
    3
  ],
  "snapshot": "example",
  "manifests": [
    {
      "chunk_key": "0.0",
      "object": "/data/granule.h5",
      "byte_range": [
        4096,
        12
      ],
      "checksum": null
    },
    {
      "chunk_key": "1.0",
      "object": "/data/granule.h5",
      "byte_range": [
        8192,
        12
      ],
      "checksum": null
    }
  ]
}

The semantic equivalence is real: both are chunk manifests over byte ranges. The difference is governance. DMR++ is a sidecar generated for a file. An Icechunk manifest belongs to a snapshot with transactional consistency and store-level versioning.

What DMR++ does not prove

The reference above contains no digest binding it to the current contents of granule.h5. Offsets and lengths say where the reader should look, not that the bytes at that location are still the bytes the sidecar was built from.

assert all("checksum" not in reference for reference in references)
assert sum(reference["length"] for reference in references) == 24
print("references contain offsets and lengths, but no content hash")

references contain offsets and lengths, but no content hash

If a granule is regenerated, replaced, or rewritten in place, a stale DMR++ file can still point into a perfectly valid object and return the wrong chunk. The reader cannot verify that the reference is up to date before reading. Production systems need one of:

• object versioning or immutable object names;
• a checksum/ETag binding at the storage layer;
• regeneration of the sidecar whenever the object changes; or
• a transactional store that records both the reference and the object version.

DMR++ speeds up discovery and access. It is not a content-addressed integrity mechanism.

Where this comes from

The idea of exposing a scientific file’s internal structure so that it can be read and understood later is old. The HDF4MapReader project was an early effort to surface internal metadata from scientific files in a form that humans could inspect years after the original tooling disappeared. Modern DMR++ sidecars and virtual chunk manifests are operational descendants of that idea: make the internal map explicit, portable, and useful to software other than the original reader.