AVHRR NDVI CDR Icechunk Demo

This dataset is a small demonstration Icechunk repository built from NOAA Climate Data Record (CDR) AVHRR NDVI NetCDF files. The Icechunk repository stores metadata and virtual chunk references, while the original NetCDF data chunks remain in the public NOAA S3 bucket.

This example is intended for teaching cloud-native access patterns for archival NetCDF data using:

• Icechunk
• VirtualiZarr
• Xarray
• Public object storage

This example is based on a notebook by Rich Signell and modified by Eli Holmes. Notebook used to create this demo virtualizarr_ndvi_cdr_append-cloud.

Dataset summary

What is in this repository?

This repository contains an Icechunk dataset. The Icechunk repository does not copy the full NDVI data out of the original NetCDF files. Instead, it stores virtual references that point back to byte ranges in the original public NOAA S3 files.

That means:

• The Icechunk repo is small.
• The original data remain in NOAA's public bucket.
• Users can open the dataset with Xarray as if it were a Zarr-like dataset.
• Reading actual NDVI values will fetch the needed byte ranges from the original NetCDF files.

Source data

The source NetCDF files are from the public NOAA CDR NDVI bucket:

1s3://noaa-cdr-ndvi-pds/
1s3://noaa-cdr-ndvi-pds/

Example source file:

1s3://noaa-cdr-ndvi-pds/data/2000/AVHRR-Land_v005_AVH13C1_NOAA-14_20000101_c20170623095628.nc
1s3://noaa-cdr-ndvi-pds/data/2000/AVHRR-Land_v005_AVH13C1_NOAA-14_20000101_c20170623095628.nc

The Icechunk repo contains virtual chunk references back to these source NetCDF files.

Open the dataset in Python

Replace the bucket and prefix below with the Source Cooperative bucket and prefix for this dataset.

1import icechunk
2import xarray as xr
3
4# -------------------------------------------------------------------
5# 1. Authorize access to the original NOAA NetCDF chunks
6# -------------------------------------------------------------------
7# The Icechunk repo contains virtual references back to the public
8# NOAA CDR NDVI S3 bucket. Use anonymous access for these chunks.
9credentials = icechunk.containers_credentials({
10    "s3://noaa-cdr-ndvi-pds/": icechunk.s3_credentials(anonymous=True)
11})
12
13# -------------------------------------------------------------------
14# 2. Tell Icechunk which external virtual chunk locations are allowed
15# -------------------------------------------------------------------
16config = icechunk.RepositoryConfig.default()
17config.set_virtual_chunk_container(
18    icechunk.VirtualChunkContainer(
19        url_prefix="s3://noaa-cdr-ndvi-pds/",
20        store=icechunk.s3_store(region="us-east-1", anonymous=True),
21    ),
22)
23
24# -------------------------------------------------------------------
25# 3. Point to the public Icechunk repo on Source Cooperative
26# -------------------------------------------------------------------
27storage = icechunk.s3_storage(
28    bucket="us-west-2.opendata.source.coop",
29    prefix="eeholmes/chlaz",
30    region="us-west-2",
31    anonymous=True,
32)
33
34# -------------------------------------------------------------------
35# 4. Open the Icechunk repo and read it with xarray
36# -------------------------------------------------------------------
37repo = icechunk.Repository.open(
38    storage,
39    config=config,
40    authorize_virtual_chunk_access=credentials,
41)
42
43session = repo.readonly_session(branch="main")
44
45ds = xr.open_zarr(
46    session.store,
47    consolidated=False,
48    chunks=None,
49)
50
51ds
1import icechunk
2import xarray as xr
3
4# -------------------------------------------------------------------
5# 1. Authorize access to the original NOAA NetCDF chunks
6# -------------------------------------------------------------------
7# The Icechunk repo contains virtual references back to the public
8# NOAA CDR NDVI S3 bucket. Use anonymous access for these chunks.
9credentials = icechunk.containers_credentials({
10    "s3://noaa-cdr-ndvi-pds/": icechunk.s3_credentials(anonymous=True)
11})
12
13# -------------------------------------------------------------------
14# 2. Tell Icechunk which external virtual chunk locations are allowed
15# -------------------------------------------------------------------
16config = icechunk.RepositoryConfig.default()
17config.set_virtual_chunk_container(
18    icechunk.VirtualChunkContainer(
19        url_prefix="s3://noaa-cdr-ndvi-pds/",
20        store=icechunk.s3_store(region="us-east-1", anonymous=True),
21    ),
22)
23
24# -------------------------------------------------------------------
25# 3. Point to the public Icechunk repo on Source Cooperative
26# -------------------------------------------------------------------
27storage = icechunk.s3_storage(
28    bucket="us-west-2.opendata.source.coop",
29    prefix="eeholmes/chlaz",
30    region="us-west-2",
31    anonymous=True,
32)
33
34# -------------------------------------------------------------------
35# 4. Open the Icechunk repo and read it with xarray
36# -------------------------------------------------------------------
37repo = icechunk.Repository.open(
38    storage,
39    config=config,
40    authorize_virtual_chunk_access=credentials,
41)
42
43session = repo.readonly_session(branch="main")
44
45ds = xr.open_zarr(
46    session.store,
47    consolidated=False,
48    chunks=None,
49)
50
51ds

Plot NDVI

An interactive hvPlot version:

1import hvplot.xarray
2
3ds["NDVI"].isel(time=0).hvplot.quadmesh(
4    rasterize=True,
5    x="longitude",
6    y="latitude",
7    cmap="turbo_r",
8    clim=(-0.2, 1.0),
9    title="AVHRR NDVI — 2000-01-01",
10    width=800,
11    height=400,
12    xlim=(-180, 180),
13    ylim=(-90, 90),
14)
1import hvplot.xarray
2
3ds["NDVI"].isel(time=0).hvplot.quadmesh(
4    rasterize=True,
5    x="longitude",
6    y="latitude",
7    cmap="turbo_r",
8    clim=(-0.2, 1.0),
9    title="AVHRR NDVI — 2000-01-01",
10    width=800,
11    height=400,
12    xlim=(-180, 180),
13    ylim=(-90, 90),
14)

Citation and attribution

Please cite the original NOAA CDR NDVI product when using the data scientifically. This Icechunk repository is a derived access layer for demonstration and teaching purposes; the underlying data are from NOAA.

License

See the license and use constraints for the original NOAA CDR NDVI product. This repository provides virtual access to that public source data and does not modify the original NetCDF files.