README

walkthru-earth / indices

Name: Walkthru-Indices
Creator: walkthru.earth

Open planetary indices that turn raw scientific data into a single, H3-hexagonal grid anyone can query with one SQL statement. Every index shares the same cell ID (h3_index), so joining terrain + buildings + population + weather is a one-line JOIN ON h3_index.

Why? Cities shape how people feel — but the data to prove it is scattered across dozens of agencies, formats, and projections. We unify it into one queryable, open layer so researchers, planners, and communities can finally see the full picture. Read more: walkthru.earth/hormones-cities

1flowchart LR
2    subgraph Raw["Raw sources"]
3        DEM_RAW["GEDTM-30m\n(30 m GeoTIFF)"]
4        GBA_RAW["Global Building Atlas\n(2.75 B polygons)"]
5        WP_RAW["WorldPop SSP\n(1 km rasters)"]
6        NOAA_RAW["NOAA AI-NWP\n(0.25° NetCDF)"]
7    end
8
9    subgraph H3["H3 conversion"]
10        direction TB
11        DT["dem-terrain\nelev · slope · aspect · TRI · TPI"]
12        BI["walkthru-building-index\ndensity · height · volume"]
13        PI["walkthru-pop-index\n2025 → 2100 projections"]
14        WI["walkthru-weather-index\ntemp · wind · precip · pressure"]
15    end
16
17    DEM_RAW --> DT
18    GBA_RAW --> BI
19    WP_RAW --> PI
20    NOAA_RAW --> WI
21    DT -. topo corrections .-> WI
22
23    subgraph S3["s3://…/walkthru-earth/"]
24        T[dem-terrain/v2/h3/]
25        B[indices/building/v2/h3/]
26        P[indices/population/v2/]
27        W[indices/weather/]
28    end
29
30    DT --> T
31    BI --> B
32    PI --> P
33    WI --> W
34
35    T -. JOIN h3_index .-> Q((Query))
36    B -. JOIN .-> Q
37    P -. JOIN .-> Q
38    W -. JOIN .-> Q
1flowchart LR
2    subgraph Raw["Raw sources"]
3        DEM_RAW["GEDTM-30m\n(30 m GeoTIFF)"]
4        GBA_RAW["Global Building Atlas\n(2.75 B polygons)"]
5        WP_RAW["WorldPop SSP\n(1 km rasters)"]
6        NOAA_RAW["NOAA AI-NWP\n(0.25° NetCDF)"]
7    end
8
9    subgraph H3["H3 conversion"]
10        direction TB
11        DT["dem-terrain\nelev · slope · aspect · TRI · TPI"]
12        BI["walkthru-building-index\ndensity · height · volume"]
13        PI["walkthru-pop-index\n2025 → 2100 projections"]
14        WI["walkthru-weather-index\ntemp · wind · precip · pressure"]
15    end
16
17    DEM_RAW --> DT
18    GBA_RAW --> BI
19    WP_RAW --> PI
20    NOAA_RAW --> WI
21    DT -. topo corrections .-> WI
22
23    subgraph S3["s3://…/walkthru-earth/"]
24        T[dem-terrain/v2/h3/]
25        B[indices/building/v2/h3/]
26        P[indices/population/v2/]
27        W[indices/weather/]
28    end
29
30    DT --> T
31    BI --> B
32    PI --> P
33    WI --> W
34
35    T -. JOIN h3_index .-> Q((Query))
36    B -. JOIN .-> Q
37    P -. JOIN .-> Q
38    W -. JOIN .-> Q

Indices

Index	What	H3 Res	Update	Size
dem-terrain	Elevation, slope, aspect, TRI, TPI from GEDTM-30m	1–10	One-time	~287 GB, 10.5 B cells
walkthru-building-index	Density, height, footprint, volume from 2.75 B buildings	3–8	One-time	~2.6 GB, 53 M cells
walkthru-pop-index	SSP2 projections 2025–2100 (16 timesteps) from WorldPop	1–8	One-time	~4.5 GB, 44 M cells
walkthru-weather-index	Temperature, wind, precip, pressure + topo corrections	0–5	Every 12 h	~3.6 GB / forecast

All datasets: Apache Parquet, ZSTD compression, sorted by h3_index.

v2 (recommended): h3_index is BIGINT (int64) — no geometry, lat, lon, or area_km2 columns (all derivable from the H3 index via DuckDB's h3 extension). Sorted by int64 for better compression and fast range-based spatial queries.
v1 (legacy): h3_index is VARCHAR (hex string) with native Parquet 2.11+ GEOMETRY and lat/lon/area_km2 columns.

Quick start

1-- DuckDB: join buildings + population in one query (v2 — BIGINT h3_index, no geometry)
2INSTALL httpfs;  LOAD httpfs;
3INSTALL h3 FROM community; LOAD h3;
4SET s3_region = 'us-west-2';
5
6SELECT b.h3_index,
7       h3_h3_to_string(b.h3_index) AS h3_hex,          -- hex string for deck.gl
8       h3_cell_to_lat(b.h3_index)  AS lat,              -- derive lat on the fly
9       h3_cell_to_lng(b.h3_index)  AS lon,              -- derive lon on the fly
10       b.building_count, b.building_density,
11       p.pop_2025, p.pop_2050,
12       (b.building_count::FLOAT / NULLIF(p.pop_2025, 0))::DECIMAL(6,3) AS bldg_per_person
13FROM read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/indices/building/v2/h3/h3_res=5/data.parquet') b
14JOIN read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/indices/population/v2/scenario=SSP2/h3_res=5/data.parquet') p
15  ON b.h3_index = p.h3_index                            -- BIGINT join — fast
16WHERE p.pop_2025 > 10000
17ORDER BY bldg_per_person DESC
18LIMIT 10;
1-- DuckDB: join buildings + population in one query (v2 — BIGINT h3_index, no geometry)
2INSTALL httpfs;  LOAD httpfs;
3INSTALL h3 FROM community; LOAD h3;
4SET s3_region = 'us-west-2';
5
6SELECT b.h3_index,
7       h3_h3_to_string(b.h3_index) AS h3_hex,          -- hex string for deck.gl
8       h3_cell_to_lat(b.h3_index)  AS lat,              -- derive lat on the fly
9       h3_cell_to_lng(b.h3_index)  AS lon,              -- derive lon on the fly
10       b.building_count, b.building_density,
11       p.pop_2025, p.pop_2050,
12       (b.building_count::FLOAT / NULLIF(p.pop_2025, 0))::DECIMAL(6,3) AS bldg_per_person
13FROM read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/indices/building/v2/h3/h3_res=5/data.parquet') b
14JOIN read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/indices/population/v2/scenario=SSP2/h3_res=5/data.parquet') p
15  ON b.h3_index = p.h3_index                            -- BIGINT join — fast
16WHERE p.pop_2025 > 10000
17ORDER BY bldg_per_person DESC
18LIMIT 10;

S3 layout

1s3://us-west-2.opendata.source.coop/walkthru-earth/
2├── dem-terrain/
3│   ├── v1/h3/h3_res={1-10}/data.parquet          ← legacy (VARCHAR h3, has geometry/lat/lon)
4│   └── v2/h3/h3_res={1-10}/data.parquet          ← recommended (BIGINT h3, lean schema)
5└── indices/
6    ├── building/
7    │   ├── v1/h3/h3_res={3-8}/data.parquet        ← legacy
8    │   ├── v2/h3/h3_res={3-8}/data.parquet        ← recommended
9    │   └── globalbuildingatlas/zoomlevel={2-14}/data*.parquet
10    ├── population/
11    │   ├── v1/scenario=SSP2/h3_res={1-8}/data.parquet  ← legacy
12    │   └── v2/scenario=SSP2/h3_res={1-8}/data.parquet  ← recommended
13    └── weather/
14        └── model=GraphCast_GFS/date=YYYY-MM-DD/hour={0,12}/h3_res={0-5}/data.parquet
1s3://us-west-2.opendata.source.coop/walkthru-earth/
2├── dem-terrain/
3│   ├── v1/h3/h3_res={1-10}/data.parquet          ← legacy (VARCHAR h3, has geometry/lat/lon)
4│   └── v2/h3/h3_res={1-10}/data.parquet          ← recommended (BIGINT h3, lean schema)
5└── indices/
6    ├── building/
7    │   ├── v1/h3/h3_res={3-8}/data.parquet        ← legacy
8    │   ├── v2/h3/h3_res={3-8}/data.parquet        ← recommended
9    │   └── globalbuildingatlas/zoomlevel={2-14}/data*.parquet
10    ├── population/
11    │   ├── v1/scenario=SSP2/h3_res={1-8}/data.parquet  ← legacy
12    │   └── v2/scenario=SSP2/h3_res={1-8}/data.parquet  ← recommended
13    └── weather/
14        └── model=GraphCast_GFS/date=YYYY-MM-DD/hour={0,12}/h3_res={0-5}/data.parquet

License

CC BY 4.0 by walkthru.earth