Walkthru-Indices · walkthru.earth · Source Cooperative | Source Cooperative
walkthru-earth / indices
Open planetary indices that turn raw scientific data into a single, H3-hexagonal grid anyone can query with one SQL statement. Every index shares the same cell ID (h3_index), so joining terrain + buildings + population + weather is a one-line JOIN ON h3_index.
Why? Cities shape how people feel — but the data to prove it is scattered across dozens of agencies, formats, and projections. We unify it into one queryable, open layer so researchers, planners, and communities can finally see the full picture. Read more: walkthru.earth/hormones-cities
Indices
All datasets: Apache Parquet , ZSTD compression, sorted by h3_index.
v2 (recommended): h3_index is BIGINT (int64) — no geometry, lat, lon, or area_km2 columns (all derivable from the H3 index via DuckDB's h3 extension). Sorted by int64 for better compression and fast range-based spatial queries.v1 (legacy): h3_index is VARCHAR (hex string) with native Parquet 2.11+ GEOMETRY and lat/lon/area_km2 columns.
Quick start
S3 layout
License
CC BY 4.0 by walkthru.earth
1 s3://us-west-2.opendata.source.coop/walkthru-earth/
2 ├── dem-terrain/
3 │ ├── v1/h3/h3_res={1-10}/data.parquet ← legacy (VARCHAR h3, has geometry/lat/lon)
4 │ └── v2/h3/h3_res={1-10}/data.parquet ← recommended (BIGINT h3, lean schema)
5 └── indices/
6 ├── building/
7 │ ├── v1/h3/h3_res={3-8}/data.parquet ← legacy
8 │ ├── v2/h3/h3_res={3-8}/data.parquet ← recommended
9 │ └── globalbuildingatlas/zoomlevel={2-14}/data*.parquet
10 ├── population/
11 │ ├── v1/scenario=SSP2/h3_res={1-8}/data.parquet ← legacy
12 │ └── v2/scenario=SSP2/h3_res={1-8}/data.parquet ← recommended
13 └── weather/
14 └── model=GraphCast_GFS/date=YYYY-MM-DD/hour={0,12}/h3_res={0-5}/data.parquet
1 s3://us-west-2.opendata.source.coop/walkthru-earth/
2 ├── dem-terrain/
3 │ ├── v1/h3/h3_res={1-10}/data.parquet ← legacy (VARCHAR h3, has geometry/lat/lon)
4 │ └── v2/h3/h3_res={1-10}/data.parquet ← recommended (BIGINT h3, lean schema)
5 └── indices/
6 ├── building/
7 │ ├── v1/h3/h3_res={3-8}/data.parquet ← legacy
8 │ ├── v2/h3/h3_res={3-8}/data.parquet ← recommended
9 │ └── globalbuildingatlas/zoomlevel={2-14}/data*.parquet
10 ├── population/
11 │ ├── v1/scenario=SSP2/h3_res={1-8}/data.parquet ← legacy
12 │ └── v2/scenario=SSP2/h3_res={1-8}/data.parquet ← recommended
13 └── weather/
14 └── model=GraphCast_GFS/date=YYYY-MM-DD/hour={0,12}/h3_res={0-5}/data.parquet
1 flowchart LR
2 subgraph Raw["Raw sources"]
3 DEM_RAW["GEDTM-30m\n(30 m GeoTIFF)"]
4 GBA_RAW["Global Building Atlas\n(2.75 B polygons)"]
5 WP_RAW["WorldPop SSP\n(1 km rasters)"]
6 NOAA_RAW["NOAA AI-NWP\n(0.25° NetCDF)"]
7 end
8
9 subgraph H3["H3 conversion"]
10 direction TB
11 DT["dem-terrain\nelev · slope · aspect · TRI · TPI"]
12 BI["walkthru-building-index\ndensity · height · volume"]
13 PI["walkthru-pop-index\n2025 → 2100 projections"]
1 flowchart LR
2 subgraph Raw["Raw sources"]
3 DEM_RAW["GEDTM-30m\n(30 m GeoTIFF)"]
4 GBA_RAW["Global Building Atlas\n(2.75 B polygons)"]
5 WP_RAW["WorldPop SSP\n(1 km rasters)"]
6 NOAA_RAW["NOAA AI-NWP\n(0.25° NetCDF)"]
7 end
8
9 subgraph H3["H3 conversion"]
10 direction TB
11 DT["dem-terrain\nelev · slope · aspect · TRI · TPI"]
12 BI["walkthru-building-index\ndensity · height · volume"]
13 PI["walkthru-pop-index\n2025 → 2100 projections"]
1 -- DuckDB: join buildings + population in one query (v2 — BIGINT h3_index, no geometry)
2 INSTALL httpfs; LOAD httpfs;
3 INSTALL h3 FROM community; LOAD h3;
4 SET s3_region = 'us-west-2' ;
5
6 SELECT b . h3_index ,
7 h3_h3_to_string( b . h3_index ) AS h3_hex, -- hex string for deck.gl
8 h3_cell_to_lat( b . h3_index )
1 -- DuckDB: join buildings + population in one query (v2 — BIGINT h3_index, no geometry)
2 INSTALL httpfs; LOAD httpfs;
3 INSTALL h3 FROM community; LOAD h3;
4 SET s3_region = 'us-west-2' ;
5
6 SELECT b . h3_index ,
7 h3_h3_to_string( b . h3_index ) AS h3_hex, -- hex string for deck.gl
8 h3_cell_to_lat( b . h3_index )
14 WI["walkthru-weather-index\ntemp · wind · precip · pressure"]
21 DT -. topo corrections .-> WI
23 subgraph S3["s3://…/walkthru-earth/"]
25 B[indices/building/v2/h3/]
26 P[indices/population/v2/]
35 T -. JOIN h3_index .-> Q((Query))
14 WI["walkthru-weather-index\ntemp · wind · precip · pressure"]
21 DT -. topo corrections .-> WI
23 subgraph S3["s3://…/walkthru-earth/"]
25 B[indices/building/v2/h3/]
26 P[indices/population/v2/]
35 T -. JOIN h3_index .-> Q((Query))
AS
lat,
-- derive lat on the fly
9 h3_cell_to_lng( b . h3_index ) AS lon, -- derive lon on the fly
10 b . building_count , b . building_density ,
11 p . pop_2025 , p . pop_2050 ,
12 ( b . building_count :: FLOAT / NULLIF ( p . pop_2025 , 0 )):: DECIMAL ( 6 , 3 ) AS bldg_per_person
13 FROM read_parquet( 's3://us-west-2.opendata.source.coop/walkthru-earth/indices/building/v2/h3/h3_res=5/data.parquet' ) b
14 JOIN read_parquet( 's3://us-west-2.opendata.source.coop/walkthru-earth/indices/population/v2/scenario=SSP2/h3_res=5/data.parquet' ) p
15 ON b . h3_index = p . h3_index -- BIGINT join — fast
16 WHERE p . pop_2025 > 10000
17 ORDER BY bldg_per_person DESC
AS
lat,
-- derive lat on the fly
9 h3_cell_to_lng( b . h3_index ) AS lon, -- derive lon on the fly
10 b . building_count , b . building_density ,
11 p . pop_2025 , p . pop_2050 ,
12 ( b . building_count :: FLOAT / NULLIF ( p . pop_2025 , 0 )):: DECIMAL ( 6 , 3 ) AS bldg_per_person
13 FROM read_parquet( 's3://us-west-2.opendata.source.coop/walkthru-earth/indices/building/v2/h3/h3_res=5/data.parquet' ) b
14 JOIN read_parquet( 's3://us-west-2.opendata.source.coop/walkthru-earth/indices/population/v2/scenario=SSP2/h3_res=5/data.parquet' ) p
15 ON b . h3_index = p . h3_index -- BIGINT join — fast
16 WHERE p . pop_2025 > 10000
17 ORDER BY bldg_per_person DESC