A climate model and a weather dashboard want the same data. They access it completely differently. Default chunking makes one of them 100x slower.
Chunk shape determines I/O patterns. A chunk of shape (1, 1000, 1000) stores one time-step across all latitude/longitude. Perfect for maps—fetch the current time, visualize all space in one request. Terrible for time-series—pulling a single grid point across 10 years means 10,000 separate file requests.
A chunk of shape (10000, 10, 10) stores 10,000 time-steps in a 10×10 spatial grid. Perfect for graphs—fetch all history for one location in one request. Terrible for maps—viewing the current state requires opening thousands of chunks.
The physics: object storage has high throughput but high latency. You want fewer, larger requests. Wrong chunking multiplies requests exponentially. A time-series query against spatially-chunked data opens one file per time-step. A map query against temporally-chunked data opens a file for every spatial cell. Both waste network I/O.
Target 50MB–100MB uncompressed per chunk. Smaller chunks mean request overhead dominates (latency kills). Larger chunks mean you fetch wasted data (bandwidth cost).
You usually can’t optimize for both simultaneously without duplicating data. Analyze your primary use case first. If 80% of queries are maps, optimize spatial chunks. If 80% are time-series, optimize temporal chunks.
The rechunker library transforms datasets when patterns shift, but that’s an offline operation—not a runtime fix.
The rule: Optimize for the read pattern, not write convenience. Profile real queries. Align chunks with the axis you’re slicing most.
Whenever you’re ready, here are 4 ways I can help you grow in GIS & spatial data:
Spatial Lab – My private community where GIS professionals, data engineers, and analysts connect, swap workflows, and build repeatable systems together.
Modern GIS Accelerator – A guided program to help you break out of legacy GIS habits and learn modern, scalable workflows.
Career Compass – A career-focused program designed to help GIS pros navigate the job market, sharpen their pitch, and find roles beyond traditional GIS paths.
Sponsorship: Interested in sponsoring this newsletter (or other content)? Learn more here and fill out the form to get in touch!
