Object storage charges by request, not by GB. Opening a 10KB file costs the same as opening a 100MB file—100ms of latency, one API call. Streaming spatial ingestion creates thousands of tiny files. Your query engine waits on I/O instead of processing.
The small file problem is real. High-frequency spatial data (IoT sensors, vehicle tracks, log streams) naturally creates many small writes. Each write becomes a file. After a month, you have a million 5MB objects. A simple bounding box query requires a million API calls. That’s not I/O efficiency—that’s I/O torture.
Compaction merges tiny files into optimal sizes. Iceberg’s rewrite_data_files consolidates thousands of small files into, say, 128MB or 512MB targets. Fewer files. Fewer API calls. Same data, better layout.
The hidden value: spatial sorting during compaction. While merging files, sort geometries by Z-order or Hilbert curve. This aligns nearby features physically on disk. A bounding box query now reads contiguous bytes instead of scattered blocks. Spatial indexes become effective because the underlying data respects their structure.
The trade-off is explicit: background compute (merging, sorting) costs money. But read queries save that cost back hundreds of times over. Pay for compaction once per day. Save on every single query.
Scheduling matters. Micro-batch compaction during peak hours increases latency. Off-peak compaction (nightly, weekly) is standard practice.
The rule: Pay for compute once to compact; save on I/O every time you query. A data lake without compaction is just an expensive file dump. Optimize file sizes for the reader, not the writer.
Whenever you’re ready, here are 4 ways I can help you grow in GIS & spatial data:
Spatial Lab – My private community where GIS professionals, data engineers, and analysts connect, swap workflows, and build repeatable systems together.
Modern GIS Accelerator – A guided program to help you break out of legacy GIS habits and learn modern, scalable workflows.
Career Compass – A career-focused program designed to help GIS pros navigate the job market, sharpen their pitch, and find roles beyond traditional GIS paths.
Sponsorship: Interested in sponsoring this newsletter (or other content)? Learn more here and fill out the form to get in touch!
