How to make scientific data archives AI-ready

The article argues that scientific data archives are poorly suited to AI use because data is locked in opaque files, and proposes a four-step architecture: place data in scalable object storage, index chunks rather than filenames, co-locate dataset-level metadata with the data in object storage, and catalog repositories rather than individual files. It says this approach lets agents and humans quickly find and fetch only the chunks a query needs, citing a 576x reduction in data fetched for a GOES-16 timeseries and a 100x speedup in common NASA queries. The piece describes this as how Earthmover's Arraylake works, built on Icechunk and Zarr, and notes the same recommendations apply to humans and AI alike.