Eclipse Aerospace simplifies flight-data ingestion with Databricks

Eclipse Aerospace worked with Databricks to replace a multi-stage AWS Lambda and Avro pipeline with an Arrow-native PySpark custom data source that wraps its existing Rust flight-data decoder, reading proprietary telemetry files directly in Spark on serverless compute. The approach, enabled by the Python Data Source API available from Databricks Runtime 15.4 LTS, cut latency, removed duplicate Avro storage and a Lambda tier, and brought decoding into the Spark UI and job metrics. Eclipse reports the connector can process records within a minute of upload and commonly handle multi-gigabyte batches in parallel in under two minutes.