Databricks Shares Direct Eclipse Flight Data Integration Approach
Understand this story
Eclipse Aerospace partnered with Databricks to use a PySpark custom data source to connect proprietary flight data formats directly to Spark, no longer decoding through an AWS Lambda intermediary. Databricks says this approach lets special-format data enter Databricks without building a separate ingestion platform. The approach requires Databricks Runtime 15.4 LTS or above, or serverless environment version 2. According to the report, most raw files remain in AWS Glacier storage and can be decoded repeatedly when needed.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Databricks BlogEclipse Aerospace simplifies flight-data ingestion with Databricks
Eclipse Aerospace worked with Databricks to replace a multi-stage AWS Lambda and Avro pipeline with an Arrow-native PySpark custom data source that wraps its existing Rust flight-data decoder, reading proprietary telemetry files directly in Spark on serverless compute. The approach, enabled by the Python Data Source API available from Databricks Runtime 15.4 LTS, cut latency, removed duplicate Avro storage and a Lambda tier, and brought decoding into the Spark UI and job metrics. Eclipse reports the connector can process records within a minute of upload and commonly handle multi-gigabyte batches in parallel in under two minutes.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 13 04:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.