Databricks says most teams should choose Spark over Flink
Understand this story
Databricks has published a selection comparison guide for Apache Spark and Apache Flink, concluding that most teams should choose Spark. Databricks says Spark's Real-Time Mode reduces end-to-end streaming latency to the millisecond level: in a benchmark test at 10,000 records per second lasting 30 minutes, stateful aggregation p99 was 14 ms and stream-to-static-table join was 9 ms, while Flink was 45 ms and 95 ms respectively. These latency figures come from Databricks' own benchmark tests. Real-Time Mode requires the update output mode and reads and writes from Kafka or a row-level foreach sink; stateful operators will be available in open-source Apache Spark 4.3, and are currently generally available on Databricks, where they can run on classic compute or Lakeflow Spark Declarative Pipelines.
AI overview based on source reports · Updated 54 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Databricks BlogApache Spark vs Apache Flink: which should you use?
This Databricks post argues that Apache Spark is the better choice for most teams because it unifies batch ETL, SQL analytics, machine learning and streaming on one engine, and its Real-Time Mode now brings streaming to millisecond end-to-end latency. It presents a feature comparison table and a Databricks-run benchmark in which Spark Real-Time Mode matched Flink on stateless work and showed lower p99 latency on stateful aggregation and stream-static joins, while noting Flink remains a fit for specialized workloads such as very large keyed state, complex event processing and event-time timers. The article also covers how Real-Time Mode is enabled as a trigger on Structured Streaming queries and argues that running one engine avoids duplicating code, skills and governance.
Activity for this story
Not enough continuous observations to show a trend yet.