iCreatAI News
Skip to content
Databricks Blog·· 5 days agoFeaturedAI score60

What is Real-Time Mode in Apache Spark?

What is Real-Time Mode in Apache Spark?

AI summary

Real-Time Mode is a Structured Streaming trigger that processes each record as it arrives rather than waiting for micro-batch boundaries, bringing end-to-end latency from seconds to milliseconds on the same engine and APIs. It replaces micro-batch scheduling with long-running tasks, pipeline scheduling and a streaming shuffle; the trigger interval (minimum 5 seconds) controls checkpoint commits, not per-record wait time. On Databricks it is generally available on classic compute with Databricks Runtime 16.4 LTS or later, supports Kafka sources and sinks plus stateful operators, and is rolling out in open source Spark from 4.1 through 4.3.

Why it matters

It gives a detailed, benchmark-backed overview of Spark's new low-latency streaming mode, including support limits, latency figures and production use cases.

Source: Databricks Blog · databricks.com