What is Real-Time Mode in Apache Spark?
What is Real-Time Mode in Apache Spark?
Real-Time Mode is a Structured Streaming trigger that processes each record as it arrives rather than waiting for micro-batch boundaries, bringing end-to-end latency from seconds to milliseconds on the same engine and APIs. It replaces micro-batch scheduling with long-running tasks, pipeline scheduling and a streaming shuffle; the trigger interval (minimum 5 seconds) controls checkpoint commits, not per-record wait time. On Databricks it is generally available on classic compute with Databricks Runtime 16.4 LTS or later, supports Kafka sources and sinks plus stateful operators, and is rolling out in open source Spark from 4.1 through 4.3.
The article gives a detailed, benchmark-backed look at Spark's move into millisecond streaming, including support limits, latency figures and when a cheaper trigger is the better choice.
Source: Databricks Blog · databricks.com