iCreatAI News
Back to main site
Skip to content
Trending storyWatching

Apache Spark 4.3 Released: Real-Time Mode Supports Stateful Processing

1 reports1 sources1 days agoUpdated

Understand this story

AI overview

Apache Spark 4.3 has been released and is available in Databricks Runtime 19. The core change is bringing stateful processing into Real-Time Mode; according to the Databricks blog, the P99 end-to-end latencies for the three benchmarks dropDuplicates, window aggregation, and transformWithState are 136 ms, 139 ms, and 141 ms, respectively. The release also introduces shuffle-free single-task execution, suitable for eligible small single-partition query fragments; the single-task optimization and Arrow caching are both opt-in. The capability is available through Python, SQL, and Spark Connect, allowing users to apply changes automatically without manual merging. Spark's built-in file formats (including Parquet) are not covered.

AI overview based on source reports · Updated 1 days ago

Latest developmentNov 5 08:00
Apache Spark 4.3 Released and Available on Databricks Runtime 19

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 5
  1. Databricks BlogFeatured
    Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

    Apache Spark 4.3 extends Real-Time Mode to stateful processing, adding streaming aggregation, dropDuplicates, and Scala/Java transformWithState, with three benchmarked stateful workloads reporting P99 end-to-end latency below 150 ms. The release also adds SCD Type 2 support to Auto CDC, Project Feather's opt-in shuffle-free single-task execution and Arrow-native caching, Spark Connect observability and JDBC improvements, and SQL features including JSON_TABLE, VARIANT transformations, ASOF JOIN, and a nanosecond timestamp preview. It also begins a quarterly feature-release cadence, moving from one release every six months to one every three months.

Activity for this story

Current comparable activity
4
Peak comparable activity
10Nov 5 09:00
Change in the past 24 hours
-51%

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.