Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

Apache Spark 4.3 extends Real-Time Mode to stateful processing, adding streaming aggregation, dropDuplicates, and Scala/Java transformWithState, with three benchmarked stateful workloads reporting P99 end-to-end latency below 150 ms. The release also adds SCD Type 2 support to Auto CDC, Project Feather's opt-in shuffle-free single-task execution and Arrow-native caching, Spark Connect observability and JDBC improvements, and SQL features including JSON_TABLE, VARIANT transformations, ASOF JOIN, and a nanosecond timestamp preview. It also begins a quarterly feature-release cadence, moving from one release every six months to one every three months.