Rethinking Memory for AI Inference

Moving compute closer to memory can cut data movement, power consumption and latency, but different AI workloads need different approaches to memory architecture, interconnects and processing. Software support, standardization and thermal management remain major barriers to wider adoption, with processing-in-memory and near-memory processing still early despite efforts such as LPDDR6-PIM and compute on custom HBM base dies. The article surveys industry views on how edge inference, where models are known ahead of time, may benefit from ahead-of-time compilation and near-data compute.