Your Experts Have Partners. There's HOPE for Keeping Them Together

This post introduces HOPE (Higher-Order Pruning of Experts), a second-order expert-pruning method for Mixture-of-Experts language models that accounts for pairwise expert interactions via an F-matrix and a quadratic program, rather than scoring each expert independently. Evaluated on Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air with two calibration datasets, HOPE achieved the best average rank across conditions and gains of up to +6.1% on agentic coding tasks over the next-best method, with its advantage growing at aggressive pruning rates such as 50%. The post also describes a three-step calibrate/solve/prune package and notes limitations, including weaker advantages at 10–20% pruning and no cross-layer interaction modeling.