Your Experts Have Partners. There’s HOPE for Keeping Them Together

Back to Articles Your Experts Have Partners. There’s HOPE for Keeping Them Together Enterprise Article Published November 9, 2026 Upvote – Alex M Tseng alexmtseng amazon Higher-Order Pruning of Experts in Mixture-of-Experts Language Models Introduction Mixture-of-Experts (MoE) models have enabled state-of-the-art LLMs to be scaled to hundreds of billions of parameters, but they come with a fundamental and costly caveat: every expert MLP must occupy memory at inference time, even though the vast majority are idle for any given token. For example, Qwen3.5-122B has 256 experts in each of its 48 MoE layers, and 95% of its ~250 GB parameter memory footprint is taken up entirely by these expert MLPs alone. However, for any given token, only 8 experts…