Researchers Discover That MoE Routing in AI Models Follows Huffman Coding and a 'Frequency-Diversity Law'
A new study reveals that Mixture-of-Experts (MoE) models like Phi-3.5-MoE and Gemma-4-27B-A4B route tasks using a principle equivalent to Huffman Coding. The 'Frequency-Diversity Law' explains how these models allocate sparse expert resources to common tokens and diverse expert committees to rare tokens, acting as information-theoretic engines.