5 stories tagged Moe
A new arXiv study pinpoints the damage from halving routed experts during prefill in MoE models: the F1 loss on short-answer QA is mostly answer formatting, concentrated in just a few prompt tokens like the chat-template tokens before the answer.
A new study reveals that Mixture-of-Experts (MoE) models like Phi-3.5-MoE and Gemma-4-27B-A4B route tasks using a principle equivalent to Huffman Coding. The 'Frequency-Diversity Law' explains how these models allocate sparse expert resources to common tokens and diverse expert committees to rare tokens, acting as information-theoretic engines.
A new arXiv study introduces Expert-Aware Contrast Decoding (EACD) for mixture-of-experts (MoE) models, showing it reduces AI hallucinations by leveraging expert-specific knowledge. The method outperforms standard contrastive decoding on transformer models, offering a path to more reliable AI assistants.
OpenMythos is an open-source implementation of Claude Mythos, using a looped transformer with Mixture-of-Experts routing. This project aims to advance theoretical AI research by providing a transparent, modular framework.
Researchers demonstrate that complex routing mechanisms in sparse Mixture-of-Experts (MoE) models don't significantly affect language modeling performance. A simple geometric routing approach with 80% fewer parameters performed comparably to standard methods.