Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss
Summarized by AI from reporting by ArXiv cs.CL, published under our editorial policy.
A new arXiv study pinpoints the damage from halving routed experts during prefill in MoE models: the F1 loss on short-answer QA is mostly answer formatting, concentrated in just a few prompt tokens like the chat-template tokens before the answer.
Key takeaways
- Halving the number of routed experts during prefill in MoE models causes F1 score loss primarily due to answer formatting issues, not the inability to find the answer.
- In Qwen3-30B-A3B, the F1 loss from expert reduction is concentrated in the two chat-template tokens immediately before the answer.
- In DeepSeek-V2-Lite, the F1 loss from expert reduction is concentrated in specific tokens before the answer.
- The study used token-, layer-, and attention-path interventions to locate the damage across four MoE model families.
A new study published on arXiv (2610.22454) reveals that reducing the number of routed experts in mixture-of-experts (MoE) models during the prefill stage can significantly impact short-answer question-answering performance. The research uses token-, layer-, and attention-path interventions to locate exactly where the damage occurs across four MoE model families.
The Damage Is in Answer Formatting, Not Answer Finding
The study examined four different MoE model families and found that when the number of experts is halved, the F1 score loss is primarily due to issues with answer formatting. The models still locate the correct answers but often fail to complete the responses properly. In three of the four families, the model finds the answer but stops answering briefly.
The Loss Is Concentrated in Specific Prompt Tokens
The research highlights that the damage to performance is not uniform across all tokens. Instead, it is concentrated in specific tokens, particularly those that precede the answer. For instance, in Qwen3-30B-A3B, the loss is attributed to the two chat-template tokens before the answer. In DeepSeek-V2-Lite, the loss is also concentrated in specific tokens before the answer. This means that even small reductions in the number of experts can have a significant impact on the model's ability to format answers correctly.
Why This Matters for AI Serving and Applications
Running prompt tokens through fewer routed experts during prefill is a simple way to make MoE serving cheaper and faster. However, this research shows that the cost savings come with a specific, localized trade-off: degraded answer formatting in short-answer tasks. This is particularly important for applications where precise and well-formatted answers are crucial, such as customer service bots or educational tools.
Implications for Model Deployment
For developers and researchers deploying MoE models, this study provides a precise map of where performance degrades under expert reduction. The findings suggest that mitigation strategies should focus on the chat-template tokens and answer-initialization tokens rather than treating all tokens equally.
Frequently asked
- What is the prefill stage in an AI model?
- The prefill stage is the initial processing phase where the model processes the input prompt tokens before generating a response. This study specifically examines the effect of reducing routed experts during this prefill stage.
- What does it mean to reduce the number of 'routed experts' in an MoE model?
- In a mixture-of-experts model, each token is processed by a subset of specialized sub-models called 'experts.' Reducing routed experts means forcing each token to use fewer of these experts during processing, which makes the model faster and cheaper to run.
- Which specific models were studied in this research?
- The study examined four MoE model families, with specific findings highlighted for Qwen3-30B-A3B and DeepSeek-V2-Lite.