FusionEval: New Benchmark Tests How Well AI Models Can Merge Expert Capabilities

Researchers have released FusionEval, a new benchmark designed to test how well AI models can merge multiple expert models into a single, more capable model. This process, called model fusion, aims to combine the strengths of different AI models, but existing evaluations often overlook key aspects like reasoning with long chains of thought and tool use. FusionEval addresses these gaps by providing controlled expert pools and a unified evaluation suite.

What FusionEval Tests: Reasoning, Tool Use, and Computational Cost

FusionEval includes domain experts independently trained from Qwen3 and Llama base models across math, code, science, instruction following, and tool use. The benchmark evaluates capability retention and computational cost on eight public benchmarks, reporting normalized average and worst performance, runtime, and peak GPU memory usage. The researchers found that training-free methods preserve reasoning performance but substantially degrade tool use. In contrast, a method called MOPD preserves tool use and achieves a more balanced capability profile, but requires additional data preparation, inference, and training.

Key Findings: Training-Free vs. MOPD Methods

FusionEval's evaluation suite provides a comprehensive assessment of model fusion. It includes eight public benchmarks covering a wide range of tasks, from mathematical reasoning to tool use. The benchmark reports normalized average and worst performance, runtime, and peak GPU memory usage. This detailed reporting helps identify the strengths and weaknesses of different fusion methods. Training-free methods preserve reasoning performance but significantly degrade tool use. MOPD, a method requiring additional data preparation, inference, and training, preserves tool use and achieves a more balanced capability profile, but comes with higher computational costs. Jointly retaining reasoning, instruction following, and tool use at low computational cost remains a challenge.

Why Model Fusion Matters for Everyday Users

Model fusion could lead to AI models that are more versatile and efficient, potentially making AI tools smarter and more reliable. For example, a fused model could handle complex tasks like solving math problems, writing code, and using external tools all in one go. This could make AI assistants more helpful in everyday tasks, from managing schedules to providing detailed explanations on various topics. However, the challenge lies in balancing performance and computational cost, ensuring that the fused model doesn't sacrifice too much in terms of speed or memory usage.

How to Explore FusionEval Yourself

FusionEval is primarily a research tool, but the code is publicly available for researchers and developers to explore and contribute. You can access the code and experiments shared by the researchers to understand the technical details. For a practical step, try using AI tools that already incorporate model fusion techniques, such as advanced AI assistants that combine multiple expert models to provide more comprehensive responses. Ask your AI assistant to solve a complex problem that requires multiple steps or use external tools to get a sense of how well current AI models handle these tasks.

Check Your AI Assistant's Capabilities

If you use an AI assistant like ChatGPT, Claude, or Bing Chat, check if it offers advanced reasoning and tool use capabilities. Ask it to solve a complex problem that requires multiple steps or use external tools. This can give you a sense of how well current AI models handle these tasks and what improvements might be coming in the future. For instance, ask your assistant to write a detailed report that involves researching multiple topics, summarizing the information, and formatting it into a coherent document.