41 Open-Weight Language Models Benchmarked for Zero-Shot Intent Classification: Study Finds Best Models for Accuracy vs. Efficiency

Researchers have released a systematic evaluation of 41 open-weight language models for zero-shot intent classification, a core component of task-oriented dialogue systems. The study, published on ArXiv cs.CL, aims to help practitioners select deployable models under real-world constraints like compute, latency, and robustness. It tested models spanning 15 families and sizes from 135 million to 9 billion parameters across eight English single-label intent-classification datasets, plus a five-shot auxiliary result on the ATIS dataset.

## Models Tested: 15 Families from 135M to 9B Parameters The study evaluated models across a wide range of popular families, including Mistral, Llama, and Falcon. Each model was tested in a zero-shot setting, where they classify intents without prior training on the specific dataset. The researchers also included a five-shot evaluation on the ATIS dataset, providing a glimpse into few-shot performance with five labeled demonstrations.

## Key Findings: Larger Models Win on Accuracy, Smaller Models on Speed The study found that larger models generally performed better in terms of accuracy, but smaller models offered superior efficiency. The 9-billion-parameter models achieved the highest accuracy, while the 135-million-parameter models were the fastest. The researchers also noted that some mid-sized models struck a good balance between accuracy and speed, making them ideal for real-world applications where both factors matter.

## Practical Guidance for Developers Building Dialogue Systems This research provides a valuable resource for developers building dialogue systems. By offering a systematic evaluation, it helps practitioners choose models that fit their specific needs. For example, developers with limited compute resources can opt for smaller models, while those prioritizing accuracy can choose larger ones. The study also highlights the importance of considering robustness and latency in model selection.

## How to Apply These Findings Today Developers can start by reviewing the study's results to identify models that fit their requirements. If you're working on a project with limited computational resources, consider models like the 135-million-parameter variants. If accuracy is your priority, explore the larger models. The full study on ArXiv provides detailed benchmarks and recommendations for each model across all eight datasets.