Inference is the process of running a trained AI model on new input to produce an output, such as generating a response to a prompt, as opposed to training the model in the first place.
Inference is what happens every time you send a prompt to an AI model and get a response back. The model has already been trained; inference is simply using that finished model's learned weights to process new input and produce a prediction or generated output, without changing the model itself.
Training is the expensive, one-time (or periodic) process of teaching a model by adjusting its weights across a massive dataset, often taking weeks and enormous computing resources. Inference is what happens afterward, every single time the model is used, and while each individual inference request is far cheaper than training, the cumulative cost of serving millions of requests is a major factor in running an AI product.
Larger models generally require more computation per token generated, making inference slower and more expensive. Techniques like quantization, which reduces the precision of a model's numbers, and specialized inference hardware are widely used to make serving models faster and cheaper without retraining them.
Most of the real-world cost and latency users experience with AI products comes from inference, not training. As models are deployed to more users, optimizing inference — through better hardware, smaller distilled models, or techniques like caching — becomes as important to a product's viability as the quality of the model itself.
IBM has developed a new AI accelerator called Spyre that allows large language models to run directly on enterprise data systems, eliminating the need to move sensitive data. This cloud-native architecture is designed for secure, high-throughput AI inference in regulated industries.
Researchers from ArXiv cs.AI introduce Probabilistic Concept-Aware Steering, a new inference-time technique that uses concept-specific direction vectors to guide large language models (LLMs) toward more coherent, on-topic, and trustworthy responses — addressing a key limitation of existing steering vector methods.
ZML, a French AI startup backed by Yann LeCun, has released ZML/LLMD, a free tool that speeds up AI inference across many chips. This could lower costs for developers and businesses using AI.
Hugging Face introduced a new native-speed backend for vLLM that dramatically accelerates inference for large language models, not training. This could significantly reduce costs and latency for developers deploying AI models.
OpenAI unveiled Jalapeño, a custom AI inference chip built with Broadcom, marking a major step toward reducing reliance on Nvidia. The move highlights a broader industry trend by Google, Apple, and SpaceX to avoid single-supplier risk and could lead to faster, cheaper AI services.