20 stories tagged Inference
Microsoft unveiled its third-generation Maia 300 AI chip, designed as the primary inference system for its cloud services. The chip improves cost and efficiency for running AI workloads and supports advanced reasoning and agentic tasks, reducing Microsoft's reliance on Nvidia GPUs.
Z.ai used its own AI model, GLM-5.3, to build and optimize the inference infrastructure for its faster version, GLM-5.3-Flash. The system achieved production readiness in under two weeks, with end-to-end throughput tripling relative to the initial baseline.
OpenAI's custom inference chip, Jalapeño, achieves up to 2.5x higher throughput and 30% lower latency than the best available GPUs, setting new industry benchmarks for speed and power efficiency in AI inference.
IBM has developed a new AI accelerator called Spyre that allows large language models to run directly on enterprise data systems, eliminating the need to move sensitive data. This cloud-native architecture is designed for secure, high-throughput AI inference in regulated industries.
Researchers from ArXiv cs.AI introduce Probabilistic Concept-Aware Steering, a new inference-time technique that uses concept-specific direction vectors to guide large language models (LLMs) toward more coherent, on-topic, and trustworthy responses — addressing a key limitation of existing steering vector methods.
ZML, a French AI startup backed by Yann LeCun, has released ZML/LLMD, a free tool that speeds up AI inference across many chips. This could lower costs for developers and businesses using AI.
Hugging Face introduced a new native-speed backend for vLLM that dramatically accelerates inference for large language models, not training. This could significantly reduce costs and latency for developers deploying AI models.
OpenAI unveiled Jalapeño, a custom AI inference chip built with Broadcom, marking a major step toward reducing reliance on Nvidia. The move highlights a broader industry trend by Google, Apple, and SpaceX to avoid single-supplier risk and could lead to faster, cheaper AI services.
LLM-d is a new open-source tool that facilitates distributed inference for large language models across multiple GPUs. It aims to make running powerful AI models more accessible for hobbyists and small developers without requiring specialized hardware or deep technical expertise.
OpenAI has unveiled Jalapeño, its first custom AI chip designed specifically for inference workloads. Built in collaboration with Broadcom, the chip aims to make AI processing faster and more energy-efficient, potentially lowering costs for users.
Baseten, a startup specializing in AI inference, is reportedly raising $1.5 billion just months after its last major funding round. This highlights the intense competition and high stakes in the AI infrastructure market.
AI chip startup Groq is raising $650 million to focus on AI inference, the process that makes AI responses faster and more accurate. This move comes after Nvidia’s massive $20 billion deal in the same space.
Kog AI has developed a method to run large language models at 3,000 tokens per second on standard GPUs, making advanced AI faster and more accessible. This breakthrough could significantly reduce the cost and complexity of AI applications.
DeepInfra is now part of Hugging Face's Inference Providers, making it easier to deploy AI models. This collaboration simplifies running models for developers and researchers.
Researchers introduce a framework that dynamically allocates compute resources and adapts generation strategies during inference. The method outperforms static approaches by focusing computation on challenging queries.
Researchers propose TTKV, a temporal-tiered KV cache that prioritizes recent memories in LLMs, improving efficiency for long-context inference. This method mimics human memory systems, offering a more scalable solution than existing approaches.
Researchers introduce Tide, a method for optimizing LLM inference by allowing early exits at the token level. This could significantly reduce computational costs for AI applications.
Cloudflare researchers introduced Unweight, a technique to compress large language model weights without losing accuracy. This could significantly reduce the memory footprint and computational cost of LLM inference.
Researchers propose a method to improve the calibration of large language models (LLMs) without labeled data. The technique leverages the models' internal signals to reduce overconfidence in answers.
OMLX is a new tool designed to optimize large language model inference specifically for Mac users. It promises to make running LLMs on Apple Silicon faster and more efficient.