4 stories tagged Latency
Researchers have developed a runtime-aware framework that predicts how long large language models (LLMs) will take to respond on heterogeneous edge devices like smartphones and IoT gadgets, achieving 95% accuracy by accounting for hardware, runtime backend, thermal variation, and prompt behavior.
Researchers propose latency-aware LLM query routing that considers generation speed at model instances, not just cost and accuracy. This could make chatbots and AI assistants respond faster while maintaining quality.
OpenAI has shared how they make voice AI respond almost instantly. This matters because faster responses make AI assistants feel more natural and useful in daily life.
OpenAI's new WebSocket implementation in the Responses API reduces latency and API overhead for agentic workflows. The update enhances model performance by enabling connection-scoped caching.