How OpenAI Scaled Habitat to Store ChatGPT Data for 1 Billion Users
OpenAI transformed Habitat from a Python library into a globally distributed storage platform. It now handles over 22 million requests per second for ChatGPT's 1 billion users.
#scalability
Scalability
6 stories tagged Scalability
BatchDAG: LLM-Planned Execution Graphs for Scalable Enterprise Data Analysis
Researchers introduce BatchDAG, a system where an LLM generates typed directed acyclic graphs (DAGs) of operations—SQL queries, semantic searches, and parallel fan-outs—to enable scalable, ad-hoc analysis over enterprise-scale datasets without context overflow or linear latency.
New AI System GraphDC Solves Complex Graph Problems by Breaking Them Down
Researchers created GraphDC, an AI system that divides complex graph problems into smaller parts for easier solving. This could help with tasks like network analysis and logistics planning.
TTKV: A New Approach to Optimizing Long-Context LLM Inference
Researchers propose TTKV, a temporal-tiered KV cache that prioritizes recent memories in LLMs, improving efficiency for long-context inference. This method mimics human memory systems, offering a more scalable solution than existing approaches.
GM's Compound AI: A New Architecture for Safe, Scalable Autonomy
General Motors has unveiled Compound AI, a novel architecture designed to enhance the safety and scalability of autonomous systems. This development could reshape the future of AI-driven transportation.
6-Month Lessons from Running 14 AI Agents in Production
Operating 14 AI agents for half a year revealed critical insights on scalability, cost, and reliability. The experience highlights the need for robust infrastructure and continuous monitoring.