KVBoost: Chunk-Level KV Cache Reuse Speeds Up LLM Inference by Reusing Data at Arbitrary Positions
Researchers introduced KVBoost, a chunk-level key-value cache reuse system for HuggingFace-compatible decoder models that enables reuse regardless of content position, reducing prefill latency in large language models.