Four Minor AI Design Choices Can Compound to Severely Hurt Long-Context Performance
A new study from ArXiv cs.CL reveals that four seemingly minor architectural decisions, each adopted by at least one of the Olmo, Llama, or Qwen model families, have a compoundingly negative effect on long-context extensibility. Any one choice alone has a minor impact, but combining three or more can cause significant performance drops.