27B AI Model Rewrites Its Own Weights to Get Smarter Without Training
Summarized by AI from reporting by ArXiv cs.AI, published under our editorial policy.
Researchers have found that a 27-billion-parameter language model can edit its own internal weights to improve its accuracy by 6.6 points on unseen problems, using only its own reasoning and no gradients or training. This self-rewriting ability emerges at scale and could make AI systems more adaptable without traditional retraining.

Key takeaways
- A 27-billion-parameter language model can rewrite its own weights to improve its accuracy by 6.6 points on 490 unseen problems.
- The self-rewriting process uses only the model's own reasoning and does not require gradients or traditional training.
- The ability to self-rewrite emerges with scale: a 2B model made no edits, 4B and 9B models broke themselves, and only the 27B model succeeded.
- The model can repair a damaged copy of itself, restoring its accuracy on six image benchmarks from 7% to 70% (83% of the undamaged model).
- Self-rewriting gains persist in the model's weights and add no computational cost per answer, unlike test-time scaling methods.
Researchers have discovered that a large language model can rewrite its own internal weights to improve its performance, using only its own reasoning and no gradients or training. This breakthrough, dubbed "self-rewriting" in a paper published on arXiv, allows a 27-billion-parameter model to diagnose its own failures and edit its weights to become more accurate. The model achieved a 6.6-point improvement in accuracy on 490 problems it had never seen before, and this improvement persisted without additional computational cost. The finding could fundamentally change how AI systems learn and adapt, potentially making them more efficient and self-sufficient.
How Self-Rewriting Works
The study found that a 27-billion-parameter model could analyze its own performance, identify areas for improvement, and make changes to its internal settings. This process does not require traditional training methods or gradients. Instead, the model uses its own reasoning and the ability to run itself to make these edits. The model only applies changes when its own tests support them, ensuring that each edit leads to measurable improvement. Unlike traditional training, which requires extensive computational resources and infrastructure, self-rewriting only needs an open model and a few hours on a single GPU.
The model's ability to self-improve was demonstrated through multiple sessions. Over its first eight sessions, the model's accuracy only rose, showing consistent improvement. The model could also repair damaged copies of itself by locating and rewriting corrupted matrices. This repair capability was tested on six image benchmarks, where the model's accuracy improved from 7% to 70%, which is 83% of the undamaged model's performance. This demonstrates the model's ability to not only improve itself but also recover from damage.
Scale Matters: Only the 27B Model Succeeds
The researchers tested models of four different sizes from the same family. The 2-billion-parameter model made no edits at all. The 4-billion and 9-billion models attempted edits but broke themselves, becoming worse. Only the 27-billion-parameter model successfully rewrote its own weights to improve. This suggests that the self-rewriting ability emerges with scale, requiring a certain level of capability before a model can effectively diagnose and fix its own errors.
How the Model Edits Itself
The 27B model works on itself in a way analogous to how a human would. It compares its internal hidden states on answers that finish successfully against answers that run out of tokens. It then weakens the direction in its weights that separates these two types of answers, effectively reducing the tendency to get stuck or fail. This is a form of introspective debugging, where the model uses its own internal representations to identify and correct its weaknesses.
Why This Matters for AI Development
Unlike test-time scaling techniques that add computational cost per answer, the gains from self-rewriting persist in the model's weights and add no cost per answer. Unlike traditional training, it needs no pre- or post-training infrastructure, only an open model and hours on one GPU. This could lead to more adaptable and efficient AI systems that can learn from their mistakes and improve their performance without needing extensive retraining. The ability of AI models to self-improve could also reduce the need for frequent updates and maintenance, making them more user-friendly.
What the Research Means Going Forward
While this research is still in its early stages, it suggests a new paradigm for improving language models. As the paper states, "like a human, a model need not separate training from inference; improving a language model may come to take a conversation rather than a training run." This could fundamentally change how we think about AI development, moving from a model of separate training and deployment phases to a more continuous, conversational process of improvement.
Frequently asked
- Can all AI models rewrite their own weights?
- No, the ability to self-rewrite emerges with scale. In this study, only the 27-billion-parameter model succeeded; smaller models either made no edits or broke themselves.
- Does self-rewriting require additional computational resources at inference time?
- No, the improvements made through self-rewriting are persistent in the model's weights and do not add any extra cost per answer. The editing process itself only needs a few hours on a single GPU.
- Can self-rewriting repair damaged AI models?
- Yes, the 27B model located and rewrote corrupted matrices in a damaged copy of itself, restoring its accuracy on six image benchmarks from 7% to 70% without ever reading the undamaged weights.
- How does the model decide what to change in its weights?
- The model compares its internal hidden states on answers that finish successfully versus answers that run out of tokens, then weakens the direction in its weights that separates these two types of answers.