QwenPU: A Language Model Running Inside Another Language Model's Weights
Summarized by AI from reporting by Hacker News AI, published under our editorial policy.
Researchers have created QwenPU, a modified version of the Qwen3.5-4B language model that includes a 32-bit RISC-V CPU and a vector unit compiled into its weights, allowing it to run another neural network, Andrej Karpathy's stories260K, inside its own forward passes.
Key takeaways
- QwenPU is a modified version of the Qwen3.5-4B language model that includes a 32-bit RISC-V CPU and a vector unit compiled into its weights.
- The vector unit in QwenPU allows it to run Andrej Karpathy's stories260K neural network inside its own forward passes.
- The current QwenPU checkpoint is 26.7 GB, requiring a GPU with at least 40 GB of memory to load it entirely.
- The stories260K model runs at about 3,300 trace tokens a second on an L4 GPU, with a story token taking approximately 16 seconds.
- QwenPU is part of the broader Qwenputer project, which integrates a CPU into the transformer layers of the Qwen3.5-4B model.
What QwenPU Actually Does
The QwenPU project, developed by divagr1925, is a modified version of the Qwen3.5-4B language model. It includes a 32-bit RISC-V CPU and a vector unit compiled into its weights. This unique setup allows QwenPU to run another neural network, specifically Andrej Karpathy's stories260K (a 260K-parameter Llama model from llama2.c), inside its own forward passes. The stories260K model runs as a program on QwenPU and generates stories token by token.
Technical Specifications and Performance
QwenPU is part of the broader Qwenputer project, which integrates a CPU into the transformer layers of the Qwen3.5-4B model. The vector unit in QwenPU includes separate memories in DeltaNet heads, with 64-lane matrix rows representing a token. The vector results are streamed through the token stream, enabling the execution of complex operations within the model's forward passes. Every weight is set by a compiler, and Qwen3.5's Gated DeltaNet state serves as the CPU's memory.
The current checkpoint of QwenPU is 26.7 GB, requiring a GPU with at least 40 GB of memory to load it entirely. However, the story demo and checks can run on a 24 GB GPU by utilizing only the computer's layers. The project is still a work in progress, with efforts underway to trim the hidden size and reduce the checkpoint size.
The stories260K model, a C program, has its int4 weights loaded into the vector unit's weight memory. Every matrix row, residual add, softmax, and attention read is a vector operation. The model's memories are DeltaNet heads, consisting of 16 vector registers of 64 words, 384 int8 activations, and 4,152 row-chunks of int4 weights. The vector unit's operations are exact, with zero mismatches in random test programs and the first 30,000 steps of the story program. SiLU and exp are implemented as fixed-point function units.
The circuit's executor writes the whole story free-running on a GPU, checked against a reference at every one of its 3,184,920 steps. It runs at about 3,300 trace tokens a second on an L4, with a story token taking approximately 16 seconds, compared to 41 seconds for Qwenputer-npu.
Why This Matters for AI Research
The concept of running a language model inside another language model is a novel research direction. It demonstrates the potential for creating more efficient and powerful AI systems by leveraging the capabilities of existing models. Because the stories260K program is not compiled into the checkpoint, any model that fits can run the same way, making the approach generalizable.
How to Explore QwenPU Today
While QwenPU is still in the experimental stages, you can explore the project on Hugging Face and GitHub. If you are interested in AI and language models, you can try running the story demo or the vector unit checks on your own GPU. The storyrun tool writes the story on the computer's executor, and vxcheck runs the vector unit on this checkpoint's layers. The chip-only layers are about 13 GB and fit a 24 GB GPU.
To get started, visit the QwenPU page on Hugging Face or the Qwenputer GitHub repository. There, you will find the necessary code and instructions to run the model and explore its capabilities.
Frequently asked
- Is QwenPU free to use?
- Yes, QwenPU is released under the Apache-2.0 license, making it free to use and modify.
- Do I need a powerful GPU to run QwenPU?
- Yes, the current checkpoint of QwenPU is 26.7 GB, requiring a GPU with at least 40 GB of memory to load it entirely. However, the story demo and checks can run on a 24 GB GPU by loading only the computer's layers.
- Can I contribute to the QwenPU project?
- Yes, the project is open-source and welcomes contributions. You can find the code and instructions on the Qwenputer GitHub repository.