LLMs Are Terrible at Designing AI Systems That Use LLMs
Summarized by AI from reporting by Hacker News AI, published under our editorial policy.
Sean Goedecke explains why large language models (LLMs) consistently fail to design software that uses LLMs, opting for complex symbolic systems instead of leveraging their own capabilities. The root cause may be a general weakness in software design, not just training data limitations.
Key takeaways
- LLMs often fail to use themselves effectively in designing AI systems, opting for complex symbolic systems instead of LLM-based solutions.
- When tasked with designing a memory tool to flag contradictions, an LLM proposed a symbolic system to convert memories into logical atoms, which failed because natural language cannot be reduced to logic.
- Most training code predates LLMs, and reinforcement learning environments may not include API access for models to call other LLMs, contributing to this weakness.
- Current AI agents are generally poor at software design, making ordinary design errors like choosing doomed symbolic approaches over simple LLM calls.
Sean Goedecke, a software engineer and AI researcher, recently highlighted a significant limitation of large language models (LLMs): they are often terrible at designing software that uses LLMs themselves. In a detailed post, Goedecke describes how LLMs frequently opt for complex, symbolic systems instead of leveraging their own capabilities, even when the task at hand is well-suited for LLM-based solutions.
The Problem: LLMs Choose Ineffective Approaches
Goedecke shares an example where an LLM was tasked with designing a memory tool to flag contradictions between stored memories. Instead of using the LLM's natural language processing capabilities, the model proposed a complex symbolic system to convert memories into structured logical atoms for comparison. This approach failed because natural language is inherently ambiguous and cannot be neatly reduced to logical atoms. When the user questioned the model's approach, it acknowledged the problem but continued to iterate on the doomed symbolic system for weeks.
This isn't an isolated incident. Goedecke notes that when building Autodeck, an AI-generated spaced repetition tool, his agents would try to categorize cards using regex or keyword searches instead of simply making another LLM call. This tendency to reach for outdated or ineffective methods is a recurring issue with LLMs.
Three Explanations for Why LLMs Make These Mistakes
Goedecke offers three potential explanations for this behavior. The first is that most training code predates the widespread use of LLMs, so the models may not have learned to rely on LLM-based solutions. However, this explanation is incomplete because LLMs do not consistently reach for outdated technologies in other contexts, such as preferring jQuery over modern JavaScript.
The second explanation is that reinforcement learning environments used to train LLMs may not include the necessary credentials or API access for models to call other LLMs. This could discourage the models from developing this skill during training.
The third and most compelling explanation is that current AI agents are simply not very good at software design. They often pick the wrong tool for the job, whether it's attempting to build complex consensus algorithms instead of tolerating stale data or introducing synchronous checks in event-driven systems. Goedecke suggests that these mistakes are not alignment issues but rather ordinary errors of software design.
The Implications for Everyday Users
This limitation has significant implications for how we use LLMs in everyday applications. LLMs are good and bad at seemingly random tasks: they can work hard for hours but also make shockingly low-effort mistakes. They can draw on inhumanly broad expertise but fail to apply basic common sense. If users can predict where LLMs will excel and where they will falter, they can extract enormous value from these models. However, without this predictive ability, users may waste time and resources pursuing hopeless approaches.
For example, when a user implied that Opus 5.5 should use a judge model, the LLM instead constructed a series of regex checks. This mistake highlights the importance of asking the right questions and understanding the state of the field when working with LLMs. Users should be aware of the limitations and tendencies of these models to avoid similar pitfalls.
What You Can Do Today
If you're working with LLMs, start by asking the model about the state of the art for the task at hand. For instance, if you're using an LLM to design a system, explicitly ask it to consider LLM-based solutions. Additionally, be prepared to question the model's approach if it suggests something that seems outdated or impractical. Remember that LLMs are powerful tools, but they are not perfect, and their strengths and weaknesses can be unpredictable.
Frequently asked
- Why do LLMs choose symbolic systems over using themselves for AI system design?
- Goedecke proposes three explanations: most training code predates LLMs, reinforcement learning environments may not allow LLMs to call other LLMs, and current AI agents are simply not very good at software design.
- How can users avoid LLMs wasting time on hopeless approaches?
- Users should ask the model about the state of the art for the task, explicitly request LLM-based solutions, and question any outdated or impractical suggestions the model makes.
- Is this problem unique to system design or does it affect other LLM tasks?
- Goedecke notes similar errors in other contexts, such as building complex consensus algorithms instead of tolerating stale data, suggesting it is a general software design weakness.