Written by Alec Graves on 2025.03.22
LLMs make prototyping easier and allow you to work using APIs that you might not understand.
However, their use also makes it so that you do not need to understand the API.
This could be considered objectively good when the API is poorly constructed and riddled with arbitrary limitations but is stable through time (e.g., Windows APIs), but a failure to learn the API could indeed be limiting if the API deals with fundamentals of the language or hardware system. For example, a failure to learn the basic rules and patterns of Verilog and understand how those rules and patterns map to hardware layout will lead to a decreased capacity to think effectively to develop new designs that are not in the training set of LLMs. Further, since LLMs tend to generate mediocre software designs and patterns, their use for basic tasks could doom users to be stuck generating mediocre designs forever -or at the very least longer than would otherwise have occurred.
However, for an expert who does fully grasp the basics of their computing environment, the use of LLMs could very well be an incredible productivity multiplier, allowing the expert to offload non-critical (as deemed by the expert) minutiae to the automated agent while mental capacity is instead spent on designs and verification. This idea is similar to the ‘surgical programming teams’ model dreamed up by Brooks in his classic “The Mythical Man Month” - only LLMS are much faster than junior programmers at typing. However, again, over-reliance on these tools, especially if planning decisions are delegated, will cripple the architecture and design of your systems. Further, due to Amdhal’s law, your system is only as fast as its slowest parts; thus, all components must be attended intentionally for a fast/efficient resulting product.
Prototyping is one use case where the employment of LLMs may be practical. In prototyping, the goal is to quickly assemble functionality as fast as possible such that prospective users and others may experience the software and get started. This is really only a good idea for closed-source products and demos that can be thrown away completely for a full re-implementation by experts acting as experts. The majority of costs of software engineering come from maintenance and updates, and starting with a crippled architecture mired in LLM-generated mediocre code will cost your investors many-fold more than the cost of a re-implementation by experts who know how to build performant and working full-scale systems. Additionally, in the case of open-source, a release of a crippled prototype architecture could result in thousands of other products and projects relying on that (barely) working architecture, wasting a large amount of programming resources in dealing with limitations of a mediocre design.
Learning is another potential use case of LLMs, with LLMs having the ability to rapidly search through and present information from documentation. This is similar to using LLMs as a power-hungry manpage generator (complete with examples, which for some reason were left out of man pages for many GNU/Linux tools). Ideally, we would use LLMs as a learning tool that forces us to learn to build complex systems and internalize knowledge of computational machinery, perhaps by creating curricula that improves our capacity for critical thought in this space over time. This could take the form of asking LLMs to generate questions to test your knowledge, short answer questions of the technology, prompts for short code katas to help you apply concepts using the technology, open-ended short-answer architecture questions, examples of solutions using a technology to critique, and open-ended weekend project suggestions to apply your knowledge in a new manner.
Ultimately, writing, coding, or any work that results in input and output of long-form sequential processes is work that trains your mind to think differently (even better, as Neil Postman argues in “Amusing Ourselves to Death”). By offloading this work to LLMs, your brain and capacity to think natively in useful ways will atrophy. To remain an effective expert, you need to continually exercise your ability to think critically in the design space in which you operate, whether through writing or reading. LLMs could very well be useful as a learning tool, if you use them for this; however, reliance on LLMs (or external experts) for production will limit your ability to critically analyze the code generated by LLMs and think productively at all levels of the design space. As Mr. McCloy, my high school marching band instructor, drilled into us, “Use your own brain!” Your performance will be mediocre if you only uncritically follow the tools.
Perhaps using LLMs for common boilerplate generation or assisting in learning could be fantastic, but using them for writing production code that you do not fully understand is probably a bad idea. I do think a discerning expert could effectively integrate LLMs into their workflow to speed up certain tasks, but many times that expert will have a better result just writing the code themselves and not relying on mediocre boilerplate as a starting point. Perhaps when code-generating LLMs are used with syntax checking, static and dynamic analysis tools, formal verification, and robust object-oriented architectural tools (in the Alan Kay and Margaret Hamilton sense, not Java), they will be able to generate usable code. Right now though, software development is more of a fragile art where a mistake or a series of small sub-optimal decisions can wreck stability, performance, and maintainability in ways that are expensive to fix. In their current state, LLMs are not well suited for software development of real-world programs that perform important functions and have access to sensitive data (which is most programs you run on your computers and servers).