We’re standing at the edge of a major shift in how we interact with AI. What began as casual late-night conversations has grown into a bold, emerging discipline: generative computing. This isn’t just tech jargon—it’s a fundamental change in how we design, manage, and integrate AI into software systems.
IBM Research is leading the charge with a simple but powerful idea: What if we stopped treating large language models (LLMs) like mysterious black boxes and instead used them like reliable, modular components in a software system?
From Guesswork to Engineering
Right now, working with an LLM can feel a bit like rolling the dice. A small change in your prompt can lead to wildly different results. That unpredictability isn’t acceptable in high-stakes industries like banking, healthcare, or government. IBM’s Chief Scientist Ruchir Puri compares it to the early days of internet search—when even a misplaced comma could mess everything up. You can’t build solid systems that way.
David Cox, Director at IBM Research, says our current method of “prompt engineering” isn’t really engineering—it’s more like tinkering. Generative computing changes that. It treats LLMs like functions—predictable, testable units that play a clear role in a larger system. Cox coined the term “generative computing” to describe this shift: LLMs aren’t replacing code; they’re becoming a new form of it.
Bringing Structure and Control to AI
The heart of generative computing is a smart orchestration layer—think of it like a conductor directing a symphony of AI models. Instead of feeding one long, complicated prompt into an LLM, developers can now break problems down into smaller, manageable parts. These “mini prompts” are then handled, routed, and checked at runtime, which ensures everything stays on track.
Cox uses the internet as an example: we don’t just send data and cross our fingers—it’s all about protocols, retries, and routing. That same reliability is now being built into AI systems, allowing for better tracking, error handling, and fallbacks. In short: more control, less guesswork.
With this structure, classic software principles like testing and debugging return. Developers can now write checks—“assertions”—to ensure the LLM behaves as expected. If something’s off, the system can re-run the task or switch to a backup.
Building Trust in AI
One major obstacle to using AI more widely is trust. What if a chatbot makes something up? Or a medical AI makes the wrong recommendation? Generative computing tackles this directly. New tools like hallucination detectors and context checkers are built in to prevent errors and ensure compliance.
Cox explains that developers can even insert a “guardian model”—a second layer of AI that checks the main model’s output and raises a flag if something looks wrong. This layered system brings a level of reliability that today’s LLMs can’t achieve with simple prompt engineering. It’s not just another chatbot—it’s a real part of the software stack that gets tested and validated like everything else.
A Blueprint for Smarter Machines
Cox draws comparisons to past milestones in computing—like the rise of software design patterns in the 1980s. Generative computing is the next evolution: a new framework for managing and directing LLMs. Instead of trying to understand every inner detail of these models, we put strong guidelines in place to make sure they act within set boundaries.
As Puri puts it, “We create accountability.” Cox compares the idea to a car engine—complicated, but enclosed in a safe system where everything is monitored, logged, and recoverable. These checks aren’t optional—they’re built in.
IBM’s own Granite models are made for this approach. They’re lightweight, efficient, and great at handling lots of small tasks instead of one massive prompt. It’s faster, cheaper, and more reliable.
New Tools, New Mindset
Generative computing doesn’t just change the code—it changes the way developers work. In one real-world test, a client used this method to break down a complex legal brief summary into nine separate stages. If one step failed, the system could fix it without restarting the whole process—something that’s impossible with a single prompt.
Puri believes this modular thinking will become the standard. LLMs won’t be seen as magical end-to-end tools anymore. Instead, they’ll be treated as infrastructure—as parts of a larger, testable system. Developers will get new tools, like LLM-specific functions that work directly with the orchestration layer for deeper, faster responses.
Cox predicts the rise of smart developer tools—IDEs that come with built-in LLM workflows, unit tests for prompts, and tools to track and compare how different models behave. Developers won’t need to abandon what they know. They’ll build on it, using new skills alongside familiar principles like inputs, outputs, and debugging.
Teaching AI to Behave
As this approach matures, expect new layers of abstraction, better standards, and even entirely new job roles. Cox sums it up best: “We’ve spent the last decade making AI sound smart. Now it’s time to teach it how to behave.