The Missing Ingredient in AI: Learning When, Not Just How

A year ago I steered my AI coding assistant through everything: run the tests, check the edge cases, think it through before writing a line. Today it does all of that unprompted, but the steps did not disappear. Instead, the judgment of when to take the steps quietly moved from my hands into the model. So how did it move, and can we make it move on purpose?

In an earlier article I called this the information-judgment gap: models know almost everything about how to do a task, but not when to do it, and I argued that reinforcement learning is how we close it. Today, this article is about the part I left out. How does judgment actually get built into a model, and does the recipe hold true everywhere? Answer, it does not, and where it breaks tells us a lot about what is still required when building secure software with AI.

The Gap: Knowing How vs. Knowing When

Models arrive already knowing almost everything we ask of them. How to debug a stack trace. How to write a test. How to run a security check. What they lack is the judgment of when to reach for each one.

Think of a medical student who has memorized every textbook and then freezes in the first real emergency. The facts are all there. Knowing what to do with them, right now, with this patient, is a different thing entirely. My coding assistant a year ago was that student. It could recite every best practice, but I had to be the attending physician standing over its shoulder saying “check for edge cases here,” “run the tests now.” The knowledge was never the bottleneck, but the timing was.

So here is the obvious question. If the knowledge is already in there, why not just close the last inch by telling the model what to do?

Three Tempting Shortcuts, and Why Each One Fails

1. The first instinct is to write a better prompt. Give the model one sharp standing instruction and let it apply everywhere. But a standing instruction commits to a single answer, and the answers that matter are conditional. Take a call the surgeon faces constantly: whether to operate now or keep watching. Cut too early and you put a patient through real surgical risk they might never have needed; wait too long and the window to fix it safely closes. No scan settles it, and no protocol can, because the right call shifts with each patient’s age, condition, and how fast things are moving. That is the ceiling every general instruction hits: one answer for all cases, where judgment needs a different answer for each. The knowledge was never the gap. The when is, and no prompt can hand that over.
2. The second instinct is to record what human experts do and add it to the training data. This data is expensive to collect, and worse, it captures the wrong thing. It is like handing the student a stack of order sheets from expert doctors: the decisions are all there, but not the reasoning behind them. The student copies the order without understanding the judgment that produced it. And the parts that are easy to write down were almost certainly written down already, absorbed during pre-training, and still did not teach the model when to act.
3. The third instinct is to let the model figure it out through trial and error: throw it at the task, signal whether it worked, let it adjust. This is reinforcement learning in its rawest, unguided form, and it is the right idea with a brutal failure mode. On a genuinely hard task, the model never succeeds. It is like dropping the student, alone, into a surgery far beyond their expertise. They will not stumble into pulling it off; so there is no success to reward and nothing to learn from. This is exactly why the reinforcement-learning environments for software engineering were so naive a year ago. On anything ambitious, the model scored zero every time, and zero everywhere teaches the model nothing.

What Works: Grow It

What actually works is to not aim for the hard task directly; instead you grow into it.

A year ago, engineers looked closely at what their models could and could not do on their own. Then, they built harnesses for tasks sitting just past that edge, e.g., an environment of real software-engineering problems like SWE-Gym (Pan et al., 2025), a training environment of real GitHub issues with executable tests that score each fix, that the model could attempt with support but not yet finish on its own.

With the harness in place, the model started producing successes. Those first successes became examples to imitate directly, a step called behavior cloning (literally cloning the successful behavior back into the model), which on its own can lift agents by well over ten points on standard benchmarks.

Then reinforcement learning took over, drilling the model on those same tasks until it could do them reliably without the scaffold. Once a tier was mastered, the engineers found the next set of tasks now just barely within reach, and ran the loop again. This echoes curriculum learning, the long-standing idea that ordering training tasks from easy to hard makes learning more efficient. The difference is that here the ordering is set by human engineers rather than an algorithm.

This is not a new idea. It is residency. A medical resident does not walk in and run the trauma bay on day one. An attending physician assigns cases just past the resident’s current reach, supervises the first ones, and hands over harder cases as each tier is mastered. Judgment grows case by case, always at the edge of what the resident can just barely do. That is how a student who freezes becomes a doctor who acts, and it is how my coding assistant went from needing me to say “run the tests” to simply running them. The judgment was grown into it, one reachable rung at a time.

But notice what this loop quietly depends on. At every rung, you need a success you can actually recognize: something the model did that you can verify was right, clone, and reward. Take away the ability to tell success from failure and the whole loop stops turning. Which raises the question that decides everything for security. What happens when success is not something you can cleanly verify?

Where the Recipe Stalls for Security

For part of security, success is beautifully clean. Take vulnerability discovery, finding a flaw an attacker could exploit. You can verify a candidate flaw by actually attacking it, a practice called penetration testing (running the attack to confirm it works). The exploit either lands or it does not. That yes-or-no is exactly the signal the grow-it loop feeds on. This is why models like Mythos now show genuinely strong offensive security capabilities. The loop had a clean signal to grow on, so it grew.

The trouble starts the moment you switch from attacking to defending, because two things the defender cares about most refuse to produce a clean signal.

1. The first is long-term consequences. Many security judgments are bets on the future. A design choice made today shapes a system’s security for years, but nothing at design time tells you whether you got it right. Was this the architecture that quietly invites a breach in three years? Is the codebase still maintainable enough to patch safely when the next threat appears? Is the risk tolerance right for what this project actually faces? These are like a treatment whose harm only surfaces two decades later. There is no exploit to run today that returns a verdict; so the loop has nothing to train on.
2. The second is completeness. Finding one real vulnerability is enough for the attacker, and enough to score well on a vulnerability-discovery benchmark. The defender has the opposite job. They have to find all of them, because the one they miss is the one that gets used. A single success proves nothing about coverage. Pointing a strong attacker model at the problem and letting it run does not rescue this: every new run can only add another finding, and no pile of findings ever amounts to a proof that nothing remains. It is the difference between diagnosing one disease and clearing a patient on a full physical, where the whole point is to have missed nothing. The reward the loop would need here is “nothing was missed,” and for most real systems there is no reliable way to confirm that: the findings in hand say little about the ones still hidden. The grow-it loop grows best on rewards it can check, and completeness is usually not one of them, so this gap does not close on its own.

Both failures trace back to the same missing thing: on the defender’s side, nothing hands the loop a clean verdict on whether it got the answer right.

The Frontier

Is it possible to make judgment move into the model on purpose? Yes. The grow-it loop is the real mechanism, and it has already worked its way through coding and through the offensive half of security. The judgment my coding assistant now shows was grown into it deliberately, rung by rung.

The difficult, unfinished part is the defender’s: judgment about consequences that stay invisible for years, and about completeness that no single success can prove. Closing the grow-it loop takes more than pointing the existing loop at security data. It needs feedback signals built for problems where success is not a clean yes or no.

This is the challenge we are focused on at Trent AI. We are building AI agents with real security judgment for defenders: not pattern matching, not the attacker’s clean wins, but the harder call of whether a decision holds up over the long haul and whether anything has been missed. And especially on the long-haul calls, the goal is not to replace human judgment but to amplify it.