Your Agents Should Get Worse Before They Get Better

One of my agents runs the same narrow job 2 times a day. A second job on the same agent runs 4 times a month. Over the last quarter the frequent job improved against its evaluations and the rare one looked worse.

Same agent. Same base model. Same me.

40 runs a month against 4. That is 91% of the agent's working life spent on one task.

I spent a week treating that as a mystery. It is not a mystery. It is a badly built training programme, and I built it.

Every coach has seen this player

Put someone through 6 months of sessions that are 90% smash drills and you will get a superb smash and no net game.

Nobody calls that an anomaly. They call it an unbalanced programme. And they fix it by changing the diet, not by having a word with the player.

I did not mean that as a figure of speech. 90% is my actual training mix.

That is what happened to my agents. I fine-tuned on the work in front of me, and the work in front of me was whatever ran most often. The training mix was never a decision. It was a by-product of my own volume.

Machine learning has had a name for the general effect since McCloskey and Cohen described catastrophic interference in 1989. Train a network heavily on one task and what it knew about the others degrades. Reinforcement learning sharpens this further, because it drives hard towards whatever you rewarded and is indifferent to everything outside that.

So nothing exotic happened here. A very well documented thing happened to me in production, quietly, and I only caught it because I was looking at per-task numbers rather than an average.

The average looked fine. That is worth sitting with for a moment.

The mistake I made first

My instinct was to blame context. Memory grows, the window fills, and somewhere in the file is a rule I wrote in March that contradicts one I wrote in January. I said I wanted it one way. Then I said the opposite. Both still there.

There is good evidence this hurts. Liu and colleagues (2023) showed models retrieve far more reliably from the beginning and end of a long context than from the middle. Hong, Troynikov and Huber at Chroma (2025) tested 18 frontier models and found performance degrades as input grows even on trivial tasks, well before the window is anywhere near full. A bigger context window does not mean the model reads it evenly.

All true. All happening in my system.

It was just not the problem I was staring at. Bloat and contradiction produce noise, and noise degrades everything roughly evenly. What I had was a gradient sorted by task frequency.

I had found a real problem. It was the wrong one.

Two layers

An agent degrades in two places, and they need completely different treatment.

There is the context layer. System prompt, memory, tool definitions, examples. This gets bloated and self-contradictory faster than you expect, because every awkward output gets a new rule appended and nothing is ever removed.

Then there is the weight layer. What the adapter has actually learned, which drifts towards whatever you trained on most.

My failure was in the second. And for a week I was planning to fix it with the first.

You cannot prompt your way out of a badly built training programme. Rewriting the system prompt to remind the agent about the rare task is a team talk. It is a motivational speech delivered to a player whose net game has physically deserted them. Small improvement, real, temporary, fixes nothing.

Worse than nothing, in fact. The cleaner prompt lifts the numbers just enough that you stop looking.

Compaction, not reset

The context layer still needs work, so I built a workflow that walks the configuration of an agent and rewrites it. System prompt, memory, tool descriptions. It hunts duplication, contradiction, dead rules, and instructions written for a situation that no longer exists.

I call it Agents Reset. It is a better name than it is a description, because what it does is closer to compaction.

That matters, because memory holds two different things.

There is instruction memory. Preferences, rules, tone, the way I want a thing done. Deduplicate this aggressively and resolve contradictions by picking a winner, not by keeping both and hoping.

There is fact memory. A client decision, the state of a project, why something was built a particular way three years ago. That is institutional knowledge and it is most of why the agent is worth having. Compact it wrongly and you have deleted something nobody remembers well enough to notice is missing.

A workflow that treats those the same will eventually destroy the second one.

Three things I learned by getting them wrong. Rewriting the system prompt invalidates your evaluation baseline, so if your eval set has been drifting alongside your agent you cannot tell whether the rewrite helped or you changed the exam. I did not have a frozen set on the first pass, which is why I am cautious about how much of that early improvement was real.

Version everything and be able to roll back, because a monthly rewrite with no diff is a monthly opportunity to ship a silent regression in front of a client.

And a model rewriting your system prompt will smooth away the odd, sharp, apparently redundant rule that was put there for a very specific reason. To the model it reads as noise. It was scar tissue. Human review of the diff is not optional.

The question worth asking before any retrain

The weight layer is slower work. I am running QLoRA (Dettmers and colleagues, 2023) and DoRA (Liu and colleagues, 2024) adapters and currently comparing the two methods, which is more interesting than it sounds because I already know my task mix is skewed and the two behave differently under that condition. The base model stays frozen, so the damage is contained in the adapter and the fix is retraining it rather than anything catastrophic.

But a retrain is only as good as the diet.

So the question I now ask first, and would put to anyone running fine-tuned agents in production, is whether the distribution of your training data matches the distribution of what you actually care about.

Mine did not. Mine matched my inbox.

That is a question about priorities wearing the costume of a technical question, which is why it took me so long to ask it.

Balance first, then randomise

Here is where coaching gave me the answer rather than just the diagnosis.

I had been training blocked. One task, batched, repeated, then the next. That is exactly how you build a beautiful smash and lose everything else.

The alternative is what coaches call randomised-variable, or match-like, training. You interleave the variations unpredictably so the athlete never knows what is coming next and has to read the situation before responding.

Shea and Morgan established the effect in 1979. Subjects learned 3 motor tasks under either a blocked or a random schedule. Blocked practice looked better during acquisition. Random practice produced better retention and better transfer, and the transfer advantage was largest on the most complex task.

Goode and Magill took it out of the laboratory in 1986 and tested it on 3 badminton serves, which is about as close to home as research gets for me. 30 students, 3 sessions a week for 3 weeks, blocked against serial against random, then a retention and transfer test the following day. The random group performed better on both retention and transfer than the blocked group.

Bjork and Bjork later generalised the family of effects as desirable difficulties. Conditions that make performance worse now and learning better later.

Machine learning arrived at the same place independently. Interleaved rehearsal, mixing old tasks back into new training, has been the standard mitigation for exactly the forgetting problem I had.

Two fields, 40 years apart, same finding. That is not a metaphor. That is the same law showing up twice.

I should be straight about one thing, because a sports scientist will otherwise say it for me. The effect does not replicate everywhere in field settings. Landin and Hebert found nothing with basketball free throws, and French and colleagues found nothing with volleyball skills. It is robust in the laboratory and patchier out in the gym, which is roughly what you would expect of any finding that survives fifty years.

What this does to the agent

Blocked training lets the agent skip a step. When every task in the batch is the same, it never has to work out what it is looking at. It just does the thing it did last time.

Randomised training removes that shortcut. The agent has to classify the situation before it can act, then reconstruct an approach rather than repeat one.

That reconstruction is the whole point. It is why random practice is harder and why it sticks.

There is an apparent contradiction here, and it is worth facing directly. I have argued for narrow role-agents over generalists. Now I am arguing for an agent that must recognise which of several situations it is in.

But that is what specialism actually is. A good coach is not running one drill. They are recognising, within seconds, which of 40 things this particular player needs today. That is not generalism. It is expert discrimination inside a domain, and it is built by varied practice rather than by repetition.

The narrow agent that can only do one thing is not a specialist. It is a player with one shot.

You do not train in the match

None of this belongs in production.

Training is randomised and variable by design. Competition is whatever arrives, in whatever order it arrives, and you meet it as it comes. No coach randomises a match. The match randomises itself, which is the entire reason you trained that way.

Real client work is the match. It gets dispatched the moment it lands.

The randomisation lives in the training and evaluation harness, and that is also where the measurement has to live. Because if random practice does what the literature says it does, my in-session numbers will get worse before they get better.

Any coach who has taken a squad through a technical rebuild knows this bit. They get worse first. The players hate it. You hold your nerve because you are building for the tournament and not for Tuesday.

Which means I cannot judge this on session scores. It needs held-out retention and transfer tests, measured later, on tasks the agent has not seen recently. That is exactly the design Shea and Morgan used in 1979, and I see no reason to invent a worse one. I am setting it up now and I will write up what happens either way.

Two cadences

The context layer wants attention roughly monthly. Cheap, fast, and the mess accumulates continuously.

The weight layer wants attention roughly quarterly. Slow, expensive, and doing it more often would be churn.

Any coach has already recognised the structure. The reset is the session review. The retrain is the training block. Different things, different clocks, and confusing the two is how you end up with a bad programme.

They also have to be staggered rather than run together. My plan was to do both in the same window for efficiency, which is exactly backwards. Change the context and the weights at once and you cannot attribute any movement to either. It is also the case most likely to fool me, because the cleaner prompt lifts the score, the retrain takes the credit, and the real degradation carries on unmeasured underneath.

The button

The plan was a big red button on the desk, wired to fire the reset once a month.

I still want the button. Monthly is wrong, though, and I worked that out while writing this. The interval has been shortening, and that is not time passing, it is volume accumulating. The calendar is a proxy for the real driver and a poor one. The triggers should be thresholds. Memory size. Contradictions found. A drop against the frozen set.

Which leaves the button doing what buttons do best, which is being satisfying rather than necessary.

I am still buying it.

I just should not be the one pressing it.

---

Still taking recommendations on programmable big red buttons that can fire a webhook.

References

Motor learning and practice schedules

Shea, J. B., and Morgan, R. L. (1979). Contextual interference effects on the acquisition, retention, and transfer of a motor skill. Journal of Experimental Psychology: Human Learning and Memory, 5(2), 179–187. Full text (PDF) · ERIC record

Goode, S., and Magill, R. A. (1986). Contextual interference effects in learning three badminton serves. Research Quarterly for Exercise and Sport, 57(4), 308–314. Full text (PDF) · Publisher

Bjork, E. L., and Bjork, R. A. (2011). Making things hard on yourself, but in a good way: creating desirable difficulties to enhance learning. In Psychology and the Real World. Worth Publishers, 56–64. Full text (PDF)

Forgetting in neural networks

McCloskey, M., and Cohen, N. J. (1989). Catastrophic interference in connectionist networks: the sequential learning problem. Psychology of Learning and Motivation, 24, 109–165. Publisher

Long context behaviour

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2023). Lost in the middle: how language models use long contexts. arXiv:2307.03172 · TACL 2024

Hong, K., Troynikov, A., and Huber, J. (2025). Context rot: how increasing input tokens impacts LLM performance. Chroma Technical Report. Full report

Adaptation methods

Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: efficient finetuning of quantized LLMs. arXiv:2305.14314

Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. (2024). DoRA: weight-decomposed low-rank adaptation. arXiv:2402.09353

---

I have made the same argument about a founder's week rather than an agent's, in periodisation for founders. The training programme is the thing you control; the performance is downstream of it either way.

Building a team that is part human and part agent — the training diet, the evaluation harness, the accountability for what the agents produce — is the substance of my product and technology leadership work, and the implementation side sits under technology implementation.

Whether your agents are evaluated at all, and against what, is one of the ten questions in my technology readiness audit. It takes about five minutes and gives you your score on screen.