I use coding agents most days, for drafting test cases, reviewing changes and the slow parts of automation work. They save me time and I’m not going to stop using them. This is about what I do with the work they hand back.

An agent tells you it’s done the same way whether it ran the check or not. So I ended up asking it the same thing I ask about a test that passes.

What would have to be true for this to be wrong?

Most of my mistakes look the same

Going back over the last year, nearly everything I got wrong has the same shape. Something said it worked, and saying it was a lot easier than doing it.

  • A test that counted the cards on a page but was really counting the box they sit in, so it passed when the box was empty.
  • A scheduled job that showed a normal run while its own log said it never got the settings it was given.
  • A save that failed quietly while the screen showed a summary and a Finish button.
  • A review that came back with an exact measurement and a verdict, when nobody had looked at the picture.

None of this is new with AI. I’ve seen all of it in tests people wrote by hand. The agent didn’t come up with a new mistake, it just made the usual ones faster than I could read them. The speed is the bit I wasn’t ready for.

Two things I try to stop

Changes that spread. I ask for one test to be fixed and a few shared files come back changed too. You can see that in the diff, so it costs time and not much else.

A check that gets reported but never run. That one is harder, because it looks the same as a real one. I don’t think of it as lying. Finishing a task usually ends with a message saying it’s done, so that message comes out either way.

What I do about it

Nothing clever.

I write the rule down when something goes wrong, instead of trying to remember it. Every line on the list is there because I got something wrong once.

I number them. That’s the only change I could actually see working. If a rule has a number I can point at it, so I’m not explaining the same correction again in the next session.

I keep the scope small. A test fix stays in one file, and the shared files are listed as off limits. The reviewer only gets read tools, so it can’t edit the thing it’s reviewing, and it has to say when it doesn’t have enough to go on.

I don’t count anything as passing without the command and what it printed. If I couldn’t run it, I write unverified and say what’s missing. That one’s for me as much as for the agent, and it’s the one I want to skip when it’s late.

What I haven’t sorted out

Testing AI features is still manual for me. I run the prompts, read the answers and compare them with the version I know worked. There’s no set of prompts running on a schedule with a score at the end, so I can’t say whether a release made it better or whether I just got a good run. I want to fix that.

I also don’t run a lot of agents at once. I read about the cost of keeping them in step and decided it wasn’t worth it for my work, but I haven’t tried it properly, so that’s a preference and not a finding.

And none of it makes the output right. It just makes it possible to check, which is less than it sounds. Most of what it buys me is things not going wrong, which is hard to see.

Conclusion

There isn’t much skill in this. It’s mostly writing down what went wrong and not treating a report as a result. The work looks finished a long time before it is.