I’ve used AI coding agents every day at work since the spring. I’m not an AI engineer, and most of what follows is about checking work rather than producing it.

Starting with the slow parts

The first jobs I handed over were the ones I liked least. Turning a user story into test cases in the format our test management tool imports. Writing app maps, which are notes on every screen, field and API call in a product, so I didn’t have to rediscover them every time.

That went well, mostly. The test cases still needed editing, but editing a draft is quicker than starting from an empty file.

Suites that passed for the wrong reasons

Over the summer an agent and I built Playwright suites for two of the products. I directed and reviewed, and the agent wrote most of the code. They grew quickly and stayed mostly green, which I took as a good sign.

In September I went through them one test at a time, asking what each would do if the feature it covered was broken. A worrying number would have passed anyway.

  • One test accepted a form, an empty state or a table, and the form check also matched the app’s search box. It was true on nearly every page.
  • A cleanup step was wrapped so that if it failed, nobody heard about it.
  • A wait looked like it gave the page five seconds, but Playwright ignores the timeout option on isVisible(), so it looked once and moved on. There were dozens of those, some in shared helpers every test depended on.
  • A test written that same month wrapped a retry around a delete. The first call deleted the record, the retry got back that it was already gone, and the test passed. It passed whichever answer I told it to expect.

I’ve seen all of these in suites written entirely by people. The difference with an agent was how quickly they turned up. I’d been reviewing its code the way you review a pull request from a colleague you trust, and that was too light.

Writing the rule down each time

The thing that helped most was pretty boring. Every time a report or a test went wrong, I wrote a rule down so it wouldn’t go wrong the same way again.

Later I started numbering the rules. I could point at a number in the next session instead of explaining the same correction again, and a review stopped being my opinion against the agent’s. I wrote more about them in Checking What the Agent Told Me.

When the number went down

We had a coverage figure for one of the suites, and it looked respectable. When I checked what it was counting, most of it was a page loading and a heading appearing. Nothing had checked the data on the page.

There was a less obvious problem too. In Playwright, an annotation on a describe block is copied onto every test inside it, and we used annotations to record coverage. So one passing test could keep an operation marked as covered while the test that actually exercised it was failing.

We rebuilt the count. A test only counts now if it passed in a recorded run, and the annotation has to sit on the test itself. The figure dropped a long way. It was probably the most useful thing I did that month, and not a nice number to look at.

Measuring the attempts that fail

One button in the product failed occasionally, but only during full runs, never when its test ran alone. I ran the same action through the API dozens of times and clicked another button in the same flow over and over at different delays. Everything worked, and I had to withdraw two theories I’d built on that.

Every one of those measurements came from an attempt that worked. They told me the healthy path was healthy, which I already knew.

So I changed the test to print the server’s response and the deployment history in its own failure message. It failed again the next day and recorded everything. I still don’t know the cause.

Notes go stale

My agents keep notes between sessions, which helps, because a lot of what we learn about an environment is hard to find out. The trouble is that a note doesn’t tell you when it’s out of date.

Twice in September a note said something couldn’t be done. The first had been carried over from an earlier session and never retested. I asked twice for it to just be tried, and it worked straight away. The second was wrong too, and I’d already done a job by hand because of it.

Now I date new notes and say where they came from, and anything about access gets tried before it gets repeated.

What I haven’t got right

Plenty of my older notes still have no date or label, which makes them hard to trust.

I’m also still learning to write Playwright properly myself, and I’d rather say that here than have it come out in an interview.

Where that leaves me

A lot of the talk about AI skills is about prompts. Most of what helped me was the same stuff QA has always asked for. Does a check fail when it should, and is the evidence from today or from last week? Working with an agent meant asking those questions far more often.