A new SF startup is building autonomous taxi-drones
and are all in on agentic development.
They move fast. They write specs and hand them off
to Fable to grind out...
Agents are smart, right? No need to read the code.
Autopilot v0.1.0 is out and they are looking for some volunteers.
You game?
Hi! I'm Tyson!
I've worked as a dev for 20 years now, mostly on systems where if you screwed up it's millions of dollars on the line; or worse, someone gets hurt. Mining, medical (I built an MRI!) and finance industries.
It's not the kind of software where you can just YOLO it with agents and if users report bugs in prod you just fix 'em.
Humans on the periphery?
All us devs are coming under increasing pressure to deliver faster, using AI. And the prevailing concensus is we get pushed to the periphery - write specs, black box agent in the middle, and we test+PR what it spits out.
This post is about my method - how to bank the speed up LLMs give us; but not acquire the downsides - congnitive surrender, loss of understanding and those scary, silent bugs that your tests don't catch and creep into your code base.
It keeps the human in the loop.
TLDR - What changes?
My full method:
- I create a copy of my code in ~/sandbox
- I give a goal.md spec file to an agent and it implements
- When its done, I ask it to generate replay-guide.md - which steps me through what it built;
- I replay it by hand (or with the Human Replay vscode extension), block by bl5ck
Everything is here:
https://github.com/utilitydelta/human-replay
Sounds Dumb?
You might think "This is dumb" and that's what PR's are for. But hear me out.
You replay it, you own it, you understand it
Code you replay, block by block, is code you own. You've given your brain time to digest what the agent did.
It's too easy to skim code the agent just wrote into your code base and then flog it to your teammate for a PR review. Slow down, replay it, hack it, and go back and forward with the agent asking it questions. Draw diagrams. Understand it. Really understand it.
If you skip the understanding part, that's cognitive surrender and it's genuinely terrifying.
Three agents were given a spec...
And came up with three different solutions! Specs never have enough detail, so the agent wings it. How do you know it's built right?
Sure we loop - We run adversarial reviews, unit & integration tests. Feed it back to the agent. But it's only as good as your tests' falsifier depth; And the behaviour of the system can never be fully exised by your test suite (see the Halting Problem).
So that leaves us with reading the code. Because code is the design of the system (See What is Software Design? Jack W. Reeves 1992.
Still Fast
The agent still did 80% of the work. It reasoned about what to build; mapped the delta to the existing code base; implemented it and tested it.
You genuinely still saved a ton of time. And all the 'agentic harness' tricks we do still apply here (see my build-method that I use)
We just can't get greedy. Human gotta stay in the loop.
Only for high risk code?
I don't use human-replay for everything I write with agents. For low-risk code, easy to implement specs, or where agents have good oracles to check correctness - just get the agent to smash it out and do a brief review.
For the genuinely hard work like distributed systems with a time dimension or safety critical or high risk code where people can get hurt; I use human-replay.
Just like everything in life, when you get off [social media platform], there is actually no black and white, only shades of grey. Use your judgement and experience as a professional software engineer.
The vsCode extension
Not gonna lie - I don't type out everything the agent wrote line by line, char by char. Not crazy :)
The Human Replay vscode extension is my sweet spot these days.
It reads the replay-guide.md file, reads the code with the AST (I use tree-sitter under the hood) and all
I have to do is hit Tab on my keyboard. Replay auto-stops per-phase and throws me the key invariants and some retrospective questions to dwell on
before I move to the next stage.
Sometimes the agent gets it right. So it's just hitting Tab to smash it out block by block. Sometimes there is a 'WTF' moment and I throw it back over
to the sandbox agent to re-work something. Sometimes I just hack it myself, fully old-school with a keyboard and FIM autocomplete.
Take-aways
Would love for you to give it a red hot go. Install the human-replay Claude plugin and the Human Replay vscode extension and you're set. Currently supports replaying JS/TS, C#, Rust, Python and Markdown files.
Resources
Human Replay Claude Skills Plugin: https://github.com/utilitydelta/human-replay
VSCode Extension: https://marketplace.visualstudio.com/items?itemName=UtilityDelta.human-replay
My build-method agentic harness skills: https://github.com/utilitydelta/build-method
Relevant essays & papers: