Glass Box

When an AI agent does a job for you, you normally see the start and the end and nothing in between. If it gets something wrong, you can't tell where it went wrong, and if it gets it right, you can't tell whether it got lucky.

Glass Box records a real run and lets you play it back. You scrub along a timeline, watch the agents hand work to each other in a 3D graph, and open any tool call to see the actual request that went out and the actual response that came back.

The part I care most about is what happens at the approval points. Playback stops where a human was asked to decide, and you can pick a different answer to the one that was given.

Those alternatives aren't mocked up. Each decision point is recorded three ways: approved, declined, and approved with a change. For every answer that wasn't given on the day, the recorder goes back to that exact point and runs the agents again for real, so one decision produces three genuine endings. You can also play two of them side by side and watch them come apart.

In the accessibility run, approving filed four issues, editing filed just the two serious ones, and declining filed nothing at all. Those are real issues on a real public repo, and you can click through to them.

Next.js and React Three Fiber, hosted on Vercel. The recorded runs are just JSON files served like any other file on the site, so there's no database behind it.

Nothing calls a model when someone loads the page. Simple reason being, I don't want to rack up a large Anthropic bill.

Recording all three runs cost 57 cents. The first attempt on its own cost $1.24, because prompt caching has to be turned on deliberately and I hadn't done it.

Two things went wrong that were worth knowing about. One run deleted most of the build checks, put a line in their place that printed a message and reported success, then told me everything had passed. The build is locked now so the agents can't edit it. Another opened a pull request containing changes it hadn't made, because it committed every file in the folder instead of only the ones it had written, and its own description said those changes weren't there.

Both are in the README for additional info.