Agent benchmark

The benchmark asks a model to produce a building, runs real validation, returns diagnostics and gives it one repair turn. A pass requires valid NDJSON and zero blocking diagnostics after repair.

Current result — gpt-5.6-sol, 20 briefs, given buildPromptContext({ examples: 1 }) at the abridged card size that existed when the run was recorded:

6 of 20 validate first time; 11 of 20 validate after one repair.

A score with no model beside it says nothing: you cannot tell a hard language from a weak run. The number above belongs to one model on one prompt budget, and it will not transfer.

The failures are useful: they cluster around references, multi-level topology, openings that overrun hosts and site intent that cannot be repaired by changing one number. Those are exactly the places where a narrower authoring surface or better diagnostic earns more than a larger prompt.

Run it locally:

npm run benchmark:llm              # ask the model, validate, repair, score
npm run benchmark:revalidate       # re-score saved answers against today's rules
npm run benchmark:refresh-prompts  # rebuild the prompts without calling a model

benchmark:revalidate is the one that keeps this page honest: it re-runs the current validator over answers a model already gave, so a change in the rules shows up as a change in the score rather than silently rewriting history.

The output directory keeps the prompt, first response, repair response, diagnostics and compiled result together. Nothing is scored from prose.

Inspect every run

Each link opens the saved NDJSON in the model inspector. “First” is the agent's untouched answer; “repair” is the answer after it received structured diagnostics once.