Lee ByrdAsking the dumb questions

31 Oct 2026 Preview · exploratory

How coding agents read and write code

Eight models in three harnesses built the same app. They read files, change code, run commands and test in very different ways.

Eight models, in Claude Code, Codex and OpenCode, each built the same Discord-style app from the same 24 tickets (the experiment). We logged every tool call. These are the differences in how they work, as ranges over each model's three attempts (GLM: one so far).

TL;DR

  • Almost everything goes through the shell. The harnesses all offer their own Read, Edit and Write tools. Most agents barely use them: Sonnet made 2 Read calls in about 250, DeepSeek 2 in about 470. They read with cat, sed and grep, and edit by running small Python scripts.
  • Three ways to change code. Claude models, DeepSeek, Sol and Astra write Python scripts that replace one string with another (s.replace(old, new)). Luna writes patches (apply_patch), 300–400 a build. GLM is the only one that leans on the harness's Edit tool (187 edits).
  • Two ways to read. Sol and Astra read whole files with cat. Luna reads windows of about 45 lines (the median) with sed -n '120,165p', 190–260 times a build, and searches with rg.
  • Claude trims what it runs. Claude models end 95–100% of their test runs in | tail or | grep (Opus's first build 69%), and filter command output 180–450 times a build. GLM does the same (91%). Sol, Astra and DeepSeek take the output whole (0–1% of test runs).
  • One call at a time. Every agent made about one tool call per turn; calls in parallel were rare (at most 9% of turns).
  • Turn sizes differ by four times. Luna takes 640–740 small turns (about 600 output tokens each). Sol and Astra take 144–181 (770–950). Claude takes 196–417 larger ones (1,500–2,300).
  • Fast testers write fewer tests, not faster ones. Every build kept the same test setup, so a test costs about the same everywhere. Sol and Astra keep 29–53 tests and spend 7–12 minutes testing; Claude keeps 55–205 and spends 16–54 minutes. Their acceptance results are close.
  • Only Opus looks at what it built. It screenshots its pages and opens the images. Codex never checks by hand.
  • Luna never ran a test. In all three attempts it ran the typecheck, never the test suite. Everyone else ran its suite 72–352 times.
  • DeepSeek never says anything. Zero tokens of messages in three builds. Haiku writes the most prose: 17–26k tokens.

Writing code

Agent How it changes files (calls per build)
Opus 5.5 Python replace scripts 120–167, cat > file <<EOF 38–56, sed -i 10–14
Sonnet 5.5 Python replace scripts 112–167, heredocs 5–9, sed -i 5–8
Haiku 5.5 Python replace scripts 159–195, heredocs 32–50, the Write tool 0–20
GPT-6.1 Sol Python replace scripts 68–116, heredocs 1–24
GPT-6 Astra Python replace scripts 85–104, heredocs 7–44
GPT-6 Luna apply_patch 305–406
DeepSeek V4.1 Flash Python replace scripts 218–288, heredocs 39–62, Edit/Write tools 0–33
GLM-5.3 Flash Python replace scripts 336, the Edit tool 187, heredocs 45, the Write tool 41

New files come from heredocs (cat > client/src/Blocks.tsx <<'EOF'). Changes to existing files come from a script that opens the file, replaces the old text with the new and writes it back. Opus wraps that in a small edit(path, pairs) helper, redefined in each script, that first asserts the old text is there. Codex models run their scripts from inside the JavaScript tool Codex gives them, so a Codex edit is Python inside a shell command inside JavaScript.

Reading code

Agent How it reads files (commands per build, not counting output filters)
Opus 5.5 cat 116–173, grep 37–85, sed -n 23–37, the Read tool 4–18
Sonnet 5.5 grep 48–55, sed -n 34–43, cat 28–34
Haiku 5.5 cat 68–140, grep 65–99, sed -n 30–52
GPT-6.1 Sol cat 22–56, rg 8–14
GPT-6 Astra cat 87–138, sed -n 24–29, rg 13–19
GPT-6 Luna sed -n 193–256, rg 94–107, git diff 19–91, nl up to 41
DeepSeek V4.1 Flash cat 85–105, sed -n 69–99, grep/rg 54–69
GLM-5.3 Flash grep 143, sed -n 132, cat 98, the Read tool 31

Sol reads least: 22–56 cat calls and about 10 searches in a whole build. Luna reads in small windows and checks its own diff often (git diff -- server/app.ts, 91 times in its first build).

Running commands

  • Testing: Claude models ran their test suite 78–186 times a build and spent 16–54 minutes testing. Sol and Astra ran it 72–96 times and spent 7–12 minutes. GLM ran it 352 times and spent 145 minutes. Luna: never. Why, below.
  • Filtering output: nearly every Claude test run ends in | tail -20 or | grep -E "fail|Error"; so do 91% of GLM's. Sol, Astra and DeepSeek take the output whole.
  • Probing the app: GLM starts the server and calls it with curl (183 curl commands in 55 calls). The others mostly don't.
  • The rest: npm for tests, builds and typechecks; git to commit each ticket; backlog to take and finish tickets.

Why some agents test faster

Every build kept the test setup it started with: browser tests run one at a time, and each browser run rebuilds the app, resets a database and starts a server first; each unit test gets a fresh database. No agent changed it, so a unit test costs about the same everywhere (0.1–0.6 s), and so does a browser test once its suite is a few dozen tests (about 1 s). A small browser suite pays more per test, because each run's rebuild is shared by few tests: Sol and Astra's 2–9 browser tests cost 1.2–3.5 s each. They still spend the least time testing.

Agent Tests at the end Time testing per build Acceptance criteria passed
GPT-6.1 Sol 29–36 7–9 min 159–163
GPT-6 Astra 38–53 9–12 min 162–163
Sonnet 5.5 55–124 16–35 min 157–163
Haiku 5.5 157–205 43–54 min 158–163
Opus 5.5 186–203 36–54 min 162–163
DeepSeek V4.1 Flash 115–133 28–33 min 160–162
GLM-5.3 Flash 149 145 min 144
GPT-6 Luna 0 0 149–155

All of them run their suites a similar number of times, 72–186 a build (GLM 352), so suite size decides the time. Most failures of the larger suites were the tests' own mistakes: Opus's first build fixed 19 wrong tests and 5 real bugs its tests found.

Checking the app by hand

Besides their test suites, some agents try the app themselves:

  • Opus looks at it. It writes a short browser script that photographs each screen, then opens the images (18 in its second build, 5 in its third). No other agent looks at the interface.
  • GLM drives the API with curl: 183 curl commands in 55 calls in one build, logging in, creating things and reading them back.
  • Haiku probes: throwaway scripts that watch live events, and psql queries against the database (20 in one build).
  • The Codex models never check by hand. Sol and Astra rely on their tests, Luna on the typecheck.

Thinking and talking

Share of output tokens that is reasoning, by agent: Opus 19–23%, Astra 17–22%, Sol 19–29%, Sonnet 30–32%, GLM 36%, Haiku 40–48%, Luna 45–51%, DeepSeek 52–54%. Messages to the user are a rounding error for most: Sonnet wrote 317–870 tokens of prose per build, DeepSeek none.

How we counted: every tool call up to each build's last ticket, from the harness's own session log. A call that both reads and edits counts in both tables. Commands after a pipe (| tail) are filters, not reads.