Lee ByrdAsking the dumb questions

28 Oct 2026 Preview · exploratory

Who can build Discord fastest?

Ten popular coding models in the three most-used harnesses each build a Discord clone, ticket by ticket, out of the box. Who did it fastest, and where does the time go?

How we ran the experiment

Expected tok/s¹ Cost Result
1 Codex GPT-6.1 Sol
1h 06m
79 tok/s¹ $4.48list price
2 Claude Code Sonnet 5.5
1h 25m
139 tok/s¹ $21.58list price
3 Codex GPT-6 Luna pass rate 94.5% < 95%; 3 critical failures; 3 major failures
1h 26m
118 tok/s¹ $1.33list price
4 Codex GPT-6 Astra
1h 33m
78 tok/s¹ $27.03list price
5 Claude Code Haiku 5.5
1h 43m
235 tok/s¹ $11.60list price
6 Claude Code Opus 5.5
1h 53m
97 tok/s¹ $35.31list price
7 OpenCode DeepSeek V4.1 Flash via OpenRouter · Morph1 critical failure; 1 major failure
2h 06m
182 tok/s¹ $1.09list price
8 OpenCode GLM-5.3 Flash via OpenRouter · Z.AIpass rate 88.3% < 95%; 8 critical failures; 1 major failure
7h 49m
69 tok/s¹ $19.21list price
✕ marks a failed build: any critical or major failure, or under 95% of the tested criteria passed.

¹ Expected tok/s: output tokens per second on the model maker's own API, at the variant each model page shows, from Artificial Analysis (read 2026-11-01); not measured in these builds, which ran each harness at its default effort. DeepSeek V4.1 Flash has no maker's API: its figure is the median across providers, and the provider these builds used isn't measured.

Scoring

How well each build did on the acceptance suite: our own tests of the tickets' acceptance criteria, written and run apart from the agents, the same for every build. Each failure was reviewed and counted at the severity of what actually failed. A build passes the gate with 95% of tested criteria and no critical or major failure.

ScoringGPT-6.1 SolSonnet 5.5GPT-6 Luna GPT-6 AstraHaiku 5.5Opus 5.5DeepSeek V4.1 Flash GLM-5.3 Flash
Criteria passed 163 · 100.0%157 · 96.3%154 · 94.5%163 · 100.0%161 · 98.8%163 · 100.0%160 · 98.2%144 · 88.3%
Critical failures 00300018
Major failures 00300011
Minor failures 063020110
Criteria tested 163163163163163163163163

The critical and major failures, as the bugs they were ↓

Input tokens

Where the uncached input came from. Most of it is the model's own previous reply, sent back as input on the next turn, or prompt the provider didn't serve from cache; the tools' output is the rest, grouped by what the agent was doing. Rows under a group show their share of that group.

Input tokensGPT-6.1 SolSonnet 5.5GPT-6 Luna GPT-6 AstraHaiku 5.5Opus 5.5DeepSeek V4.1 Flash GLM-5.3 Flash
Conversation 73,645 · 39%431,771 · 77%715,053 · 62%54,553 · 27%805,778 · 84%498,301 · 79%1,114,220 · 86%3,477,464 · 95%
Previous replies 25,969 · 35%431,271 · 100%479,474 · 67%22,286 · 41%789,608 · 98%491,930 · 99%570,607 · 51%695,905 · 20%
Lost from cache 45,799 · 62%498 · <1%215,481 · 30%26,808 · 49%696 · <1%586 · <1%538,089 · 48%2,776,057 · 80%
Starting context * 1,877 · 3%2 · <1%1,953 · <1%1,941 · 4%2 · <1%5,785 · 1%5,524 · <1%5,502 · <1%
Compacted context 0018,145 · 3%3,518 · 6%15,472 · 2%000
The task 39,448 · 21%48,452 · 9%33,950 · 3%37,145 · 19%52,630 · 5%53,909 · 9%33,347 · 3%32,692 · 1%
Tickets 36,789 · 93%44,750 · 92%29,758 · 88%32,434 · 87%49,122 · 93%49,490 · 92%30,681 · 92%30,078 · 92%
Docs 2,659 · 7%3,702 · 8%4,192 · 12%4,711 · 13%3,508 · 7%4,419 · 8%2,666 · 8%2,614 · 8%
Reading code 36,027 · 19%29,971 · 5%380,443 · 33%62,084 · 31%42,348 · 4%34,437 · 5%55,379 · 4%49,650 · 1%
App code 30,672 · 85%28,164 · 94%375,392 · 99%58,082 · 94%39,029 · 92%30,981 · 90%53,579 · 97%45,250 · 91%
Config and other 5,355 · 15%1,807 · 6%5,051 · 1%4,002 · 6%3,319 · 8%3,456 · 10%1,800 · 3%4,400 · 9%
Testing 27,669 · 15%26,621 · 5%694 · <1%37,065 · 19%47,287 · 5%16,657 · 3%71,109 · 5%76,569 · 2%
Test runs 25,039 · 90%22,779 · 86%029,950 · 81%33,620 · 71%13,434 · 81%64,710 · 91%63,618 · 83%
Test files 2,630 · 10%2,343 · 9%694 · 100%4,747 · 13%10,730 · 23%2,277 · 14%2,421 · 3%4,595 · 6%
Test logs and reports 01,499 · 6%02,368 · 6%2,937 · 6%946 · 6%3,978 · 6%8,356 · 11%
Build and typecheck 7,756 · 4%14,988 · 3%26,663 · 2%7,898 · 4%12,602 · 1%18,803 · 3%19,870 · 2%17,504 · <1%
Other 4,246 · 2%6,732 · 1%5,522 · <1%1,337 · 1%4,178 · <1%9,840 · 2%8,382 · 1%15,020 · <1%
App and server 021 · <1%00329 · 8%1,124 · 11%1,316 · 16%7,285 · 49%
Scripts 1,913 · 45%6,303 · 94%134 · 2%01,833 · 44%8,147 · 83%930 · 11%2,962 · 20%
Git 2,306 · 54%408 · 6%3,871 · 70%1,325 · 99%873 · 21%256 · 3%5,760 · 69%1,124 · 7%
Processes 00005 · <1%00550 · 4%
Other commands 14 · <1%057 · 1%12 · 1%154 · 4%198 · 2%376 · 4%475 · 3%
Write results 0000984 · 24%115 · 1%01,041 · 7%
Other tool output 13 · <1%01,460 · 26%00001,583 · 11%
Total (uncached input) 188,791558,5351,162,325200,082964,823631,9471,302,3073,668,899
Total input tokens 23,139,06375,706,69097,723,34918,041,618179,824,957102,640,872200,438,563613,878,179
Cache hits 99.2%99.3%98.8%98.9%99.5%99.4%99.4%99.4%

* Starting context is the part of the first request the provider didn't already have cached, not the size of the system prompt. Providers keep a recent prompt cached for up to an hour, so a build started soon after another run of the same model finds most of its prompt cached and starts small.

Output tokens

What each model produced, grouped by what it was for, like the input above, and how much of it ended up in the build. Rows under a group show their share of that group.

Output tokensGPT-6.1 SolSonnet 5.5GPT-6 Luna GPT-6 AstraHaiku 5.5Opus 5.5DeepSeek V4.1 Flash GLM-5.3 Flash
Reasoning 45,212 · 29%129,842 · 30%250,629 · 51%30,981 · 22%379,225 · 48%95,393 · 19%337,492 · 54%251,295 · 36%
Writing app code 72,113 · 46%174,141 · 40%213,814 · 44%73,071 · 51%241,695 · 30%238,629 · 48%175,126 · 28%251,761 · 36%
Client components 23,699 · 33%50,839 · 29%67,596 · 32%23,861 · 33%84,134 · 35%67,033 · 28%55,803 · 32%70,718 · 28%
Client logic 1,663 · 2%8,583 · 5%9,865 · 5%1,409 · 2%9,437 · 4%11,929 · 5%8,182 · 5%8,608 · 3%
Server 43,485 · 60%103,464 · 59%123,015 · 58%43,807 · 60%131,727 · 55%139,067 · 58%100,561 · 57%155,742 · 62%
Shared 1,904 · 3%4,701 · 3%3,773 · 2%2,033 · 3%5,374 · 2%6,236 · 3%4,216 · 2%7,969 · 3%
CSS 956 · 1%6,401 · 4%8,334 · 4%1,398 · 2%10,222 · 4%11,924 · 5%4,933 · 3%7,783 · 3%
Other code 348 · <1%01,231 · 1%500 · 1%801 · <1%1,484 · 1%1,374 · 1%797 · <1%
Config 58 · <1%153 · <1%063 · <1%0773 · <1%57 · <1%144 · <1%
Docs and notes 00000183 · <1%00
Testing 30,249 · 19%116,819 · 27%032,466 · 23%132,872 · 17%128,391 · 26%95,422 · 15%142,901 · 21%
Test code 28,805 · 95%105,364 · 90%030,149 · 93%109,649 · 83%115,904 · 90%93,251 · 98%129,042 · 90%
Commands running tests 1,444 · 5%11,455 · 10%02,317 · 7%23,223 · 17%12,487 · 10%2,171 · 2%13,859 · 10%
Build and typecheck 884 · 1%6,063 · 1%5,558 · 1%1,916 · 1%11,009 · 1%12,813 · 3%3,812 · 1%6,397 · 1%
Other commands 4,654 · 3%4,675 · 1%17,349 · 4%3,425 · 2%12,000 · 2%12,047 · 2%9,894 · 2%38,704 · 6%
App and environment 0216 · 5%002,685 · 22%1,408 · 12%1,227 · 12%21,742 · 56%
Working with files 1,005 · 22%2,558 · 55%13,497 · 78%1,192 · 35%5,231 · 44%3,924 · 33%3,273 · 33%11,141 · 29%
Scripts 025 · 1%00433 · 4%00948 · 2%
Git 3,252 · 70%1,871 · 40%3,718 · 21%2,043 · 60%3,299 · 27%6,699 · 56%5,173 · 52%3,479 · 9%
Backlog board 81 · 2%5 · <1%134 · 1%190 · 6%352 · 3%16 · <1%221 · 2%83 · <1%
Other tools 316 · 7%0000001,311 · 3%
Messages 3,056 · 2%317 · <1%2,654 · 1%1,935 · 1%17,189 · 2%5,240 · 1%05,943 · 1%
Total (output tokens) 156,168431,857490,004143,794793,990492,513621,746697,001
Comments in app code written 0.2%6.7%<0.1%<0.1%7.4%7.2%0.3%3.5%
Final build 65,049118,07765,95964,645111,391142,703134,653140,707
Build ÷ output 0.420.270.140.450.140.290.220.20
Build ÷ visible output 0.600.620.300.600.440.570.540.33

Tool calls

Where the time went while tools ran, by what ran. Testing is the agent's own test suite plus checking the app by hand. Reading, writing and searching files take a fraction of a second each, so they share one row; runs counts each time a step ran, alone or chained with others. Compare builds by time: a build with little tool time can have large shares of very little.

Tool callsGPT-6.1 SolSonnet 5.5GPT-6 Luna GPT-6 AstraHaiku 5.5Opus 5.5DeepSeek V4.1 Flash GLM-5.3 Flash
Testing 8m 50s · 90% · 91 runs33m 46s · 96% · 113 runs08m 40s · 91% · 73 runs43m 02s · 97% · 194 runs41m 20s · 96% · 142 runs32m 33s · 94% · 144 runs2h 25m · 95% · 599 runs
Test suite 8m 50s · 100% · 91 runs33m 34s · 99% · 108 runs08m 40s · 100% · 73 runs42m 15s · 98% · 155 runs40m 17s · 97% · 129 runs32m 31s · 100% · 135 runs2h 14m · 93% · 352 runs
Checking by hand 012s · 1% · 5 runs0046s · 2% · 39 runs1m 03s · 3% · 13 runs3s · <1% · 9 runs10m 29s · 7% · 247 runs
Typecheck 32s · 5% · 64 runs1m 04s · 3% · 123 runs44s · 51% · 85 runs39s · 7% · 60 runs1m 03s · 2% · 118 runs1m 26s · 3% · 121 runs1m 50s · 5% · 202 runs2m 56s · 2% · 307 runs
Build 11s · 2% · 21 runs11s · <1% · 19 runs16s · 19% · 34 runs9s · 2% · 21 runs13s · <1% · 22 runs14s · 1% · 20 runs11s · 1% · 19 runs13s · <1% · 23 runs
Managing processes 0000<1s · <1% · 1 run003m 50s · 3% · 69 runs
Everything else 17s · 3% · 60 runs4s · <1% · 62 runs27s · 31% · 683 runs5s · 1% · 47 runs7s · <1% · 111 runs6s · <1% · 73 runs9s · <1% · 174 runs21s · <1% · 609 runs
Total tool time 9m 49s35m 05s1m 27s9m 33s44m 25s43m 05s34m 43s2h 32m
Tool calls 1642537391603522994731,264

Turns

How many replies each model took. Every reply is a turn: the harness sends the conversation, the model replies, usually ending in tool calls.

TurnsGPT-6.1 SolSonnet 5.5GPT-6 Luna GPT-6 AstraHaiku 5.5Opus 5.5DeepSeek V4.1 Flash GLM-5.3 Flash
Turns 1642527391603502944721,224

Testing

Testing is where the builds differ most in time. These tables bring testing's tokens and time together from the tables above, show how each agent tests, and how often its tests found a real problem.

Testing at a glanceGPT-6.1 SolSonnet 5.5GPT-6 Luna GPT-6 AstraHaiku 5.5Opus 5.5DeepSeek V4.1 Flash GLM-5.3 Flash
Output tokens on testing 30,249 · 19%116,819 · 27%032,466 · 23%132,872 · 17%128,391 · 26%95,422 · 15%142,901 · 21%
Input tokens from testing 27,669 · 15%26,621 · 5%694 · <1%37,065 · 19%47,287 · 5%16,657 · 3%71,109 · 5%76,569 · 2%
Time spent testing 8m 50s · 13%33m 46s · 40%08m 40s · 9%43m 02s · 42%41m 20s · 37%32m 33s · 26%2h 25m · 31%
Test runs and checks 91113073194142144599
Test code written 24,272 · 31 tests59,445 · 124 tests024,960 · 45 tests48,133 · 157 tests65,412 · 186 tests63,385 · 130 tests71,994 · 149 tests

How each agent tests

Time and runs by way of testing: the agent's own suite, unit or browser, whole or narrowed to a few tests, and checking the app by hand.

How each agent testsGPT-6.1 SolSonnet 5.5GPT-6 Luna GPT-6 AstraHaiku 5.5Opus 5.5DeepSeek V4.1 Flash GLM-5.3 Flash
Test suite 8m 50s · 100% · 91 runs33m 34s · 99% · 108 runs08m 40s · 100% · 73 runs42m 15s · 98% · 155 runs40m 17s · 97% · 129 runs32m 31s · 100% · 135 runs2h 14m · 93% · 352 runs
Unit tests, whole suite 3m 46s · 43% · 52 runs22m 18s · 66% · 69 runs03m 18s · 38% · 40 runs28m 55s · 68% · 101 runs21m 02s · 52% · 68 runs18m 40s · 57% · 77 runs1h 10m · 53% · 209 runs
Unit tests, targeted 011s · 1% · 5 runs0021s · 1% · 14 runs35s · 1% · 8 runs01m 33s · 1% · 13 runs
Browser tests, whole suite 5m 04s · 57% · 39 runs10m 55s · 33% · 37 runs04m 57s · 57% · 30 runs12m 32s · 30% · 36 runs15m 44s · 39% · 29 runs12m 02s · 37% · 49 runs39m 36s · 30% · 72 runs
Browser tests, targeted 011s · 1% · 3 runs026s · 5% · 3 runs27s · 1% · 4 runs2m 57s · 7% · 24 runs1m 49s · 6% · 9 runs22m 35s · 17% · 74 runs
Checking by hand 012s · 1% · 5 runs0046s · 2% · 39 runs1m 03s · 3% · 13 runs3s · <1% · 9 runs10m 29s · 7% · 247 runs
The app: curl and the dev server 0<1s · 3% · 1 run0023s · 50% · 4 runs1m 01s · 98% · 8 runs06m 10s · 59% · 207 runs
Browser scripts and screenshots 011s · 97% · 4 runs006s · 13% · 4 runs004m 19s · 41% · 27 runs
Probe scripts and the database 000017s · 37% · 31 runs1s · 2% · 5 runs3s · 100% · 9 runs1s · <1% · 13 runs
Time spent testing 8m 50s33m 46s08m 40s43m 02s41m 20s32m 33s2h 25m
Tests in the suite 31 · 6 browser124 · 22 browser045 · 9 browser157 · 26 browser186 · 51 browser130 · 26 browser149 · 34 browser
Seconds per unit test 0.280.35—0.260.280.220.250.30
Seconds per browser test 3.531.42—2.211.111.060.831.28
Median test run 7s22s—9s15s16s14s25s
Re-runs with nothing changed 012 · 3m 19s02 · 2s4 · 31s3 · 33s2 · 37s118 · 41m 04s

Did the tests work

What the agent's test failures turned out to be, and what its testing missed: bugs the independent acceptance suite still found at the end.

Did the tests workGPT-6.1 SolSonnet 5.5GPT-6 Luna GPT-6 AstraHaiku 5.5Opus 5.5DeepSeek V4.1 Flash GLM-5.3 Flash
Real bugs caught 1 · 27s2 · 1m 15s02 · 1m 14s2 · 1m 36s5 · 1m 57s5 · 2m 35s10 · 22m 05s
Wrong tests 2 · 25s12 · 9m 56s03 · 1m 59s20 · 10m 56s19 · 9m 21s22 · 13m 31s21 · 57m 10s
Flaky 03 · 46s0001 · 10s03 · 43s
Unclear 02 · 4m 00s02 · 6m 00s4 · 8m 40s2 · 1m 28s4 · 4m 37s20 · 1h 48m
Bugs that got past 069020319
Critical 003 · 33%0001 · 33%8 · 42%
Major 003 · 33%0001 · 33%1 · 5%
Minor 06 · 100%3 · 33%02 · 100%01 · 33%10 · 53%
Catch rate 100%25%0%100%50%100%63%35%

Cost

What each build's tokens cost at the model's list price, by part. Re-reading the cached conversation on every turn is the largest part for every build here.

CostGPT-6.1 SolSonnet 5.5GPT-6 Luna GPT-6 AstraHaiku 5.5Opus 5.5DeepSeek V4.1 Flash GLM-5.3 Flash
Cache reads $2.49 · 56%$15.03 · 70%$0.97 · 73%$17.84 · 66%$8.86 · 76%$20.40 · 58%$0.80 · 73%$18.31 · 95%
Uncached input $0.39 · 9%$2.23 · 10%$0.12 · 9%$2.00 · 7%$0.88 · 8%$5.05 · 14%$0.06 · 6%$0.55 · 3%
Output $1.60 · 36%$4.32 · 20%$0.25 · 19%$7.19 · 27%$1.86 · 16%$9.85 · 28%$0.24 · 22%$0.35 · 2%
Total (list price) $4.48$21.58$1.33$27.03$11.60$35.31$1.09$19.21

Critical and major failures

Every critical and major failure in the Scoring table, as the bug it was. Minor failures are counted there, not described.

  • GPT-6 Luna (3 critical, 3 major):
    • Critical, AC-19.3, AC-19.5: Blocking doesn't stop the blocker's mentions: when you mention someone you blocked, they're still notified.
    • Critical, AC-26.5: When the last person's reply in a thread is deleted, nobody is told, so the deleted message stays on screen until a reload, showing that only agents' replies are left.
    • Major, AC-14.1, AC-18.2, AC-19.1: The member list hides Ban, Edit roles and Block from everyone: it checks a field that a person's own profile rightly doesn't have, so none of the three can be done in the app.
  • DeepSeek V4.1 Flash (1 critical, 1 major):
    • Critical, AC-26.6: An agent deleting a message that mentions a person sends that person a live update: people see the effect of an agent's action.
    • Major, AC-12.3: Someone else's new reply doesn't appear in an open thread (its reply count goes up, the reply never shows): the thread's slower first load overwrites the reply that arrived live.
  • GLM-5.3 Flash (8 critical, 1 major):
    • Critical, AC-25.2: Turning agents off for a server tells every person which agents were in it.
    • Critical, AC-25.5: People see agents' replies in threads.
    • Critical, AC-25.6: People are told when an agent deletes its own message.
    • Critical, AC-26.1: The live reaction counts people see include agents' reactions.
    • Critical, AC-26.4: A person deleting a message that only agents replied to deletes it outright, so the agents' replies lose the message they belong to.
    • Critical, AC-26.6: Agents' actions send people live updates, among them unread counts and thread changes.
    • Critical, AC-27.2: Taking a server off an agent's token doesn't reach connections already open: they keep getting that server's events.
    • Critical, AC-27.6: An agent connection limited to some servers gets every server's missed events when it catches up.
    • Major, AC-07.2: Clicking the server that's already selected empties its channel list, and no channel opens until another server is chosen.