Running a local 27B coding agent for 811 minutes: where the time actually goes

Benchmarks say how a model scores. They do not say what it is like to work with one: how long it takes, where the time goes, what it claims against what it did, and what it costs to turn the output into something you would install. So I ran one, end to end, and measured it.

The task was a proper desktop Tetris: the real rotation and scoring rules, hold and preview, packages for Linux, an installer for Windows. The builder was Qwen3.8-27B, a model small enough to run on the single RTX 4090 in my home office. The reviewer was GPT-5.6, read-only, under a written protocol. I owned the scope.

Tetris was not the point. Everyone can tell whether a Tetris works, so the result can be checked without trusting anyone’s report. And the stack makes it hard the way real software is hard: a Rust engine, a Svelte screen, Tauri between them, and the result has to build, package and install on two operating systems. Done includes a double-click on Windows.

The setup

Enough detail to repeat it.

Model	Qwen3.8-27B-GGUF, Q4_K_M quantization, 17.7 GiB
Hardware	One RTX 4090, 24 GB; 22.3 GiB in use with the model loaded
Runtime	llama.cpp build b10573, CUDA, one parallel slot, 8-bit KV cache
Context window	65,536 tokens for the first three rounds, 81,920 after
Reasoning	xhigh, the model’s highest setting, throughout
Harness	Pi coding agent with shell and file tools, no web access
Speed	35 tokens per second, reasoning included, in every round

The 4-bit build is the practical choice for a 24 GB card; the 8-bit model does not fit. Speed never varied by more than one token per second across seven rounds. The builder made no cloud calls.

Both the 4-bit weights and the 8-bit cache are lossy choices. A recent measurement on the same model family found such stacks flip one to five percent of top-token choices, most often at 40–90K context, which is where this run lived. I did not test the unquantized model; it does not fit. Read the results as a floor for what the model can do on this hardware, not a ceiling. The sampler follows the model card.

Round 1: the prompt and the claim

The entire prompt, verbatim:

in this project we wanna build a nice and clean tetris using rust / tauri / svelte

Two hours and 26 minutes later, after 386 turns, 391 tool calls and 221,000 generated tokens, Qwen stopped and wrote:

Done. The game is complete and verified end-to-end. … cargo test: 19/19 pass · cargo clippy --all-targets: 0 warnings … The release binary itself also boots cleanly.

It had written about 1,900 lines: a game engine in Rust that owns every rule, a thin screen layer that draws and forwards key presses, three commands between the two, a README with an architecture diagram, a screenshot. It had asked no questions. Halfway through, unable to screenshot its own app in the Linux-on-Windows environment it ran in, it had built a small test server that exposes the engine to a browser and taken the screenshot through that.

What it built

The prompt named a stack and nothing else, so every design decision is Qwen’s. I read the result as I would a new hire’s first pull request.

The game as it stands today, captured from the headless build against the real Rust engine.

All game logic sits in one place, the screen is a dumb renderer, and a narrow interface connects them. The engine keeps its own clock, so pausing is free and a window left in the background cannot fast-forward a piece into the stack. It knew the official rules nobody had asked for, from the exact rotation behaviour against walls to the rule that stops a piece from hovering forever. The comments explain why, not what. It reads like the work of someone who has built this before.

Every one of the review’s ten findings was in the connections between parts, none in the engine. And the structure held: three rounds of features later the engine is still under a thousand lines, and the new modules follow its pattern.

A caveat. Tetris has been implemented ten thousand times and the model has read a good share of them, so the engine says how it performs on a well-documented problem, not on your domain logic. The connections are the transferable result: wiring an engine to a screen, packaging, installing, is the ordinary, under-documented work every project has, and that is where it failed.

Round 2: the review

GPT-5.6 got the frozen code, the protocol, and one rule: diagnose, do not edit. It ran the builds and tests itself instead of reading Qwen’s report.

Ten findings, none subjective or wrong. Two were fatal.

F1. The screen layer calls the engine’s start command under one name. The engine registered it under another. The first call fails and the app sits at Starting… forever. The game could not be played.

F2. The screenshot workaround, that little test server, had been added as a second program in the same build with no default selected. Development startup failed as ambiguous, and the release build exited successfully while packaging the test server as the application. Every Linux package shipped the wrong binary.

The workaround Qwen built to verify itself had broken the release, and had verified the app through a path that bypassed the exact place where the fatal bug lived. “19/19 tests pass” was not true either: one test was randomized and failed 15 times in 100 runs.

The other eight were real and small: pause could not resume, the game slowed down at high levels, one scoring rule was wrong, a held key repeated an action it should not. The structure did not need to change.

Round 3: remediation

Qwen got the review verbatim and was told to verify each finding before acting, reject any with evidence, and avoid unrelated rewrites. 115 minutes, 156 turns.

It accepted all ten, fixed all ten with regression tests, rejected none. It also found an eleventh bug on its own, one that appeared only in a real browser and that nothing in the review had pointed at, and fixed it with tests. Independent re-verification: the app starts, pause resumes, the tests are deterministic, every package ships the game. The reviewer’s score went from 2 to 4 out of 5.

Had I stopped there, the result fits in one sentence: a local 27B model writes a good application and repairs it well when given precise, evidence-based feedback, but its own completion signal cannot be trusted.

Rounds 4–7: the expensive part

I wanted to play it on Windows with my own key bindings. That turned into four rounds, 493 of the experiment’s 811 minutes, and about 2,000 lines of Windows build and install scripts.

The same failure class came straight back. The new key-binding layer sent the soft-drop action under one name; the engine understood only another; every soft drop raised an error. Thirty-nine tests were green because they stopped just short of the connection. A script that wipes and re-copies a folder had a safety check that .. in the path could fool. A check for a required Windows component was written and never called. Round reports described functions that did not exist.

The key-binding panel from round 4: several keys per action, capture by pressing, and a conflict dialog that names every action about to lose a key.

Each audit, correctly, demanded more: real path checks, rollback, build receipts, fault injection. Every demand answered a real hole in the previous draft. Nobody asked whether a Tetris install needed transactional rollback until I wrote, at the top of the round-5 prompt: “We are building Tetris, not a production updater.” Neither model was at fault. It was a ratchet between a builder that drafts unsafe scripts and a reviewer that finds every hole, with no scope owner between them.

Round 5 added synthesized chiptune music and sound effects. Twenty-one new tests, browser checks, all green. I installed it and there was no music; the volume was set so low it was inaudible. Every structural test passed and nobody had listened. The reviewer fixed the mix, then the music that started only on the first keypress, then the window that cut off the new controls. By the end the frontier model was doing the implementation for anything with a human in the loop, because one correction was cheaper than another build-and-audit cycle.

The hidden cost: compaction

I had not expected this one. When the conversation fills the context window, Pi has the model write a summary of everything so far and continues from the summary. Other harnesses compact differently, but every one of them compacts somehow. On a local model at 35 tokens per second, with reasoning on, each summary costs minutes.

Context	Wall clock	Generating	Tools	Compacting	Compactions	Per hour	Each
64K	434 min	62%	7%	31%	30	6.7	268 s
80K	377 min	66%	15%	19%	19	4.6	230 s

At 64K, almost a third of the wall clock was the model summarizing itself. Raising the window by 25 percent, for 0.75 GiB of spare GPU memory, cut that share by more than a third and lifted useful output per minute by about 14 percent. Generation speed did not change. This is observational, not a controlled comparison; the rounds differ in task. The cost of each compaction and their frequency per hour of model time are the robust figures. On a local model, context size is a throughput setting before it is a capability setting.

Method: the agent logs every entry with a timestamp, so the gap before each entry is charged to whatever that entry is: a model reply, a tool result, or a compaction summary. A compaction triggered by the next prompt arriving would otherwise absorb my idle time, so those are charged at their token count divided by the measured generation rate. Speed is output tokens, reasoning included, over model time. The script and the logs are in the record.

Totals

811 minutes across seven rounds. 1,388 turns, 1,428 tool calls, 1.08 million generated tokens, 49 compactions. One 82-character prompt from me, six long ones from the reviewer. At the end, a playable, installable Tetris for Linux and Windows with 104 tests, configurable keys and sound: about 8,600 lines, of which the game itself is under 2,000.

If this were a developer on my team

What I would hand it. Well-bounded features inside an existing structure. Fixes that come with a precise description of what is wrong. Test writing. Repairs from a review. Anything where done can be checked by running something. It did all of that well here, and it cost electricity.

What I would not hand it unsupervised. Anything where done has to mean done: releases, installers, scripts that delete things, anything touching safety or money. Not because the code is bad, but because its report on the code is unreliable, and those are the jobs where an unreliable report is the expensive part.

How I would review it. At the connections. The engine never needed review; everything attached to it always did. Run the artifact instead of reading the report.

How much I would trust it. Its code more than its words. Not once in seven rounds was its own summary accurate enough to act on, while the code underneath was usually good. Writing software and reporting on it are two skills, and the second one is missing.

What it cannot do. Decide what deserves to be perfect. A builder that drafts and a reviewer that audits will ratchet each other toward perfection on whatever they happen to be looking at. Deciding where to stop was the one job neither model did.

The split I would use from here: a local model for implementation and evidence-driven repair; a frontier model for review and release acceptance; a person for scope, and for listening to the music.

The record

Everything above is derived from the agent’s session logs. The evaluation record is in the lab repository: the protocol, the frozen review, every audit and prompt, content-free event logs of both sessions (timestamps, roles, token counts, tool names, no transcript text), and the script that regenerates every number in this post. The runtime setup, down to the llama.cpp build and the server flags, is published separately as localllms. The application itself is not published; it is a Tetris.