Writing —

First place, 400 spreadsheets

How our team won the YLookup × Encode AI Hackathon with an agent that edits real workbooks — and why we spent the first hours measuring how the model reads a spreadsheet before writing anything that changes one.

The submitted repo Back to work

The task

SpreadsheetBench is 400 real spreadsheet problems scraped from Excel forums — the kind of thing a person actually got stuck on and posted about. You get a workbook and an instruction in plain English. You return the workbook with the answer filled in. The evaluator grades only the answer cells, and it grades them all-or-nothing: one wrong cell and the whole task scores zero.

The judging made it harder than a normal leaderboard. They ran our pipeline themselves, in our own Docker container, on tasks we had never seen. Model IDs fixed in code, temperature 0, keys from the environment, nothing else mounted. Anything that looked like a golden value reaching a prompt was an instant disqualification. So there was no way to quietly overfit the 400 — whatever we built had to actually generalise.

Why an agent, and not one big answer

The obvious pipeline is one model call per task: show it the sheet, ask for the final values, write them in. That falls apart the moment a task is sheet-level — filter these rows, sort by this column, carry the formatting across, fill down three thousand cells. Asking a model to emit every one of those values as JSON either runs out of room or comes back malformed.

So we stopped asking for values and started asking for operations. Our entrypoint gives the model an instruction, a compact summary of the workbook, the target answer range, and the source cells that matter. Then it works in a loop of up to twenty turns, picking one structured action at a time: inspect the data, run Python against the workbook with openpyxl, run a shell command, or recalculate formulas through LibreOffice. After every edit the harness hands back the current contents of the answer cells. The model gets to look at its own work and fix it.

That last part matters more than it sounds. A large share of the tasks we pass are second attempts — the first script ran, produced something visibly wrong, and the model corrected it. A single-shot pipeline never gets that chance.

Measure the reading before you build the writing

The tempting move at a hackathon is to start building at hour zero. We spent the first block of time on something less exciting: a representation study. Twenty-six configurations, a fixed stratified 100-task dev split, scored with the shipped evaluator and compared with paired bootstraps over task IDs. The question was narrow — how should this model see a workbook, and how much should it be allowed to think?

Three settings came straight out of it, and all three were counterintuitive.

Thinking level: medium, not maximum

The thinking sweep gave 43% with reasoning off, 65% on low, 64% on medium, 64% on adaptive. Turning thinking off costs 22 points — a real, wide effect. But low, medium and adaptive are within a point of each other, so past a floor, more deliberation buys nothing here. The cookbook's recommended renderer, xhigh, scored 52% at the 8192-token cap and 64% at 16k, for 2.5× the latency. We shipped medium: the same accuracy as low, with more headroom for the agent's longer edit turns.

The failures were truncation, not reasoning

The finding that changed the most was mundane. Every one of xhigh's 41 extra failures at the 8192 cap was a reply cut off while still thinking. The Qwen3.8 renderers prefill the think tag, so a truncated completion isn't a bad answer — it's reasoning text with no answer attached, handed to a JSON parser that never had a chance. Raise the cap to 32768 and no thinking level truncated at all; xhigh climbed to 65%.

We had been reading those as model errors. They were a budget bug. The harness now detects the cut-off from the stop reason and re-samples one thinking level lower — medium, then low, then off — instead of handing half a thought to the parser.

Show formulas, not just values

On a single-shot code loop, a view that showed formulas alongside values beat a values-only digest by 13 points. A spreadsheet's formulas are its intent; strip them and the model is reverse-engineering the logic from outputs. The agent we submitted keeps its compact digest but exposes formulas and values on demand through its inspect tools, which gets the same information in without paying for it in every prompt.

The fine-tune that didn't make it

We also ran a LoRA experiment on our own successful traces: 1,722 training action examples and 354 validation examples, reconstructed from agent runs the evaluator had marked correct. Rank 16, learning rate 1e-4, batch size 4, two epochs, 862 optimizer steps. Golden workbook values never entered a prompt or a training example — they were only ever used to decide which traces were worth learning from.

It did not beat the base model on measured spreadsheet accuracy. So we shipped the base model and left the checkpoint out of the pipeline entirely. The experiment is still in the repo, with its log, because a negative result you measured is worth more than a positive one you assumed.

Where it landed

The shipped evaluator on all 400 tasks, nothing hand-picked:

88.25%
Pass rate
82.31%
Cell accuracy
88.0%
Cell-level tasks
88.8%
Sheet-level tasks

400 graded, none missing, no errors. The gap between 88.25% pass rate and 82.31% cell accuracy is the all-or-nothing rule showing its teeth: a task can be almost entirely right and still score zero, and a handful of very large answer ranges drag the cell number down without changing many task outcomes.

What still breaks

The remaining failures are not exotic. The model misreads a business rule — an "including" that should have been "excluding". It gets a boundary off by one row. It matches case-sensitively when the forum poster meant otherwise, or writes the right number with the wrong formatting. Almost none of it is a failure to operate a spreadsheet; it's a failure to pin down what an ambiguous English sentence actually asked for. That's the part I'd work on next.

The full pipeline, traces, and the representation study are in the submission repo.