Writing —
First place, 400 spreadsheets
How our team won the YLookup × Encode AI Hackathon with an agent that edits real workbooks — and why we spent the first hours measuring how the model reads a spreadsheet before writing anything that changes one.
The task
SpreadsheetBench is 400 real spreadsheet problems scraped from Excel forums — the kind of thing a person actually got stuck on and posted about. You get a workbook and an instruction in plain English. You return the workbook with the answer filled in. The evaluator grades only the answer cells, and it grades them all-or-nothing: one wrong cell and the whole task scores zero.
The judging made it harder than a normal leaderboard. They ran our pipeline themselves, in our own Docker container, on tasks we had never seen. Model IDs fixed in code, temperature 0, keys from the environment, nothing else mounted. Anything that looked like a golden value reaching a prompt was an instant disqualification. So there was no way to quietly overfit the 400 — whatever we built had to actually generalise.
Why an agent, and not one big answer
The obvious pipeline is one model call per task: show it the sheet, ask for the final values, write them in. That falls apart the moment a task is sheet-level — filter these rows, sort by this column, carry the formatting across, fill down three thousand cells. Asking a model to emit every one of those values as JSON either runs out of room or comes back malformed.
So we stopped asking for values and started asking for operations. Our entrypoint gives the model an instruction, a compact summary of the workbook, the target answer range, and the source cells that matter. Then it works in a loop of up to twenty turns, picking one structured action at a time: inspect the data, run Python against the workbook with openpyxl, run a shell command, or recalculate formulas through LibreOffice. After every edit the harness hands back the current contents of the answer cells. The model gets to look at its own work and fix it.
That last part matters more than it sounds. A large share of the tasks we pass are second attempts — the first script ran, produced something visibly wrong, and the model corrected it. A single-shot pipeline never gets that chance.
Measure the reading before you build the writing
The tempting move at a hackathon is to start building at hour zero. We spent the first block of time on something less exciting: a representation study. Twenty-six configurations, a fixed stratified 100-task dev split, scored with the shipped evaluator and compared with paired bootstraps over task IDs. The question was narrow — how should this model see a workbook, and how much should it be allowed to think?
Three settings came straight out of it, and all three were counterintuitive.
Thinking level: medium, not maximum
The thinking sweep gave 43% with reasoning off, 65% on low, 64% on medium, 64% on
adaptive. Turning thinking off costs 22 points — a real, wide effect. But low, medium
and adaptive are within a point of each other, so past a floor, more deliberation buys
nothing here. The cookbook's recommended renderer, xhigh, scored 52% at the
8192-token cap and 64% at 16k, for 2.5× the latency. We shipped medium: the same
accuracy as low, with more headroom for the agent's longer edit turns.
The failures were truncation, not reasoning
The finding that changed the most was mundane. Every one of xhigh's 41
extra failures at the 8192 cap was a reply cut off while still thinking. The
Qwen3.8 renderers prefill the think tag, so a truncated completion isn't a bad answer —
it's reasoning text with no answer attached, handed to a JSON parser that never had a
chance. Raise the cap to 32768 and no thinking level truncated at all; xhigh climbed to 65%.
We had been reading those as model errors. They were a budget bug. The harness now detects the cut-off from the stop reason and re-samples one thinking level lower — medium, then low, then off — instead of handing half a thought to the parser.
Show formulas, not just values
On a single-shot code loop, a view that showed formulas alongside values beat a values-only digest by 13 points. A spreadsheet's formulas are its intent; strip them and the model is reverse-engineering the logic from outputs. The agent we submitted keeps its compact digest but exposes formulas and values on demand through its inspect tools, which gets the same information in without paying for it in every prompt.
The fine-tune that didn't make it
We also ran a LoRA experiment on our own successful traces: 1,722 training action examples and 354 validation examples, reconstructed from agent runs the evaluator had marked correct. Rank 16, learning rate 1e-4, batch size 4, two epochs, 862 optimizer steps. Golden workbook values never entered a prompt or a training example — they were only ever used to decide which traces were worth learning from.
It did not beat the base model on measured spreadsheet accuracy. So we shipped the base model and left the checkpoint out of the pipeline entirely. The experiment is still in the repo, with its log, because a negative result you measured is worth more than a positive one you assumed.
Where it landed
The shipped evaluator on all 400 tasks, nothing hand-picked:
- 88.25%
- Pass rate
- 82.31%
- Cell accuracy
- 88.0%
- Cell-level tasks
- 88.8%
- Sheet-level tasks
400 graded, none missing, no errors. The gap between 88.25% pass rate and 82.31% cell accuracy is the all-or-nothing rule showing its teeth: a task can be almost entirely right and still score zero, and a handful of very large answer ranges drag the cell number down without changing many task outcomes.
What still breaks
The remaining failures are not exotic. The model misreads a business rule — an "including" that should have been "excluding". It gets a boundary off by one row. It matches case-sensitively when the forum poster meant otherwise, or writes the right number with the wrong formatting. Almost none of it is a failure to operate a spreadsheet; it's a failure to pin down what an ambiguous English sentence actually asked for. That's the part I'd work on next.
The full pipeline, traces, and the representation study are in the submission repo.