Extraction Arena

I recorded a walkthrough of the harness behind this note, and the code is open source: martin-cousseau/extraction-arena.

I keep the model answers frozen and change only the scoring rule, so I can see whether I have a model problem or a definition problem. Later I add a fourth pipeline, LlamaParse through LlamaExtract, and read that run the same way.

The business question does not move. Would you accept a high extraction score if the one procedure a crew would follow still scores zero?

For product owners deciding go / no-go, subject-matter experts deciding whether order and wording are allowed to move on a rescue step, or engineers who have to turn that decision into weights. If you ship document extraction into operations, this is the meeting you skip at your cost. The average will look fine. The field a crew would actually follow might not.

Tesla publishes official rescue information for the Cybertruck. A rescue sheet (ISO 17840-1) is the quick page at the scene. An Emergency Response Guide (ISO 17840-3) is the longer procedure set used to train and operate. The four pages the models saw mix diagrams, pictograms, numbered procedures, and prohibitions in capital letters. A wrong cut zone or a skipped disable step is an operational failure, not just a formatting issue. I selected and scored the following fields: cut zones, high-voltage disable, fire, submersion, and tow.

The first experiment ran in four steps.

1. Each page is rendered as an image (the models see what a person sees on paper).
2. Every model gets the same empty form; the correct answers are never in the prompt.
3. GLM-5V-Turbo, GPT-5.4 Mini, and Grok 4.5 each return their own JSON; I map those answers onto one shared form.
4. Then I score field by field, because the business risk is not the same on manufacturer, on a prohibition, and on a five-step procedure.

I am not asking the models again. I am changing how a correct answer is defined for one field. I zoomed in on the five-step procedure for a submerged Cybertruck. Product owners pick the scoring rule that matches the risk. Subject-matter experts say whether order and wording are allowed to move.

How close must the wording be? Exact or partial. If too strict, a useful extraction looks like a failure. If too loose, a dangerous paraphrase looks like a pass. Example: exact match fails “Wear PPE” against the gold “Wear appropriate PPE for water rescue.” Partial match still scores the overlapping words.

Does order count? Sequence or set. A rescue procedure is a sequence of actions. Sequence compares step 1 to gold step 1. Set treats the list as a bag. Example: one model opened at gold step 4. Under sequence that is a miss in position 1. Under set, the step is merely present, and a crew could skip PPE.

The question, which mistake to surface first, is a product decision, not a scoring one.

Precision: of everything the model wrote, how much was on the sheet — the lens for a prohibition.

Recall: of everything the sheet required, how much the model captured — the lens for a procedure.

And F1 holds both costs.

None of the three vision models returned the five-step order.

In the illustration below a filled cell is a step the model actually wrote, a fused cell is two gold steps crushed into one line, an empty cell is a skipped step.

• One model opened at gold step 4.
• The other two started on PPE, fused remove-from-water with the high-voltage disable, then stopped.

All three saw the submersion paragraph. None produced a five-step ordered list.

I then read those frozen answers under three scenarios. Only the scoring rule changes.

• Exact wording, order required: precision 0, recall 0, F1 0.
• Flexible wording, order still required: 44, 25, 33.
• Flexible wording, order ignored: 81, 47, 58.

All of these results are true. They just answer different questions.

I did not improve the models, but I changed the question. For this field a kinder pairing rule increases the score and rewards an extracted procedure no crew should follow.

A well-designed metric should let you trust the headline. That is the goal. A composed score only earns that trust if its weights match the risk of the work.

But in this demo they do not: manufacturer, vehicle type and other easy fields pull the extraction score up to 93, while the water-rescue procedure scores zero under the strict rule.

The demo mix is weight 0.25 on the exact-field rate, weight 0.35 on average partial credit, and weight 0.40 on average F1, taken across every field on the sheet.

With this mix, GLM-5V-Turbo lands at 93. GPT-5.4 Mini at 90. Grok 4.5 at 90.

A composed score is the right tool to record a run and follow a trend. It is the wrong tool for a ship / no-ship decision when those weights have not been set to the operational risk. Use it to follow a trend across runs or tune the weights until important fields can move the headline.

People ask how to “choose weights.” Weights are a claim about which mistake you are willing to average away. Here is the method I use:

1. Write the feared failure in one sentence, with a subject. “The extractor skips PPE on a submerged-vehicle procedure.” If you cannot write this, you cannot set weights and you are just collecting scores.
2. Tag every field Gate, Credit, or Noise. Gate: a wrong answer means do not ship. Often a veto — exact match. Credit: partial overlap is allowed and useful (a warning list, a paraphrasable instruction). Noise: easy fields that inflate the average (manufacturer, model year, door count). For example, on the Cybertruck sheet: submersion ordered-steps is Gate. Fire prohibition is Gate. Manufacturer is Noise. Occupant-access methods are Credit.
3. Run the veto test. Set the gate field to zero. Leave every other field at the score you actually observed. Recompute the headline. If that headline still looks like a ship (80, 90, 93) the weights are wrong and they have not been set to the risk. This is the test this demo failed: procedure at 0, composed extraction score at 93.
4. Retune until a zero on the gate field pulls the headline below the written line. You can raise the gate field’s weight in the average or stop averaging it. For example, across the 34 fields of this demo the procedure has a 5% say. If the procedure gets 40% of the headline, the same run lands near 58 and it no longer looks shippable. The models did not get worse.
5. Write the line as a conjunction. “We ship if composed ≥ T and the gate field ≥ G.” Product owners pick the conjunction. Subject-matter experts pick G. Engineers encode both. If your current mix cannot fail the veto test you should change the mix (rather than argue about the model).

The first experiment was three vision pipelines on page images. After that I added LlamaExtract, through LlamaParse. LlamaIndex ships four extract modes: turbo, cost-effective, agentic, and agentic plus. Same rescue-sheet schema. Same scorer.

With the same mix that produced 93, 90, 90 on vision I get the results below. 87% is the best of the four, but it is lower than every previous vision headline. At least the ranking follows my instinct and picks agentic plus. A ranking against vision still picks GLM-5V-Turbo at 93. The strict procedure still fails on every mode.

Seven runs. Vision: 93, 90, 90. LlamaParse: 79, 83, 84, 87.

Agentic plus emerged as the best of the four LlamaParse pipelines, yet it remains below every vision headline. The exact sequence mode for the water-rescue procedure returned 0 on every single run. While the Extraction Score showed movement, the deployment gate did not.

In conclusion, frame the release criteria as a strict conjunction. Ship only if the headline clears threshold T and the gate field clears threshold G. Display the worst-performing field first on the dashboard. When an 87 baseline defines the trend and the rescue procedure holds the deciding veto, the directive is clear: do not ship.

Three vision models, one gold rescue sheet, and a score that moves when the question changes. The same argument, recorded. There is also a page for the walkthrough.

VIDEO

Play video (opens YouTube in a new tab)

// the player loads from YouTube only when you press play

Extraction Arena is a public TypeScript repository, martin-cousseau/extraction-arena, on the main branch: a document extraction evaluation framework. Two Node projects: a backend that turns pages into images, and a frontend that scores fields against gold answers. The note is the argument; the repository is the instrument.

git clone https://github.com/martin-cousseau/extraction-arena.git

The same piece also ran on X, LinkedIn and TikTok.