How the ARTVS open results were measured

Each model received the same 216 tasks: 12 task types, each at three difficulty levels (1, 5 and 9 on a 1-to-10 scale), six tasks per type and level. The task types cover everyday document work that businesses hand to AI: classifying invoices, emails and support tickets; reading an invoice's total; looking up, aggregating and reconciling spreadsheet data; choosing the right tool and arguments when distractor tools are on offer; and extracting several fields from noisy, conflicting or incomplete documents, including recognizing when the evidence is not enough to answer.

Difficulty is a real lever, not a label. Harder levels add more rows and planted faults to spreadsheets, more decoy values and locale-formatted numbers to documents, and conflicting sources that must be resolved by date. A model that ignored difficulty would show a flat line across levels; most do not.

Every task is generated from planted values, so its correct answer is computed at the moment the task is created. No human grades anything after the fact, and no model grades another model. Answers are checked mechanically against the key. Every extracted field must equal the key's value, in the format the task asks for (for example, ISO dates and amounts in integer cents); the scorer does not reformat an answer to rescue it. Category labels must equal the key's label; about one task in six ignores capitalization, and a true/false field also accepts yes/no. A tool call must name the right tool, include every required argument and set each argument correctly. Spreadsheet answers must equal the computed values.

An answer that is well-formed but wrong counts as a failure, the same as no answer at all. That is the failure mode that costs a business the most, because it looks like success.

Answer and format are scored separately. The pass rate measures the answer: the scorer reads the JSON object in the reply even when it is wrapped in a markdown code fence or surrounded by text. Whether the reply came in the format the instructions asked for is a second, separate measurement, reported as format adherence. It applies only where a task's instructions state a format; in this run every task's did, asking for the bare JSON object with no code fence and no text around it. A code block in another language (a program instead of an answer) is never searched for an answer.

Counted as a failure: a wrong answer; a reply with no JSON object of the expected shape; output cut off by the model's output budget; an input longer than the model's context window as served; and no answer within the time limit (240 seconds; 230 seconds for Claude models, whose requests pass through one more hop). Output that the model server itself could not parse is counted as malformed output. A task that the provider's safety filter blocked counts as a failure too: these are ordinary business documents, and a blocked request gives the business no answer.

Excluded: faults on our side that say nothing about the model, such as the serving process not answering or a rate limit. The number excluded for each model is in the downloadable CSV (infra_excluded).

Retries: if a model returned an empty reply (not counting replies cut off by the output budget), the request was sent once more. A wrong answer was never retried.

Intervals: every pass rate carries a 95% Wilson score interval. With 216 tasks, two models a few points apart are often statistically indistinguishable; the leaderboard marks every model whose interval overlaps the leader's.

Local models ran through llama.cpp's server on NVIDIA Tesla P100 GPUs (16 GB each, first sold in 2016), one card per model except the models marked "2× 16 GB", which were split across two cards. Weights are the quantized builds listed below; a quantized model is usually slightly less accurate than its full-precision original. Mixture-of-experts models with more weights than fit in 16 GB kept some experts in system RAM. Each model ran with its production serving settings on this hardware, which are not identical across models: most use an 8-bit KV cache, and any setting that plausibly affects scores is listed in the Notes column. Reasoning was turned off where the model allows it, except for the model marked "thinking"; gpt-oss-20b always reasons.

DeepSeek models ran through DeepSeek's own API. Their first run hit a context limit in our own gateway configuration that refused the largest spreadsheet tasks before DeepSeek saw them; we fixed the limit and re-ran every DeepSeek task, and only the re-run is published.

Anthropic's Claude models ran through the Claude Agent SDK (Claude Code in non-interactive mode, with every tool disabled and our own system prompt) at effort medium; these models think by default, and they do not accept a temperature setting.

Every model got the same instructions for a given task, with no per-model prompt tuning. Output budgets differ by model, as the table shows: a small budget can cut a reasoning model off before it answers, which is counted as a failure and shown separately under failure modes.

Cost per verified success is API spend divided by the number of correct answers (definition). For DeepSeek it is the spend recorded per call, priced at the per-token rates we configured from DeepSeek's published peak-hour prices, plus an estimate for calls cut off at the output budget, which our proxy records with no spend: each is priced at the full output budget plus the average input of its task type. Those long, unanswered calls are a large share of the total, which is why a model that runs out of budget costs more per correct answer. For Anthropic models it is an estimate: measured input and output tokens multiplied by Anthropic's API list price. The SDK adds roughly 400 tokens of framing to each request, so this estimate runs slightly high. Local models have no per-task API cost; their cost is hardware and power, which depends on your setup.

The documents are synthetic and in English. They are built to behave like real business documents, but your documents are not ours, and benchmark scores do not travel automatically to a new task shape. Each model attempted each task once, so small differences are within run-to-run noise. The invoice-total tasks barely separate models: in this run 22 of 22 models answered every one of them correctly. Latency reflects our hardware: local models on decade-old GPUs are slower than they would be on current cards, so compare latency only within the local group or within the cloud group. Models and prices change; these results are dated and will be re-run on the same tasks, which is why the task set carries a fingerprint (f8636eff133b0699).