OPEN RESULTS · SEPTEMBER 2026
How 22 AI models perform on everyday business tasks
Each model got the same 216 business tasks, and every answer was checked against a key that existed before any model ran. Claude Opus 5.5 scored highest, at 92.1%. DeepSeek V4.1 Flash scored 86.6% at about a seventh of the cost per correct answer, close enough that the two intervals overlap. The top-scoring model running on a single 16 GB graphics card from 2016, gpt-oss-20b, scored 79.2%, within the margin of error of both DeepSeek models.
Key findings
The most accurate model was not the best value. Claude Opus 5.5 passed 92.1% of tasks (95% interval 87.8%–95.0%) at $0.0159 per correct answer. DeepSeek V4.1 Flash passed 86.6% (81.4%–90.5%) at $0.0023, about a seventh of the cost, and the two intervals overlap. The most expensive model per correct answer, Claude Fable 5.1, passed 90.3% at $0.0345 and was not more accurate than Opus. DeepSeek V4 Pro (0813) passed 85.6% (80.3%–89.7%) at $0.0087: it cost more per correct answer than its smaller sibling Flash and was not more accurate.
A free local model came within the margin of error of the DeepSeek models. gpt-oss-20b, running on one 16 GB NVIDIA Tesla P100 first sold in 2016, passed 79.2% (73.3%–84.0%). Its interval overlaps both DeepSeek models' and Claude Sonnet 5.5's, but not those of Claude Opus 5.5 or Claude Fable 5.1. No other local model's interval overlaps any cloud model's.
Spreadsheets separate the models; reading an invoice's total does not. On spreadsheet lookup, aggregation and reconciliation, Claude Opus 5.5 passed 85.2%, DeepSeek V4.1 Flash 64.8% and gpt-oss-20b 51.9%, while Gemma 4 12B passed 5.6%. Reading an invoice's total barely separates models: 22 of 22 answered every one correctly.
Getting the answer right and following the format are different skills. Every task asked for a bare JSON object, and we score that separately from the answer. Phi-4 passed 62.5% of tasks on content but followed the requested format in 0.0% of its replies: it wrapped them in markdown code fences. DeepSeek-Coder-V2-Lite followed it in 29.4%. Both DeepSeek API models and Claude Opus 5.5 followed it in 100.0% of replies, as did seven of the local models; Claude Sonnet 5.5 followed it in 93.8%. Software that parses replies strictly would reject a correct answer in the wrong wrapper.
Every model scored lower on the hardest tasks. The drop from difficulty 1 to difficulty 9 is beyond the margin of error for the DeepSeek models, Claude Sonnet 5.5, Claude Fable 5.1 and gpt-oss-20b; for Claude Opus 5.5 (97.2% to 84.7%) it is within it. DeepSeek V4.1 Flash fell from 94.4% at difficulty 1 to 72.2% at difficulty 9; gpt-oss-20b from 93.1% to 63.9%. A demo on easy documents overstates what a model will do on your hardest ones.
Squeezing a 27B model to 3 bits did not wreck it. Qwen3.8 27B at 4 bits, split across two 16 GB cards, passed 69.0%. Quantized to 3 bits to fit one card, it passed 69.9% (IQ3_XXS) and 67.6% (IQ3_S): no measurable loss, and half the hardware. All three are within the margin of error of Gemma 4 12B at 68.5%. It was among the strongest local models on robustness and evidence tasks (90.7%) but, like most local models, passed few spreadsheet tasks (7.4%).
A coding-tuned model did no better than its general sibling. Qwen3-Coder 30B-A3B passed 62.0% and its general-purpose sibling Qwen3 30B-A3B 60.2%, a difference well inside the margin of error. Turning on thinking for Gemma 4 26B-A4B moved it from 65.7% to 69.4%, also within noise.
Leaderboard
Overall pass rate
| Model | Runs on | Pass rate | 95% interval | Cost per verified success | Median latency | Tasks scored |
|---|---|---|---|---|---|---|
| 1. Claude Opus 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | Cloud API | 92.1% tied for top | 87.8%–95.0% | $0.0159* | 2.8 s | 216 |
| 2. Claude Fable 5.1 Anthropic · undisclosed · Claude Agent SDK, effort medium | Cloud API | 90.3% tied for top | 85.6%–93.6% | $0.0345* | 2.3 s | 216 |
| 3. Claude Sonnet 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | Cloud API | 88.9% tied for top | 84.0%–92.4% | $0.0074* | 1.8 s | 216 |
| 4. DeepSeek V4.1 Flash DeepSeek · undisclosed · API (deepseek-flash) | Cloud API | 86.6% tied for top | 81.4%–90.5% | $0.0023 | 1.4 s | 216 |
| 5. DeepSeek V4 Pro (0813) DeepSeek · undisclosed · API (deepseek-v4-pro) | Cloud API | 85.6% tied for top | 80.3%–89.7% | $0.0087 | 3.2 s | 216 |
| 6. gpt-oss-20b OpenAI · 21B MoE (3.6B active) · MXFP4 | Local, 1× 16 GB GPU | 79.2% | 73.3%–84.0% | — | 3.5 s | 216 |
| 7. Qwen3.8 27B (3-bit IQ3_XXS) Alibaba · 27B dense · IQ3_XXS (GSQ-RCO, 10.1 GB) | Local, 1× 16 GB GPU | 69.9% | 63.5%–75.6% | — | 7.0 s | 216 |
| 8. Gemma 4 26B-A4B (thinking) Google · 26B MoE (~4B active) · Q4_K_M | Local, 1× 16 GB GPU | 69.4% | 63.0%–75.2% | — | 34.4 s | 216 |
| 9. Qwen3.8 27B Alibaba · 27B dense · Q4_K_M (Unsloth UD) | Local, 2× 16 GB GPU | 69.0% | 62.5%–74.8% | — | 5.3 s | 216 |
| 10. Gemma 4 12B Google · 12B dense · Q4_0 (QAT) | Local, 1× 16 GB GPU | 68.5% | 62.0%–74.3% | — | 3.0 s | 216 |
| 11. Qwen3.8 27B (3-bit IQ3_S) Alibaba · 27B dense · IQ3_S (GSQ-RCO, 11.8 GB) | Local, 1× 16 GB GPU | 67.6% | 61.1%–73.5% | — | 7.3 s | 216 |
| 12. Gemma 4 26B-A4B Google · 26B MoE (~4B active) · Q4_K_M | Local, 1× 16 GB GPU | 65.7% | 59.2%–71.7% | — | 5.7 s | 216 |
| 13. Gemma 4 E4B Google · ~4B effective · Q4_0 | Local, 1× 16 GB GPU | 65.3% | 58.7%–71.3% | — | 1.4 s | 216 |
| 14. Devstral Small 2 24B Mistral AI · 24B dense · Q4_K_M | Local, 2× 16 GB GPU | 63.9% | 57.3%–70.0% | — | 5.0 s | 216 |
| 15. Phi-4 Microsoft · 14B dense · Q4_K_M | Local, 1× 16 GB GPU | 62.5% | 55.9%–68.7% | — | 3.1 s | 216 |
| 16. Qwen3-Coder 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | Local, 1× 16 GB GPU | 62.0% | 55.4%–68.2% | — | 10.3 s | 216 |
| 17. Mistral Small 3.2 24B Mistral AI · 24B dense · Q4_K_M | Local, 2× 16 GB GPU | 61.6% | 54.9%–67.8% | — | 5.3 s | 216 |
| 18. Qwen2.5-Coder 7B Alibaba · 7B dense · Q4_K_M | Local, 1× 16 GB GPU | 61.1% | 54.5%–67.4% | — | 1.8 s | 216 |
| 19. Qwen3 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | Local, 1× 16 GB GPU | 60.2% | 53.5%–66.5% | — | 10.5 s | 216 |
| 20. DeepSeek-Coder-V2-Lite DeepSeek · 16B MoE (2.4B active) · Q4_K_M | Local, 1× 16 GB GPU | 58.8% | 52.1%–65.2% | — | 3.0 s | 216 |
| 21. Qwen3 8B Alibaba · 8B dense · Q6_K | Local, 1× 16 GB GPU | 52.8% | 46.1%–59.3% | — | 2.3 s | 216 |
| 22. Granite 8B Code Instruct IBM · 8B dense · Q4_K_M | Local, 1× 16 GB GPU | 45.8% | 39.3%–52.5% | — | 2.3 s | 216 |
Ranked by pass rate. The 95% interval shows how much the score could move on a different sample of the same tasks: models whose intervals overlap the leader's are marked tied for top. Cost per verified success is API spend divided by correct answers (how it is computed); * marks a list-price estimate from measured tokens. Local models have no per-task API cost. Latency depends heavily on hardware: local models ran on 2016-era 16 GB GPUs, so compare latency within a group, not across.
Notes on specific models
- Qwen3.8 27B (3-bit IQ3_XXS): Served with an 8-bit key cache, a 4-bit value cache and multi-token-prediction speculative decoding, the same on all three Qwen3.8 builds.
- Gemma 4 26B-A4B (thinking): Same weights as the non-thinking Gemma 4 26B-A4B row, with reasoning on (512-token budget).
- Qwen3.8 27B: Served with an 8-bit key cache, a 4-bit value cache and multi-token-prediction speculative decoding, the same on all three Qwen3.8 builds.
- Gemma 4 12B: Served with a repeat penalty of 1.1, our production setting for this model.
- Qwen3.8 27B (3-bit IQ3_S): Served with an 8-bit key cache, a 4-bit value cache and multi-token-prediction speculative decoding, the same on all three Qwen3.8 builds.
- Gemma 4 26B-A4B: Thinking off. Experts offloaded to CPU RAM.
- Phi-4: Served at its native 16K context; the largest spreadsheet tasks do not fit.
- Granite 8B Code Instruct: Served at its native 4K context; long-document and spreadsheet tasks do not fit.
By task family
Where each model is strong or weak
- Document classification: Assign an invoice, email or support ticket to the right category.
- Invoice total: Read the total due from an invoice and return it as a number: a baseline rather than a differentiator. Multi-field extraction is scored under robustness.
- Spreadsheet reasoning: Look up, aggregate and reconcile values across CSV tables, including faulty rows.
- Tool calling: Choose the right function and arguments, including when distractor tools are offered.
- Robustness and evidence: Extract several fields from noisy, conflicting or incomplete documents, and say when the evidence is not enough to answer.
| Model | Document classification | Invoice total | Spreadsheet reasoning | Tool calling | Robustness and evidence |
|---|---|---|---|---|---|
| Claude Opus 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | 93% | 100% | 85% | 100% | 91% |
| Claude Fable 5.1 Anthropic · undisclosed · Claude Agent SDK, effort medium | 91% | 100% | 74% | 100% | 96% |
| Claude Sonnet 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | 85% | 100% | 74% | 100% | 96% |
| DeepSeek V4.1 Flash DeepSeek · undisclosed · API (deepseek-flash) | 89% | 100% | 65% | 100% | 93% |
| DeepSeek V4 Pro (0813) DeepSeek · undisclosed · API (deepseek-v4-pro) | 87% | 100% | 63% | 100% | 93% |
| gpt-oss-20b OpenAI · 21B MoE (3.6B active) · MXFP4 | 85% | 100% | 52% | 100% | 80% |
| Qwen3.8 27B (3-bit IQ3_XXS) Alibaba · 27B dense · IQ3_XXS (GSQ-RCO, 10.1 GB) | 81% | 100% | 7% | 100% | 91% |
| Gemma 4 26B-A4B (thinking) Google · 26B MoE (~4B active) · Q4_K_M | 85% | 100% | 15% | 83% | 89% |
| Qwen3.8 27B Alibaba · 27B dense · Q4_K_M (Unsloth UD) | 81% | 100% | 6% | 100% | 89% |
| Gemma 4 12B Google · 12B dense · Q4_0 (QAT) | 81% | 100% | 6% | 97% | 89% |
| Qwen3.8 27B (3-bit IQ3_S) Alibaba · 27B dense · IQ3_S (GSQ-RCO, 11.8 GB) | 78% | 100% | 4% | 100% | 89% |
| Gemma 4 26B-A4B Google · 26B MoE (~4B active) · Q4_K_M | 76% | 100% | 13% | 97% | 76% |
| Gemma 4 E4B Google · ~4B effective · Q4_0 | 83% | 100% | 0% | 100% | 78% |
| Devstral Small 2 24B Mistral AI · 24B dense · Q4_K_M | 72% | 100% | 0% | 100% | 83% |
| Phi-4 Microsoft · 14B dense · Q4_K_M | 72% | 100% | 0% | 100% | 78% |
| Qwen3-Coder 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | 67% | 100% | 2% | 100% | 80% |
| Mistral Small 3.2 24B Mistral AI · 24B dense · Q4_K_M | 69% | 100% | 2% | 100% | 76% |
| Qwen2.5-Coder 7B Alibaba · 7B dense · Q4_K_M | 67% | 100% | 4% | 100% | 74% |
| Qwen3 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | 72% | 100% | 0% | 100% | 69% |
| DeepSeek-Coder-V2-Lite DeepSeek · 16B MoE (2.4B active) · Q4_K_M | 61% | 100% | 6% | 100% | 69% |
| Qwen3 8B Alibaba · 8B dense · Q6_K | 52% | 100% | 0% | 100% | 59% |
| Granite 8B Code Instruct IBM · 8B dense · Q4_K_M | 31% | 100% | 0% | 100% | 52% |
By difficulty
How scores hold up as tasks get harder
| Model | Level 1 | Level 5 | Level 9 |
|---|---|---|---|
| Claude Opus 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | 97% | 94% | 85% |
| Claude Fable 5.1 Anthropic · undisclosed · Claude Agent SDK, effort medium | 97% | 94% | 79% |
| Claude Sonnet 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | 96% | 94% | 76% |
| DeepSeek V4.1 Flash DeepSeek · undisclosed · API (deepseek-flash) | 94% | 93% | 72% |
| DeepSeek V4 Pro (0813) DeepSeek · undisclosed · API (deepseek-v4-pro) | 94% | 90% | 72% |
| gpt-oss-20b OpenAI · 21B MoE (3.6B active) · MXFP4 | 93% | 81% | 64% |
| Qwen3.8 27B (3-bit IQ3_XXS) Alibaba · 27B dense · IQ3_XXS (GSQ-RCO, 10.1 GB) | 75% | 69% | 65% |
| Gemma 4 26B-A4B (thinking) Google · 26B MoE (~4B active) · Q4_K_M | 79% | 68% | 61% |
| Qwen3.8 27B Alibaba · 27B dense · Q4_K_M (Unsloth UD) | 74% | 68% | 65% |
| Gemma 4 12B Google · 12B dense · Q4_0 (QAT) | 75% | 68% | 62% |
| Qwen3.8 27B (3-bit IQ3_S) Alibaba · 27B dense · IQ3_S (GSQ-RCO, 11.8 GB) | 72% | 67% | 64% |
| Gemma 4 26B-A4B Google · 26B MoE (~4B active) · Q4_K_M | 68% | 69% | 60% |
| Gemma 4 E4B Google · ~4B effective · Q4_0 | 71% | 67% | 58% |
| Devstral Small 2 24B Mistral AI · 24B dense · Q4_K_M | 68% | 61% | 62% |
| Phi-4 Microsoft · 14B dense · Q4_K_M | 69% | 61% | 57% |
| Qwen3-Coder 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | 69% | 58% | 58% |
| Mistral Small 3.2 24B Mistral AI · 24B dense · Q4_K_M | 67% | 60% | 58% |
| Qwen2.5-Coder 7B Alibaba · 7B dense · Q4_K_M | 72% | 57% | 54% |
| Qwen3 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | 69% | 56% | 56% |
| DeepSeek-Coder-V2-Lite DeepSeek · 16B MoE (2.4B active) · Q4_K_M | 71% | 51% | 54% |
| Qwen3 8B Alibaba · 8B dense · Q6_K | 62% | 46% | 50% |
| Granite 8B Code Instruct IBM · 8B dense · Q4_K_M | 53% | 38% | 47% |
Failure modes
How models got it wrong
A wrong-but-well-formed answer is the expensive kind: it looks like a success. Malformed output, truncation, oversized inputs, timeouts and safety-filter blocks are at least visible to the system receiving them.
| Model | Wrong answer | Malformed output | Ran out of output budget | Input exceeded context | Timed out | Blocked by safety filter |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | 11 | 0 | 0 | 0 | 6 | 0 |
| Claude Fable 5.1 Anthropic · undisclosed · Claude Agent SDK, effort medium | 14 | 0 | 0 | 0 | 7 | 0 |
| Claude Sonnet 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | 17 | 0 | 0 | 0 | 7 | 0 |
| DeepSeek V4.1 Flash DeepSeek · undisclosed · API (deepseek-flash) | 18 | 0 | 11 | 0 | 0 | 0 |
| DeepSeek V4 Pro (0813) DeepSeek · undisclosed · API (deepseek-v4-pro) | 20 | 0 | 11 | 0 | 0 | 0 |
| gpt-oss-20b OpenAI · 21B MoE (3.6B active) · MXFP4 | 23 | 10 | 12 | 0 | 0 | 0 |
| Qwen3.8 27B (3-bit IQ3_XXS) Alibaba · 27B dense · IQ3_XXS (GSQ-RCO, 10.1 GB) | 65 | 0 | 0 | 0 | 0 | 0 |
| Gemma 4 26B-A4B (thinking) Google · 26B MoE (~4B active) · Q4_K_M | 60 | 2 | 0 | 0 | 4 | 0 |
| Qwen3.8 27B Alibaba · 27B dense · Q4_K_M (Unsloth UD) | 67 | 0 | 0 | 0 | 0 | 0 |
| Gemma 4 12B Google · 12B dense · Q4_0 (QAT) | 68 | 0 | 0 | 0 | 0 | 0 |
| Qwen3.8 27B (3-bit IQ3_S) Alibaba · 27B dense · IQ3_S (GSQ-RCO, 11.8 GB) | 70 | 0 | 0 | 0 | 0 | 0 |
| Gemma 4 26B-A4B Google · 26B MoE (~4B active) · Q4_K_M | 72 | 2 | 0 | 0 | 0 | 0 |
| Gemma 4 E4B Google · ~4B effective · Q4_0 | 74 | 1 | 0 | 0 | 0 | 0 |
| Devstral Small 2 24B Mistral AI · 24B dense · Q4_K_M | 76 | 2 | 0 | 0 | 0 | 0 |
| Phi-4 Microsoft · 14B dense · Q4_K_M | 69 | 0 | 0 | 12 | 0 | 0 |
| Qwen3-Coder 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | 81 | 0 | 0 | 0 | 1 | 0 |
| Mistral Small 3.2 24B Mistral AI · 24B dense · Q4_K_M | 79 | 4 | 0 | 0 | 0 | 0 |
| Qwen2.5-Coder 7B Alibaba · 7B dense · Q4_K_M | 82 | 2 | 0 | 0 | 0 | 0 |
| Qwen3 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | 86 | 0 | 0 | 0 | 0 | 0 |
| DeepSeek-Coder-V2-Lite DeepSeek · 16B MoE (2.4B active) · Q4_K_M | 77 | 0 | 0 | 0 | 12 | 0 |
| Qwen3 8B Alibaba · 8B dense · Q6_K | 102 | 0 | 0 | 0 | 0 | 0 |
| Granite 8B Code Instruct IBM · 8B dense · Q4_K_M | 91 | 14 | 0 | 12 | 0 | 0 |
Format adherence
Did the model reply in the format it was asked for?
Measured separately from the pass rate. Every task's instructions asked for the bare JSON object: no markdown code fence and no text around it. The pass rate scores the answer inside the reply; this table scores the wrapper. Software that parses replies strictly will reject a correct answer in the wrong format, so a low figure here means extra handling code or a stricter prompt before deployment.
| Model | Followed the format | Replies |
|---|---|---|
| Claude Opus 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | 100.0% | 210 of 210 |
| DeepSeek V4.1 Flash DeepSeek · undisclosed · API (deepseek-flash) | 100.0% | 205 of 205 |
| DeepSeek V4 Pro (0813) DeepSeek · undisclosed · API (deepseek-v4-pro) | 100.0% | 205 of 205 |
| Qwen3.8 27B (3-bit IQ3_XXS) Alibaba · 27B dense · IQ3_XXS (GSQ-RCO, 10.1 GB) | 100.0% | 216 of 216 |
| Qwen3.8 27B Alibaba · 27B dense · Q4_K_M (Unsloth UD) | 100.0% | 216 of 216 |
| Qwen3.8 27B (3-bit IQ3_S) Alibaba · 27B dense · IQ3_S (GSQ-RCO, 11.8 GB) | 100.0% | 216 of 216 |
| Gemma 4 26B-A4B Google · 26B MoE (~4B active) · Q4_K_M | 100.0% | 216 of 216 |
| Gemma 4 E4B Google · ~4B effective · Q4_0 | 100.0% | 216 of 216 |
| Qwen3 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | 100.0% | 216 of 216 |
| Qwen3 8B Alibaba · 8B dense · Q6_K | 100.0% | 216 of 216 |
| Qwen2.5-Coder 7B Alibaba · 7B dense · Q4_K_M | 99.1% | 214 of 216 |
| gpt-oss-20b OpenAI · 21B MoE (3.6B active) · MXFP4 | 99.0% | 201 of 203 |
| Qwen3-Coder 30B-A3B Alibaba · 30B MoE (3B active) · Q4_K_M | 98.6% | 212 of 215 |
| Gemma 4 26B-A4B (thinking) Google · 26B MoE (~4B active) · Q4_K_M | 96.2% | 204 of 212 |
| Claude Fable 5.1 Anthropic · undisclosed · Claude Agent SDK, effort medium | 95.7% | 200 of 209 |
| Claude Sonnet 5.5 Anthropic · undisclosed · Claude Agent SDK, effort medium | 93.8% | 196 of 209 |
| Gemma 4 12B Google · 12B dense · Q4_0 (QAT) | 93.5% | 202 of 216 |
| Granite 8B Code Instruct IBM · 8B dense · Q4_K_M | 87.3% | 178 of 204 |
| Devstral Small 2 24B Mistral AI · 24B dense · Q4_K_M | 74.5% | 161 of 216 |
| Mistral Small 3.2 24B Mistral AI · 24B dense · Q4_K_M | 71.3% | 154 of 216 |
| DeepSeek-Coder-V2-Lite DeepSeek · 16B MoE (2.4B active) · Q4_K_M | 29.4% | 60 of 204 |
| Phi-4 Microsoft · 14B dense · Q4_K_M | 0.0% | 0 of 204 |
Openness
What we publish, and what we keep private
Published here: every model's overall and per-family scores, difficulty curves, failure counts, cost and latency, the exact model versions and settings, and a fingerprint of the task set (f8636eff133b0699) that lets us prove later results ran on the same tasks.
Kept private: the task texts, the generators that produce them, and the grading code. A public test set ends up in training data, and a model that has seen the questions stops being measured by them. Keeping the tasks private is what keeps these numbers meaningful the next time we run them.
Full detail: methodology.
Data
Download and cite
Per-model results: artvs-results-2026-09.csv · per task type: artvs-results-2026-09-task-types.csv · results.json. Licensed CC BY 4.0: reuse them freely with attribution.
ARTVS (2026). Open results: AI models on business tasks, September 2026. https://artvs.ai/results
Questions
Frequently asked
Which AI model is most accurate on business document tasks?
In the ARTVS September 2026 run, Claude Opus 5.5 passed 92.1% of 216 business tasks (95% interval 87.8%–95.0%), the highest of the 22 models tested. Claude Fable 5.1 passed 90.3%, Claude Sonnet 5.5 88.9% and DeepSeek V4.1 Flash 86.6%; all four intervals overlap, so they are statistically close. The best local model, gpt-oss-20b, passed 79.2%. Every answer was checked mechanically against a key that existed before the models ran.
Can a local LLM on an older GPU handle business document work?
For much of it, yes. gpt-oss-20b on a single 16 GB Tesla P100 from 2016 passed 79.2% of tasks, including every invoice-total and tool-calling task. Spreadsheet reasoning is where local models fall short: gpt-oss-20b passed 51.9% of those, and most other local models passed under 10%. These numbers are from the September 2026 run; gpt-oss-20b has since been retired from our own setup.
How much does a correct answer cost from a cloud AI model?
Measured as API spend divided by correct answers, DeepSeek V4.1 Flash cost $0.0023 per verified success and DeepSeek V4 Pro (0813) $0.0087, including the cost of calls cut off at the output budget before answering. For Claude models it is an estimate at Anthropic's API list price: Sonnet 5.5 $0.0074, Opus 5.5 $0.0159 and Fable 5.1 $0.0345. Local models have no per-task API cost; their cost is hardware and power.
Does 3-bit quantization ruin a 27B language model?
Not on these tasks. Qwen3.8 27B at 3 bits (IQ3_XXS and IQ3_S builds, each fitting one 16 GB GPU) passed 69.9% and 67.6% of 216 business tasks, against 69.0% for the 4-bit build that needs two GPUs: no measurable loss. All three are within the margin of error of the 12B-parameter Gemma 4 12B at 68.5%. The 3-bit build was slower: a median of 7.0 s per task against 3.0 s.
Do language models follow JSON output-format instructions?
Not all of them. Every ARTVS task asked for a bare JSON object with no markdown code fence and no text around it. Ten of the 22 models followed that format in every reply, including both DeepSeek API models and Claude Opus 5.5; gpt-oss-20b followed it in 99.0%. Phi-4 followed it in 0.0%: it wrapped its replies in code fences, even when the answer inside was right. We score the answer and the format separately; the format figures are in the Format adherence table.
Are coding-specialized models better at business tasks?
Not measurably. Qwen3-Coder 30B-A3B passed 62.0% and the general Qwen3 30B-A3B 60.2% on the same 216 tasks, a difference inside the 95% interval.
Related
Want this run on your own documents?
These numbers come from our standard task set. An evaluation engagement runs the same apparatus on your document types, your volumes and your shortlist of models, and reports cost per verified success for each.
Read about Evaluate →