AUTOMATED REASONING & TASK VERIFICATION SYSTEM · NORTH CAROLINA
Accuracy is a measurement, not a marketing claim.
ARTVS measures how well AI actually performs on document work — extraction, invoice processing, triage, PII identification — by testing against ground truth that is computed, not assumed. Every number we publish carries its sample size and its limits.
Start a conversationThe problem
Nobody checks the accuracy claims.
AI vendors publish accuracy percentages that buyers have no way to verify. Pricing tables quote cost per token, but the number that matters — cost per correctly processed document — depends on error rates nobody measures. Organizations pick models, buy software, and automate workflows on claims that were never tested against a known right answer.
ARTVS exists to be the instrument. We generate synthetic test documents where the correct answer is known at the moment of creation, run them through the system in question, and score the results. What comes back is not an impression. It is a table.
| Task type | Accuracy | Cost / correct result | Guess rate on ambiguous input | n |
|---|---|---|---|---|
| Invoice line matching | measured % | measured $ | measured % | ≥50 |
| PII identification (prose) | precision / recall | measured $ | measured % | ≥50 |
| Document triage | measured % | measured $ | measured % | ≥50 |
Method
Ground truth first. Everything follows from that.
Synthetic documents, known answers
Test documents are generated from templates with planted variables, so the correct answer exists before any model sees the task.
Same tasks, every system
Identical inputs and prompts across every model or product under test. The comparison is fair because the apparatus is fixed.
Against the known answer
Scorers are themselves tested with deliberately wrong answers before their scores are trusted. An unproven scorer reports nothing.
With sample sizes and limits
Every figure ships with its n, its scope, and its known limitations. Precision and recall are reported separately, always.
Services
Three ways in, one methodology underneath.
Assess
A low-effort scan of your current workflows, software spend, and automation opportunities. No new software installed, no infrastructure required.
Read about assessment → 02 · OPERATEDeploy
A local-first AI execution stack with tiered model routing, per-call cost accounting, and an audit trail. Your documents never have to leave your network.
Read about deployment → 03 · VERIFYEvaluate
Independent, ground-truth testing of any AI document system — including one you are about to buy. Built for procurement evaluation and acceptance testing.
Read about evaluation →Operating principles
The rules we hold ourselves to.
Claims attach at the point of claim. We investigate widely and tag every finding by confidence. Only measured results get stated as capability.
Scorers are guilty until proven innocent. Every scoring mechanism is fed deliberately wrong answers before its output is trusted. If it cannot catch a planted error, it does not report.
Limitations are published, not buried. Sample sizes, coverage gaps, and known failure modes appear in the report, in plain language, next to the results they qualify.
A model that guesses is worse than a model that asks. We measure how often a system fabricates an answer on ambiguous input instead of flagging it. That number is a headline metric, not a footnote.
Buyer-side only. On evaluation engagements we work for the party doing the buying. We do not write vendor proposals, and we will evaluate our own system with the same instrument.
Open results · September 2026
22 AI models, 216 business tasks, every answer checked
Claude Opus 5.5 scored highest, at 92.1%. DeepSeek V4.1 Flash scored 86.6% at $0.0023 per correct answer. gpt-oss-20b, running on a single 2016-era GPU, scored 79.2%. Every answer checked against a key that existed before any model ran.
See all results →Questions
What people ask before the first conversation.
How is this different from a vendor's own benchmark? A vendor benchmarking its own product has no reason to publish an unflattering number. ARTVS tests against ground truth computed before any model sees the task, using the same fixed apparatus across every system compared — the accuracy number comes from the test, not from the party being tested.
What does "ground truth" mean here, specifically? Test documents are generated from templates with planted variables, so the correct answer exists before any model runs. That's different from grading against a human's best guess after the fact — the answer key is fixed at creation time.
Would ARTVS ever report that ARTVS itself isn't the right fit? Per our operating principles, we work buyer-side only on evaluation engagements and evaluate our own system with the same instrument used on everyone else's. A claim that isn't measured doesn't get stated as capability — including our own.
What if the scoring mechanism itself is wrong? Every scorer is fed deliberately wrong answers before its output is trusted. If it can't catch a planted error, it doesn't report a score — an unproven scorer produces nothing rather than a number you can't trust.
Is Evaluate only for a system I haven't bought yet? It's built for procurement evaluation and acceptance testing — the two points where an independent, ground-truth number matters most before you're committed.
Contact
Start with a conversation.
Whether you are sizing up automation, deploying it, or verifying something you are about to buy, the first step is the same: a short conversation about what you handle, in what volume, and what a correct result looks like.
Email ARTVS