SERVICE 03 · VERIFY
Before you trust an AI system, hand it a test where you know the answers.
Evaluation turns our benchmark instrument outward: we generate test documents with computed ground truth, run them through any AI document system — including one you're considering buying — and score what comes back. Vendor claims become measured results.
Why this exists
Procurement runs on claims nobody can check.
When organizations buy AI document software, the evaluation typically rests on references, demonstrations, and an accuracy percentage from the vendor's own materials. There is usually no independent instrument in the process — no test set with known answers, no measured error rate, no way to verify that "99% accurate" means anything at all.
Evaluation supplies the instrument. Because our test documents are generated with the correct answer computed at creation, scoring is against fact, not opinion. The same corpus can be run against multiple candidate products, or against a delivered system before final acceptance, and the results compared on equal footing.
The report
Numbers a review board can put in the file.
Accuracy, both halves
Precision and recall reported separately for every task type, because the two errors are not the same. Missing sensitive data in a released document and over-redacting a public record are different harms, and a single blended score hides which one you're buying.
Cost per correct result
Measured token consumption per task type turns into forecastable economics: at your document volumes, what will each candidate system actually cost per correctly processed document? That figure sits next to the vendor's quote, derived from measurement instead of it.
Behavior on ambiguity
Some test documents are ambiguous by construction, where the right answer is "insufficient information." We report how often a system fabricates a confident answer instead of flagging. No vendor sheet publishes this number; it's often the most decision-relevant one.
Every report states its scope: sample sizes, the document formats tested, and what the results do and do not generalize to. A finding that overstates its coverage is a defect, and we treat it as one.
Independence
We work for the buyer. Full stop.
On evaluation engagements, our client is the party doing the buying or accepting — never the vendor being tested. We don't write vendor proposals, we don't take referral fees from vendors, and we disclose the buyer-side commitment in writing at the start of every engagement.
And because the instrument is model-agnostic, it applies to our own deployment product with no exemption. A client who wants ARTVS execution evaluated with the same corpus and the same scoring gets exactly that. An evaluator who won't sit for its own exam isn't one.
Where it fits
Three moments where measurement changes the decision.
Before a purchase: run the candidate products against the same ground-truth corpus and compare measured results alongside price. Before final acceptance: verify a delivered system performs as proposed, while the contract still has leverage in it. In operation: re-test periodically, because models change underneath deployed products and last year's accuracy number quietly stops being true.
Pricing
Scoped to the decision, not the calendar.
Acceptance Test
A single delivered system, verified against ground truth before final sign-off or payment. One task-type set, one report.
Starting at $2,500Comparative Evaluation
Two or more candidate vendors, scored on the same corpus, for a purchase decision. Every system gets the same test.
Starting at $6,000 (+$1,500 / additional system)Quarterly Assurance
Ongoing re-evaluation of a system already tested. Accuracy and cost-per-correct-result are re-measured as models and pricing shift underneath it.
Starting at $1,800 / quarterQuarterly is the minimum term on the recurring tier, not a suggestion — model pricing and vendor systems both change fast enough that a stale evaluation is worse than none. Final pricing is scoped to task-type count, document volume, and phase (structured text vs. rendered documents) after a short kickoff conversation.
Contact
Bring us the claim. We'll bring the test.
If you're evaluating AI document software, accepting a delivered system, or just want to know whether an accuracy claim survives contact with a known answer key, that's the engagement this was built for.
Email ARTVS