NOTES
Definitions and findings, published as we measure them.
Reference material from the methodology behind Assess, Deploy, and Evaluate. Each entry answers one question completely and states what it doesn't tell you.
Definitions
Why benchmark scores don't travel: what a leaderboard number actually tells you about your workload
A vendor's benchmark page says their model is best in class. Point the same model at your own workload and it stumbles on things that look basic: your document layouts, your prompt templates, your edge cases. That gap between the headline number and your experience is not a conspiracy, and it is not incompetence. It's structural. Published scores are measured on someone else's test, in someone else's harness, against someone else's idea of what "good" means. This post is about why that gap exists, and how to make model decisions that don't require taking a leaderboard on faith.
Read about Why benchmark scores don't travel: what a leaderboard number actually tells you about your workload → METHODOLOGYCost per verified success
Why raw token cost doesn't measure what a business actually pays for, and how to compute the number that does.
Read about cost per verified success → METHODOLOGYGuess rate on ambiguous input
How often a system fabricates a confident answer instead of flagging uncertainty — and why that number matters more than accuracy alone.
Read about guess rate on ambiguous input →Articles
Shadow AI
Shadow AI is not a policy problem — it is a visibility problem, and prohibition only drives usage underground.
Read about shadow AI → AI OPERATIONSLeashing AI Spend
AI spending that has no owner, no review date, and no attached job is not a strategy — it is a leak with a dashboard.
Read about leashing AI spend → AI OPERATIONSShadow AI Detection
How to find the AI tools your team is already using, without surveillance and without turning the office against you.
Read about shadow AI detection → AI OPERATIONSLocal vs. Cloud
What actually changes when AI inference runs on your own machine instead of a vendor's — and what stays exactly the same.
Read about local vs. cloud AI → AI OPERATIONSHow to report shadow AI risk to the board when you cannot measure it
A coverage table for proxy, CASB, and endpoint tiers, and how to report a known gap with a plan and a date instead of a number no control can produce.
Read about reporting shadow AI risk to the board →