There is a demo you have probably seen. Someone types "show me last month's revenue" into a chat box, an AI writes some SQL, a chart appears, everyone claps.
Here is what the demo does not show you: nobody in the room checked the number.
We have spent a lot of time checking those numbers, and the failure rate is bad enough that we designed our whole approach around avoiding it. Not because language models are bad at SQL - they are increasingly good at it - but because being good at SQL is not the same as knowing what your business means.
The trap is the table names#
Take ERPNext, which a large number of businesses run on. It is a serious system, and like every serious accounting system its schema reflects double-entry bookkeeping rather than plain English.
Ask a naive AI for "revenue last month" and it will find a table full of sales invoices, sum a column that is plausibly the total, and give you a number. The number will look right. It may even be right, on a good day, for a simple shop.
But it will not have known about credit notes that reverse part of that revenue. Or about invoices sitting in draft that were never submitted. Or that real, reconciled revenue in ERPNext lives in the general ledger, not in the invoice table at all. Stock valuation is not where you would guess either. And gross margin does not exist as a stored number anywhere - it has to be reconstructed from cost of goods sold.
None of this is exotic. It is just how the system works. And a model that has never seen your particular configuration cannot know it, so it does the most reasonable thing available: it guesses, fluently.
What we built instead#
Rai, our AI analyst for retail businesses, is not allowed to invent its own maths. It reasons over a curated catalog of business metrics - a fixed vocabulary that every shop shares, and that a human being defined once, carefully.
Revenue and gross margin, by item and by category, by day, week or month. Stock turnover. Days of stock remaining. Dead and slow-moving stock, with the capital tied up in it. Top and bottom movers. Supplier price variance for the same item across different suppliers. Receivables aging. Till position. Serial-level history - where a unit came from, which invoice it went out on, whether it is still in warranty.
Each of those is mapped, once, to a query that has been checked. When you ask a question, the AI's job is to work out which metric you mean - not to invent how to compute it. That is a much smaller job, and a much safer one.
The model chooses the question. A human already decided the answer's arithmetic.
The rule that makes it hold#
A catalog only helps if the AI is actually required to use it. Ours is not optional: Rai has to check the catalog before anything else, and it is only permitted to fall back to writing its own query when it can state plainly that no defined metric answers what you asked.
When it does fall back, three things are true. The query is read-only - it can select, and nothing else. It is fenced in with a forced row limit, a timeout, and a hard scope to your company only. And it shows its working: the query it ran and the assumptions it made, in a panel you can open.
That last one costs us something. It is an admission that this particular answer is less certain than the others. We think that is exactly the right trade. The alternative is a system where every answer looks equally confident, which is precisely how you get burned.
What this feels like from the owner's chair#
You will not notice any of this, most days. You ask what your margin on power tools was in June, you get a number, and the number is right.
You notice it on the day you ask something unusual - some question nobody anticipated - and Rai either tells you honestly that it is improvising, or tells you it cannot answer that one accurately yet. Either response is more useful than a clean, confident, wrong chart.
That is the whole design principle, really. We would rather build something that occasionally says "I am not sure about this one" than something that is never unsure and is sometimes wrong.