Most write-ups of AI products stop at what the thing can do. The two questions that actually decide whether it is still running in a year are duller than that.
Does each use of it make money or lose money? And when someone edits a prompt on a Thursday afternoon, will you find out if it broke something?
Neither has a clever answer. Both have a correct one, and getting them wrong is how AI features die - not dramatically, but by quietly costing more than they earn while slowly becoming less accurate than they were at launch.
Why AI pricing is different#
Ordinary software has a marginal cost near zero. One more user on a web app is some database rows and a bit of CPU. This is why per-seat pricing works: revenue scales, cost barely does.
An AI feature does not behave like that. Every question costs real money in tokens, and the heaviest users cost several times what the lightest ones do. Charge a flat fee and a handful of enthusiastic customers will consume the margin from everyone else.
So Rai is metered. Requests draw from a credit balance, sized against what a request genuinely costs us to serve.
But metering alone is a bad customer experience in this market. A shop owner who cannot predict this month's bill will simply not use the tool - which is the worst outcome for everyone, because usage is what makes it valuable. So the packaging is a flat monthly add-on with a bucket of credits included, and metered overage beyond that. Predictable for the shop, capped for us.
The three levers on cost#
Once you are metering, margin becomes an engineering problem with three levers.
Route by difficulty. Not every request needs your best model. Generating a routine report - the user asked for last month's revenue by category, the metric is defined, the query is known - is a small job. Advice is not: "what should I do about my slowest-moving stock?" needs genuine reasoning over several figures. Rai uses a cheap fast model for the first and a stronger one for the second. This single decision moves the unit economics more than anything else in the system.
Cache identical work. If three people in the same shop ask for the same report on the same morning, that is one computation, not three. Obvious, easy to skip, and worth a surprising amount over a month.
Size credits against reality, not against a guess. Which requires actually knowing what each request cost - which brings us to the second half.
The failure mode software does not have#
Here is what makes AI features genuinely different to operate.
You change a sentence in a prompt to fix a formatting complaint. Everything still works. Nothing errors. But a question that used to map to the gross-margin metric now maps to the revenue metric, and for the next three weeks a set of customers get an answer that is wrong in a way nobody notices.
There is no stack trace for that. No test failed, because in most codebases there was no test that could.
So we wrote the test. Rai has a fixed suite of cases in the form when a user asks X, the correct metric is Y. Real phrasings, including the awkward ones, including Swahili and Sheng ones. It runs on every deploy. A prompt change that quietly re-routes a question fails the build instead of reaching a shop.
It is a boring artefact. It is also the single highest-leverage thing in the repository, because it converts the scariest class of AI bug - the silent behavioural regression - into an ordinary red build.
An AI system without a regression suite is not a product yet. It is a demo you are charging for.
Seeing what actually happened#
The last piece is tracing. Every agent run is traced: the prompts, the tokens, the latency, the path it took. We use Langfuse's free tier, and it does double duty - it is how we debug a strange answer, and it is the source of truth for what a request really cost, which is what credit sizing depends on. Errors go to Sentry.
Both are cloud free tiers, deliberately. There is a version of this where you self-host your observability stack from day one and spend a fortnight on it. We would rather spend that fortnight on the metric catalog and move to self-hosted if volume ever makes it worth it.
And nothing goes to production without going through staging first - prompt changes and catalog changes especially, because those are the ones that break things invisibly.
The summary for anyone building one of these#
If you are putting an AI feature into a product, the interesting work is not the agent. It is:
- knowing what each request costs, from real traces rather than estimates;
- routing cheap work to cheap models;
- metering somewhere the customer cannot reach;
- and a regression suite that catches the day your prompt tweak silently changed what a question means.
None of that will appear in the demo. All of it decides whether the thing is still running, and still correct, a year later.
