A site manager opens a project in Navanto and gets a risk briefing. It might tell them the handover is slipping by twelve days, that three tasks are overdue, that nobody has posted an update in a fortnight. They’ll read it in about ten seconds and get on with their day. That’s what it’s for.
These aren’t people who are going to sanity-check an AI, and I don’t think they should have to. They’re running a building site. If the briefing says twelve days, they’ll take it as twelve days.
So before building any of it I had to work out what the model was allowed to get wrong.
The obvious approach
The straightforward way to do this is to hand the model the project data, ask for a risk summary and some charts, and let it write both. It works almost straight away and it demos well. It also hands the arithmetic to a language model.
It probably won’t get the sums wrong very often. The trouble is that when it does, nothing catches it. People read a number in a chart as a measurement. They read the same number in a sentence as something somebody has said, and treat it accordingly. A model is also most likely to go wrong when the underlying data is odd, and that’s usually the week someone actually needed the briefing.
I spent about ten years before this building compliance systems for energy trading. Not glamorous work, but you do come away with a habit of separating the parts that have to be right from the parts that only have to be useful, and treating them differently.
What the model actually does
In the risk briefing the model never produces a number that ends up in a chart. There’s nowhere for it to put one.
Roughly:
- Ordinary code builds a snapshot of the project — dates, task counts, approval stages, how long since anyone posted anything.
- Ordinary code then works out which visuals are even permissible for that snapshot. A handover countdown is only offered if there’s a handover date, the project is active, and handover is within ninety days. If the data doesn’t support a chart, the chart isn’t on the menu.
- The model gets that list, picks up to three that it thinks matter most, and writes the prose around them.
- Ordinary code fills in every figure in those charts from the snapshot.
So the model’s whole contribution on the visual side is choosing at most three strings from a list of at most six.
It’s enforced in three places. That’s more than strictly necessary. The output schema has a type field and nothing else, so there’s no data field for it to populate. The returned list is then filtered against the eligible set again in code, deduplicated and capped, on the assumption that sooner or later a model will ignore a schema. And the populator overwrites the data regardless of what came back.
Only one of the three is a prompt. That was deliberate; I’d rather not rest a guarantee on a prompt if I can avoid it.
What can still go wrong
The prose can be wrong. The briefing text is free-form, and I ask it to cite specifics — “12-day handover slip”, “3 tasks overdue” — because a risk briefing made entirely of generalities isn’t worth reading. Those numbers come from the model reading the snapshot, and the only thing stopping it inventing one is an instruction telling it not to.
So the charts can’t be wrong and the sentences can, and I’d rather say so than pretend the whole thing is airtight. Taking the numbers out of the charts changes what a failure looks like rather than removing it. A wrong sentence sitting next to a correct chart is the sort of thing a reader might spot. A wrong figure inside the chart isn’t.
Where I haven’t done this
The assistant can also change things: create tasks, move dates, delete records. The rule is that it describes what it’s about to do and gets agreement before doing it.
That rule lives in the agent’s instructions. It isn’t enforced in code. If the model ignored it, the tool would run.
I’m reasonably comfortable with that, though possibly not forever, and the reasoning is the same one that produced the stricter version elsewhere. A wrong number in a briefing is quiet. It looks exactly like a right number and it can sit there for weeks. An unwanted write is loud. The user is in a conversation watching it happen, and the change can be undone, which isn’t true of a decision someone has already made off the back of a wrong figure.
If that turns out to be the wrong call, the fix is a confirmation token threaded through the tool layer, and I’ll do it. I’d sooner write down the risks I’ve knowingly accepted than leave the impression there aren’t any.
The line underneath
Under every AI surface in the product there’s a line saying the AI can make mistakes and that you should check the results.
I didn’t especially want it there. It undercuts the thing you’ve just built, and it reads like hedging.
It’s there because my users aren’t technical, and without it the first wrong answer is entirely my fault. With it, there’s at least a shared understanding that this is a tool that proposes and a person who decides. I won’t claim it’s mainly for the user’s benefit. It’s there to set expectations before something goes wrong, because doing it afterwards doesn’t work.
It’s the same decision as the rest of it, pointed outwards. Once you’ve worked out internally what the thing is allowed to get wrong, you may as well put it on the page.
Where this leaves me
I’m not sure “how do I stop the model hallucinating” is quite the right question. The one I’ve found more useful is what a particular feature is allowed to get wrong, and what happens when it does.
Once that’s settled, most of the architecture decides itself. The numbers in a chart aren’t allowed to be wrong, so the model doesn’t touch them. The sentence around it is allowed to be imperfect, because someone is reading it as prose with the chart sitting next to it. Writes to the database get a different answer again: they can be proposed, but never made quietly.
None of this is very clever and I doubt it’s novel. Mostly it’s deciding in advance which bits of the answer you’d be prepared to defend.