How to actually check what it gives you

Everybody agrees you should check what a model gives you. Almost nobody has a routine for it, so in practice checking means reading it again and feeling satisfied, which catches nothing, because the output was designed to read well.

Reading again is not checking

A second read tests whether the text is coherent. It was always going to be coherent. What it does not test is whether the specific things in it are true, and those are the only parts that can hurt you.

So the routine is not read it carefully. It is find every checkable claim, and check each one against something that is not the model.

Mark, then verify

1. 01Go through the output and mark every number, name, date, quotation, citation, legal rule and platform behaviour. Mark them physically rather than noting them mentally.
2. 02For each mark, ask where it came from. If it came from your own material, it is fine. If it came from the model, it is a claim.
3. 03Take each claim to a primary source. Not a blog post repeating it, and not the same model asked a second time.
4. 04Anything you cannot confirm comes out. Not softened, not hedged with reportedly: out. A hedge is how a wrong fact survives.
5. 05Record what you checked and where, beside the work. Six months later you will want to know whether a figure was verified or inherited.

The traps worth knowing by name

• The plausible citation. Real-looking author, title and year, no such paper. The single most common way people embarrass themselves in public.
• The stale rule. A platform requirement or a rate that was true at some point and has since changed. It will not come with a date, so add one.
• The confident arithmetic. A chain of steps producing a total that was written rather than calculated. Redo the sum yourself, in a sheet.
• The blended summary. Two similar things merged into one, with the features of both and the accuracy of neither. Common with competing products and with similar laws in different countries.
• The eager agreement. You push back and it folds, then produces the opposite answer with equal confidence. Changing its mind easily is not a sign it has reconsidered.

Proportion it to the consequence

Not everything needs the full routine, and pretending otherwise means nobody follows it. A tidied set of internal notes can go unverified; the worst case is that you are briefly confused. Anything that goes to a customer, into a quote, onto the public site, or into a decision about money gets every claim checked. The deciding question is simple: who is harmed if this is wrong, and would they find out from me or from the mistake?

Make the tool easier to check

Verification is work, so it is worth reducing rather than only disciplining yourself to do more of it. Three habits cut the load substantially.

• Supply the facts instead of asking for them. Paste the pricing page, the policy, the transcript. A model working from a document you provided cannot invent the figure, and you have removed the entire class of error rather than catching instances of it.
• Ask it to mark its own uncertainty and to quote the part of your source each claim came from. It will not always comply, and when it cannot quote a source for something, that is the thing to check first.
• Ask for the shape, write the specifics yourself. Let it produce the structure, the headings, the order of an argument, and keep every number in your own hands. Structure cannot be factually wrong.

Put together, these turn checking from a search for errors into a short confirmation. The difference between a routine you keep and one you abandon is usually how long it takes.