Using AI to find things out
It is good at telling you what the question is and bad at answering it. A reference you cannot open does not exist, and primary sources are easier to reach than people think.
Asking a model a question feels like searching, and it is not. Search returns documents somebody wrote and leaves the judging to you. A model returns a confident paragraph with the judging already done, invisibly, by something that cannot tell what it knows from what it is guessing.
That does not make it useless for finding things out. It makes it useful for a different part of the job than most people use it for.
The part it is good at
Not answering the question. Working out what the question is.
- Naming the thing. You can describe a problem in your own words and get back the term the field uses for it, which is what makes a real search possible.
- Mapping the territory. What the main approaches are, what they are called, roughly what each is for. Enough to know what you are looking at.
- Telling you what you do not know. Ask what somebody doing this well would check that a beginner would miss, and the answer is usually genuinely useful.
- Explaining a document you already have. Paste the terms, the policy, the spec, and ask what it means. The source is in front of it, so it has something to work from.
- Drafting the questions to put to an actual expert, so you do not waste the first half of the conversation.
The part it is bad at
- Any figure. Prices, fees, limits, thresholds, dates, market sizes. Treat every one as unverified regardless of how precise it looks.
- Citations. A reference with a plausible title, author and year is exactly the kind of thing a text predictor produces well. Check that each one exists before repeating it.
- Anything that changed recently, unless the tool went and looked and told you where it looked.
- Which of two options is better for you. It does not know your constraints and will answer anyway.
A routine that works
- Ask the model for the vocabulary and the shape of the subject, not the answer.
- Take the terms it gave you and go to a primary source: the regulator, the platform's own documentation, the standards body, the company's pricing page.
- Read the primary source yourself. If the subject matters enough to write about or spend money on, it matters enough to read one real page about.
- Go back to the model with the primary source pasted in and ask it to explain the parts you did not follow. Now it is interpreting a document rather than recalling one.
- Write down where each figure came from as you go, not afterwards. Afterwards you will not remember which ones you checked.
Primary sources are more available than people think
Most business questions have an authoritative page that settles them, and most people never open it. Advertising rules are published by the advertising platform. Delivery requirements are published by the mailbox providers. Payment fees are published by the processor. Tax rules are published by the tax authority. The reason people rely on second-hand summaries is not that primary sources are hidden; it is that a blog post is easier to read than a specification.
That is a real advantage available to anyone willing to spend twenty minutes. Half the confident advice in circulation is a summary of a summary of a page that has since been updated.
Asking in a way that produces something checkable
How you ask changes how much of the answer you can verify, which matters more than how good the answer sounds.
- Ask for the source alongside each claim, not at the end. A list of references appended afterwards is assembled separately from the claims and tends not to match them.
- Ask what would have to be true for the answer to be wrong. This produces the assumptions, which is usually where the error is, and it is a question a model answers well because it requires no facts.
- Ask for the disagreement. Where practitioners differ and why. A subject presented as settled when it is not is the most common way a confident summary misleads.
- Ask it to distinguish what is specified from what is convention. A platform's documented rule and the thing everyone does are different kinds of claim and get conflated constantly.
- Give it the constraint you are actually under. Budget, country, size, legal form. An answer written for a company fifty times your size is wrong in a way that is hard to spot because nothing in it is false.
None of these makes the answer correct. They make it possible to tell whether it is, which is the only thing you can engineer from your side.
Work to do
- Take one number you repeat about your own industry and find the original source for it. Note how old it turns out to be.
- For your next decision, write the list of primary sources before you ask anything of a model.
- Pick one claim a model has given you recently and try to open its source. Decide what that tells you about the rest.