What these tools are actually good at
Very good at turning one kind of text into another, useful for a first draft, and unreliable about any fact you did not supply. The failures cluster somewhere specific.
The useful question is not whether these tools are impressive. It is which jobs they do well enough that you would hand one over, and which they do just well enough to fool you.
The honest summary is that a language model is very good at turning one form of text into another form of text, moderately good at producing a first draft of something you will edit, and unreliable at telling you a fact it was not given.
Where it genuinely earns its keep
- Transforming something you already have. A transcript into notes, notes into a plan, a long document into a summary, a list into a table. The content comes from you, so the main failure mode is missing, not invented.
- Rewriting for a different reader. The same explanation for a customer and for a supplier. You can judge the result immediately, which is what makes it safe.
- Getting past a blank page. A bad first draft you can react to is worth more than an empty document, and reacting is easier than starting.
- Pulling structure out of a mess. Forty support emails sorted into the five things people are actually asking. This is where it is strongest and most underused.
- Explaining something unfamiliar in your own terms, so you can ask a better question of a person afterwards.
- Mechanical language work: a formula, a regular expression, a spreadsheet function, a first pass at a translation you will have checked.
Where it only looks like it is working
The dangerous cases are not the ones where it fails obviously. They are the ones where the output is well written, confident and wrong, because fluency is what the tool optimises and accuracy is not.
- Any specific figure you did not supply. A price, a fee, a threshold, a date, a statistic. These are generated the same way the prose is, which is to say plausibly.
- Anything about a named company, product, person or law. The shape of the answer will be right and the particulars may not be.
- Current information about anything that changes. The tool does not know what changed last month unless it is told or goes and looks.
- Arithmetic across several steps, where it produces an answer that reads as if it were calculated.
- Your own judgement calls. It will give you an opinion on your pricing, and it has no idea what your customers are like.
Why it is wrong confidently
It is worth understanding the mechanism, because it explains the failure pattern better than any list of warnings. The model predicts likely continuations of text. A plausible-sounding fee, citation or statistic is a likely continuation, whether or not it is the real one. The model has no separate store of facts it checks against, and no sense of the difference between remembering and guessing.
This is why the failures cluster where they do. Asking it to rephrase your paragraph is safe, because the facts came from you. Asking it what a competitor charges is not, because nothing in the process distinguishes a remembered price from an invented one.
A test before you hand a job over
- Could you tell within a minute whether the output was wrong? If yes, it is a good candidate. If checking would take as long as doing it, it is not.
- Does the job need a fact the tool was not given? If so, supply the fact rather than asking for it.
- What happens if a mistake gets through? A clumsy sentence in a draft costs nothing. A wrong figure in a quote costs a customer.
- Would a reader be worse off than if you had written nothing? If the honest answer is no, the output is filler and nobody needs it.
Work to do
- List the five things you do most often that are turning text into other text. Those are the candidates.
- Take one of them and run it, then mark every specific claim in the output you cannot verify from your own material.
- Write down one job you will not hand over, and the reason. Having a line makes the rest of the decisions easier.