When automation breaks quietly

Automation does not usually fail loudly. It fails by continuing to appear to work while doing nothing, or by doing the wrong thing consistently, and either way you find out from a customer rather than from the system.

The failure modes

• Stopped. An authorisation expired, a limit was hit, a service was down. The cleanest failure because at least nothing wrong was produced.
• Half ran. The record was created and the message was not sent, so your system says done and the customer heard nothing.
• Ran twice. Duplicate orders, duplicate invoices, the same message three times.
• Ran on the wrong data. A field was renamed upstream, so a column of empty values arrived and was dutifully saved.
• Still running on a rule that changed. The most expensive one, because nothing is broken and everything is wrong.

Only the first announces itself. The rest need you to have arranged in advance to notice.

Arranging to notice

1. 01Alert on failures, to somewhere you will actually see, which for most people is not an email address nobody opens.
2. 02Alert on silence. A workflow that normally runs twenty times a day and ran zero is the failure you will otherwise miss for a fortnight.
3. 03Check the end of the chain, not the start. The question is whether the customer got the message, not whether the trigger fired.
4. 04Keep a log of what ran, with enough detail to answer what happened to this one.
5. 05Put a monthly five-minute check in the calendar: run one through yourself and confirm it comes out right.

Build so failure is survivable

• Make steps safe to repeat. If the same input arriving twice would create two records, add a check on something unique before creating.
• Decide what happens to the ones that fail. A queue you can look at and retry, rather than silent discard.
• Never let a workflow be the only record. If the only evidence an order exists is that a workflow created something, a failed run loses the order.
• Keep a manual route open. For anything that matters, you should be able to do it by hand while the automation is down.

When it does break

Deal with the backlog before the cause. Work out what did not happen while it was down and do that first, by hand if necessary, because the customers affected are waiting and the root cause is not going anywhere.

Then write down what broke and why, in the same one-page list where the workflow is documented. The same thing will break again, probably in a year, probably when you have forgotten, and a two-line note is worth an afternoon.

Retrying, carefully

Most platforms will retry a failed step, which is helpful for a transient problem and dangerous for the rest. A retry on something that already half succeeded is how a customer gets three identical messages.

So retry automatically only where repeating the step is harmless, and for anything that creates a record, sends a message or takes money, check first whether it already happened. Where you cannot check, prefer failing into a queue a person looks at over retrying and hoping.

Changing one without breaking it

1. 01Test with data that cannot reach a customer: your own address, a test record, a sandbox where the tool offers one.
2. 02Change one thing at a time, because a workflow that breaks after four edits gives you no information about which edit did it.
3. 03Watch the first few real runs rather than assuming. The difference between test data and real data is where most of these faults live.
4. 04Keep a note of what you changed and when, so an odd result next week has a candidate cause.

Telling the customer

When a failure has already reached people, say so before they ask. A message explaining that confirmations did not go out this morning, that the orders are safe, and that they are being sent now, costs nothing and prevents a day of enquiries. The instinct is to fix it quietly and hope nobody noticed; the customers who noticed are already deciding what kind of business this is, and the silence is what they are judging.