The danger of software systems is that because we talk about them as being ‘engineered’, we take them to be infallible, in a way that would be reasonable if we were talking of a bridge, or a train, or a road.

Bridges, trains and roads obey the laws of physics.

There are no such laws behind software systems, only human beings, with prejudices, pressures and sometimes perverse incentives.

We would do well to remember that, especially when the system is accusing a human of being in the wrong.