Are we about to be overtaken by machines? No. Despite the scientific breakthroughs, and the money flowing from venture funds into AI, we have nothing resembling a machine with human intelligence.
Russell’s argument is that the danger is not malice but competence aimed at the wrong objective. The standard model of AI, in which we build a machine that optimises a fixed objective we supply, fails exactly when we cannot state the objective correctly, which is nearly always.
His alternative inverts it. The machine’s only objective is the realisation of human preferences; it does not know what those preferences are; and human behaviour is its evidence about them. Uncertainty about the objective is the feature rather than the flaw, because a machine unsure of what we want has a reason to ask, to defer, and to allow itself to be switched off.
It is worth reading next to anything on inverse reinforcement learning, which is the technical shape the third principle takes.
