- Position‹2 of 2
- No, artificial intelligence will not end humankind
- Argument1 of 2›
Superintelligent AI cannot be achieved
How can we engineer something that we cannot even define? In all of human history, we never managed to work out what natural human intelligence is, so it is not clear what engineers are trying to imitate in machines. Rather than intelligence being a single, physical parameter, there are many types of intelligence, including: emotional, musical, sporting and mathematical intelligences.
The argument
This position holds that AI is parasitic on human intelligence. A system of this kind indiscriminately gorges on whatever has been produced by human creators and extracts the patterns — including some of our most detrimental habits — which means the ceiling on what it can express is set by the material it was given. Nothing in that process generates a vantage point outside the input. That is why proponents argue superintelligence is not reachable by this route. These machines do not have the goals or strategies or capacities for self-criticism and innovation to permit them to transcend their databases by reflectively thinking about their own thinking and their own goals. Transcending a database would require asking whether the patterns in it are the right ones, and that question cannot be asked by a process whose only standard is the patterns themselves. Humans will always have to control AI in order for it to accomplish tasks, because the direction has to come from outside. AI cannot think for itself, thus, cannot and will not end humankind. The deeper claim is about agency rather than capability. They are helpless, in the sense of not being agents at all: they do not have the capacity to be "moved by reasons" presented to them, and something that cannot be moved by a reason also cannot adopt one — including a reason to act against us. In the long term, ASI or artificial super intelligence is possible in principle, but proponents hold it is not desirable for them to be programmed with self-motivation. The far more constrained AI that's practically possible today is not necessarily evil, but it poses its own set of dangers — chiefly that it might be mistaken for strong AI, and trusted accordingly.
Premises
Counter-arguments
Critics reply that the confident claim that superintelligence "cannot be achieved" overstates what is known: current systems already display emergent capabilities their designers did not explicitly program, and there is no established principle that machine intelligence is permanently capped below human level or that genuine goal-directed behaviour is impossible in principle. Even granting that today's models lack agency, the danger many raise is not that AI is malicious but that a highly capable system pursuing mis-specified goals could cause catastrophic harm without any "self-motivation" — so the risk does not depend on the strong claim being false. Asserting impossibility from present limitations, they argue, is not a safe basis for ruling out the threat.
Rejecting the premises
[Rejecting P1] That current AI extracts patterns from human data does not establish that machine intelligence is permanently capped below human level; the limit is asserted, not demonstrated. [Rejecting P2] Systems already show emergent capabilities beyond what was explicitly programmed, so "cannot reflect or set goals" may not hold for future systems. [Rejecting P3] Catastrophic risk can arise from a capable system pursuing mis-specified goals without agency or malice, so the danger does not require superintelligence to be impossible to rule out.