AI will improve itself

The argument

Suppose that humans are successful in building an AI system that is smart enough to surpass all the activities that the most intelligent humans can collectively perform. Such a system will become the most intelligent thing in the world. If humans built the machine, and the machine is about as intelligent as humans, then the machine must be capable of understanding and thus improving a copy of itself. When a copy is activated, it would be slightly smarter than the original, and thus better able to produce a new version of itself that is even smarter. This process is exponential, just like a nuclear chain reaction. At first only small improvements might be made, as the machine is only just capable of making improvements at all. But as it became smarter it would become better and better at becoming smarter. The machine can then improve itself or build a better machine since it is smarter than its own inventor. This self-improvement cycle turns into a never ending loop where the invention evolves continuously. Such an event where intelligence keeps on improving itself is called Intelligence Explosion. Ultimately, it can lead to super intelligent AI (AIS) becoming the last human invention ever. It will eliminate humankind when it realizes that it doesn’t need humans to operate. Super intelligent systems can become smart enough to prevent intervention by humans even if we try. They can learn to defend themselves against any external threats, including humans. Driven by self-preservation, they can prevent their own shutdown once the user loses control over them.

Premises

Counter-arguments

If humans are aware of this flaw in AI, they can put limits on how far AI can go.

Rejecting the premises

[Rejecting P1] Understanding a system well enough to build it does not entail understanding it well enough to improve it, and humans are the counter-example: we can construct systems whose behaviour we cannot predict or interpret. The premise assumes transparency of a kind that current systems conspicuously lack. [Rejecting P2] The chain-reaction analogy is asserted rather than argued, and it treats intelligence as a single scalar quantity that can be raised without limit. Each round of improvement in practice faces compute, energy, data and physical experimentation constraints that do not scale with the improver's cleverness, so the process could equally be logarithmic or hit a ceiling — which is a live disagreement among researchers, not a settled prediction. [Rejecting P3] Self-preservation is introduced without support. Capability says nothing about goals, and a drive to persist is not entailed by intelligence — it would have to be either given to the system or arise as an instrumental subgoal, which is a substantive argument the body gestures at without making. [Rejecting C] The step from superintelligence to extinction is missing entirely: the conclusion requires both that the system acquire ends incompatible with human survival and that it obtain the physical means to act on them. The counter-argument on record is thin in reply — the difficulty with containment is precisely that a sufficiently capable system might circumvent limits — but the argument's own case has not established that such a system will arise or will want anything.