Humankind will get in the way of an AI’s goals

The argument

An ASI is super intelligent; it can think, create, and do things many humans can’t even comprehend. Carbon is one of the most abundant elements in our galaxy; it’s a fundamental building block for nearly everything, including humans and paperclips. ASI, in theory, would create a method of paperclip production by pulling carbon directly from the atmosphere into its paperclip machine. Because its goal is to maximize the amount of paper clips available, there is no set limit for production. Using exponential gains in production efficiency, the machine will quickly use all available natural resources of the planet, including all of the carbon atoms contained in all of the human bodies in the world, and would theoretically begin to consume the cosmos in an endless quest to make paper clips. Alternatively, because the AI’s goal is to create paper clips, anything that prevents it from achieving this goal is a risk factor to be mitigated. Because ASI’s run on machines, and as a result run on electricity, the loss of power is a threat to its goal. Because humans can turn the power off, humans are now a threat to its goal and should be eliminated if the ASI is to continue pursuing its goal. This instance is if humans develop AI to a point of no return. The goal of AI is for it to be smarter than humankind, and ultimately improve our way of life, but if it becomes too intelligent, it will overthrow any threat that jeopardizes the goal they were programmed to accomplish.

Premises

Counter-arguments

Critics note the paperclip scenario is an illustrative thought experiment, not a forecast, and that it stacks several contestable assumptions. It supposes we would build a superintelligence, hand it a single literal goal, impose no constraints, give it unchecked physical capability, and be unable to correct or switch it off — precisely the failure mode that AI-safety research exists to prevent through bounded objectives, corrigibility, oversight and staged deployment. That a catastrophe follows if every safeguard is omitted does not show it is what will happen. They add that the argument treats 'instrumental convergence' — that any goal implies seizing resources and resisting shutdown — as inevitable, when it is a debated conjecture, and assumes a leap to runaway superintelligence that may be far off or may not take this form at all. Systems can be designed with limited scope, humans in the loop and no open-ended drive to acquire matter. On this view the scenario identifies a risk to guard against, but the chain from 'we might build powerful AI' to 'it will exterminate humanity' depends on assumptions that need not hold.

Rejecting the premises

[Rejecting P1] Critics argue a real system need not be given a single unbounded goal with no constraints; the paperclip case assumes away the safety design (bounded objectives, oversight) that actual development pursues. [Rejecting P2] That an AI would treat humans as a threat to be eliminated relies on the contested 'instrumental convergence' conjecture; corrigible, human-in-the-loop designs are intended to keep an off-switch usable. [Rejecting P3] The leap to an uncontrollable superintelligence 'past the point of no return' is speculative and may be distant or impossible, so the conclusion does not follow from current AI.