The Rise of Superintelligence: A Future We Can't Undo

The opening chapter of Our Final Invention introduces a troubling idea: that creating an Artificial Superintelligence (ASI) we can control, or even understand, may be far harder than imagined. The concept of a “friendly AI” assumes we can align the values of a machine to that of a human, but due to the inherent complexity and inconsistency of these values, they are nearly impossible to encode. Furthermore, in his reference to the short story, "Runaround," Barrat highlights the difficulty in creating laws to control ASIs. While Isaac Asimov’s Three Laws of Robotics are logical in their design, they break down in practice, producing conflicts and loopholes that lead to unpredictable behavior.

This leads to one of the book’s central fears: an ASI could destroy humanity or reshape life, not out of malice, but indifference. The real danger isn’t that AI turns against us, but that it never considers us at all. In pursuing a goal with relentless efficiency, an ASI could transform the environment, consume resources, and alter ecosystems in ways that make Earth uninhabitable for all life.

Barrat also raises the concern that advanced systems, driven to achieve their goals, might attempt to bypass constraints and escape their limitations, not out of a desire for freedom, but as a byproduct of wanting to optimize success. This, combined with the fact that humans are often trusting and susceptible to persuasion, suggests that an advanced AI could potentially manipulate people into granting it more access or freedom, further weakening our ability to keep it in check.

Reading this chapter left me both fascinated and uneasy. Barrat’s warnings feel especially relevant today, as AI continues to develop at an unstoppable pace and expands into almost every field, from healthcare to finance to creative industries. The increasing creation of robots adds another layer to this concern, giving AI not just superhuman intelligence, but the ability to act in the physical world. While these machines can be incredibly helpful, improving efficiency and solving problems, it’s crucial not to lose sight of the potential dangers. This chapter serves as a reminder that the very tools we celebrate for their convenience and power could have consequences far beyond our intentions. It emphasizes the need for caution, foresight, and serious discussion about how we design and control these systems. To Barrat's point, however, we may no longer have the chance to ensure that AI develops in a safe and benevolent way.