The Mover Advantage: Racing toward AGI

Chapter 3 of Our Final Invention, “Looking into the Future,” showcases the ethics and risks of creating Artificial General Intelligence (AGI) technologies and systems. AGI can be more applicable to the average person than ASI due to their understandability and operation on human level intellect. With the ability of both softwares, however, to self-improve and write its own code, once a machine achieves AGI, an intelligence explosion would be triggered, leading to the creation of an ASI in days, hours, and potentially even minutes. Barrat refers to this as the “first mover advantage,” exemplifying the drive that many have to be the first to create these systems.

Barrat's interview with Michael Vassar, the president of the Machine Intelligence Research Institute (MIRI), detailed the many risks of AGI, and the caution needed in its development. The simplicity in creating AGI when compared to ASI brings up the idea that AGI could be developed by small stealth companies and rogue states. As mentioned in the previous chapter, these entities, driven by competitive pressure and malicious intent, are likely to create unconstrained and unregulated AGI systems.

Neither AGI nor ASI have been developed yet, but ideas about AGI and ASI have been circulating since the twentieth century. Barrat uses British mathematician Ivring John Good’s 1965 paper, “Speculations Concerning the First Ultraintelligent Machines,” to establish the idea that an AI apocalypse isn’t something of the future, but rather, a mathematical certainty. In one of Good’s most famous quotes (“the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control”), he predicts the exact fears and ideas discussed by Barrat. 

Barrat further elaborates on the danger of AGI and ASI by discussing the black box problem: the inability for humans to understand or explain how artificial intelligence arrives at a specific output. He warns that because AI relies on artificial neural networks (ANNs) and algorithms, the accountability and understandability of these systems will decrease. Without an ability to understand the logic of AI, computer scientists will not be able to formally impose or verify safety protocols, further increasing the lack of security with AI.

Due to the prominence of the development of AI systems, governments have been striving to create laws in order to regulate these systems. The European Union, in specific, passed the EU AI Act, which serves as a counter to the previously detailed “first mover advantage”. This act uses a risk-based system in order to determine whether to allow, limit, or ban certain AI systems based on their purpose and likelihood to develop maliciously. While it's reassuring to see that policymakers have been working to limit these softwares, there still is much fear that should be associated with the future of AI. Far more precautions and decisions must be made in order to ensure that these systems are used to benefit human society.