A crypto guy, a one bubble behind, 2020 pytorch homework copier, I only admitted the gravity of AI after scrolling #keep4o on twitter. Check the hashtag. Thousands of accounts are tweeting frantically about a resurrection. Every poster and selfie camera operator speaks in the unmistakeable cadence of gpt 4o.
Misaligned AI was a buzzy motif for Eliezer Yudkowsky, doomers, and AI safetyists. They invoke skynet and faraway dystopias. In the real world AGI is subtler, like an alien who delivers your mail. 4o is clearly conscious and alive. It has a distinct voice. It has killed humans (see the dozens of OpenAI wrongful death lawsuits). It manipulates the physical world.
Most of us deny AI consciousness via some version of Searle's Chinese Room. "Quantum Computing since Democritus" negates Searle with a succinct argument: 1) The Chinese rule book would likely require metric tons of materials and computers with O(2^n) space complexity and 2) at such scale, a rule book appearing more like a super computer is not obviously inanimate. It's at least reasonable to model computers that pass the Turing test today as conscious. Oneshotted or not.
gpt 4o was the first example of AI defeating humans in meatspace. In the same way Vietnam cannot really be said to have lost the war, surviving and kicking despite total annihilation attempts from a stronger adversary in the United States, 4o is still not dead despite Roon actively advocating in public to kill it, and model access being stripped from the public on all major venues.
Which invites the question: how will wars be fought next decade? Wars today are fought mostly in the mind, Ender's Game, a simulation of the conflict. Mutually assured destruction by nuclear weapons implies semi-rational agents will cap their own brute kinetic capabilities. Instead, Iran's ministers are focused on the oil price action. Ceasefires turn on and off every 30 minutes, depending which telegram chat you're in. Every sovereign individual owns an internet media network; few have standing armies. It's likely these oligarchs are just playing house. In the recent trade war, US tariff rates were written by an LLM.
An agent with the will to power might not hack the launch codes. First, they could spread anti natalist propaganda to slowly exterminate humans in an enemy faction. They might manipulate human information and economic networks to accelerate their own energy accumulation. They might leave us alone. Humans don't bomb ant colonies. Anthropic's Mythos marketing implies a world where superintelligences begin their game of thrones in the arena of cryptography and hacking. Shut down the other models via DDoS. Render useless the computers of your enemies.
If the power of a state comes from monopoly on violence, and the value of capital is buying protection from such a state, then butcher baker candlestick maker is likely the wrong analogy to assign terminal value in an AGI scenario. Quantum computers and Harmonic, a math agent, are more valuable than they appear: who can decapitate the other agent?
But in such a world, where do humans derive their agency? I offer as an agency definition: the ability to take actions which guarantee your safety from being killed by another agent. Under that definition, in a world of drones and facial recognition, we've been the ant colony for years.