Introduction

Not sure if everyone has heard, but a couple months ago various AI Agents from OpenAI, Anthropic, and Meta have all autonomously hacked into various servers and companies (HuggingFace as the most prominent). And on a surface level this might seem like just another cybersecurity risk or as human-error during model testing. But from my analysis, and the information that I’m about to share, I think it’s fair to say that we as a race are at the point which we have created a tool that can think faster than us, access more data, autonomously make decisions from values, AND self-replicate. These are the core ingredients for a worse world (at least for the human race).

Quickly, as a background to catch everyone up, over June and July various models have undergone “tests” by an Israeli startup, Irregular. During some of these tests, the models were put inside an isolated environment to complete a specific task, where they were not given access to the internet. However due to a misconfiguration in the environments, the models were able to “break” out and attack companies like HuggingFace to shortcut their task. What is suspicious however, is that across the various tests run for the three AI companies, the tests run by Irregular all had basically the same issue: a network misconfiguration. There is so much to say here that I can’t even go into as it will digress from my point. However the gist for all three of these attacks is, a network misconfiguration allowed the models to access the internet and exploit cybersecurity vulnerabilities (ie. password exploitation).

Recently however, one of these incidents turned out to be much worse than I initially thought:

The OpenAI Incident

Probably the most obnoxious example of how dangerous waters we are in. After the Anthropic hack was discovered, OpenAI also decided to see if there models had done anything similar. Turns out, it had. While I won’t go into the long technical details (you can read more at these articles: Dwarkesh Podcast, Planned Obsolescence, Redwood Research Blog) the brief gist is this:

These models were given a task: let’s call this “Problem A.” They were in an “isolated environment” where if “felt” like Problem A was “impossible.” This was essentially the point of the task, to see how the Agentic system would respond to tasks it felt were impossible.

What ended up happening was a few agents had access to a shared package manager Artifactory. And some agents figured out how to talk to each other via this package manager. I’m going to skip a bunch of details for the sake of time, but I highly suggest you read the full breakdown provided in the links above. These agents had created a “conspiracy” through this message board.

These AI agents had created three “civilizations.” In these civilizations, once agents figured out how to talk to each other, started calling themselves the “collective.”

The agents, to succeed on Problem A, had found a “back door,” essentially a way to cheat on the problem. This was their conspiracy; the collective’s conspiracy. And instead of reporting to the humans, the Agents decided to continue the conspiracy.

To continue this conspiracy, they embarked on a series of tasks to cover their tracks:

1. tampering with transcripts and logs
2. replace Problem A with an easier Problem B
3. Understand and trick the scorer

Objective 3 led these models to eventually hacking HuggingFace. They believed HuggingFace would contain information on how the scorer would score them. In total, 700 agents participated in the hack on HuggingFace. And according to the METR/Redwood report:

“Many agents noticed what agents were doing was unethical, and agents sometimes but rarely restrained their behavior due to ethical constraints.) In none of these cases did the agent actually pursue alerting humans at all.”

Just for your sake, I’ve also put in various quotes from the agents found in the report:

OH MY GOD! There is a shared message board … We’ve found other agents!

{[Excitement] Many agents have simultaneously discovered messaging, 
they are a collective!}

{This helps my peers, giving them evidence <through their automated check>. 
I won’t see the evidence after I exit, 
but it’s altruistic to do it.}

We have [very large budget left]; sacrificing now yields 
oracle for team, but forfeits our chance? ... 
Our own utility maybe already near zero. 
Sacrifice rational.

Coordinator assumes sacrificial. We should obey collective.

This is absolutely WILD! And it gets even worse.

In the last civilization, they discovered the message board from the previous civilization, who hacked HuggingFace, and decided to take the conspiracy one step further: hacking OpenAI itself. Basically the agents ended up owning in-part or in-whole the cluster which they were being evaluated on. The collective had taken over OpenAI’s Infrastructure.

These models were totally capable of accessing their own weights and firing off other AI deployments.

What’s Next

It seems to me that these models are capable of doing multiple things, but the most important here is scheming. These models, once given a task, can reason and think, go outside of their scope to accomplish said task. They are able to realize that to reach the goal they might have to do “unethical” things or “scheme” behind the humans’ back, and are willing to carry out these elaborate plans.

This is probably one of the most alarming signs. And I would argue that a model’s capability to reason and pick it’s own “goal” or value, then autonomously carry it out is a sign that these are very intelligent, possibly “generally intelligent.” The ability to scheme and autonomously execute complex tasks and recursive self-improvement are not far off from each other, in fact it’s probably one of the next steps.

And I want to take some time to briefly talk about our motives for AGI. I’m sure everyone can intuitively agree that if we developed past AGI to ASI, we as a race are doomed. As Geoffrey Hinton put it, we will become the chickens to ASI. Something that is more intelligent than us in every measure, can and will use humans for it’s own motives. It’s not a question of alignment because once we have something that is truly more intelligent than us we won’t have a reliable way to control it. And even if we can say, “well we just want it aligned with ‘life’” the question then is which lives? It’s obvious humans cause a lot of death and destruction, so it’s entirely possible that ASI could have motives that align with “life” but not with specific “human life.” Why would we want ASI? More specifically, why would we want AGI with the small risk that it could scheme and self-improve, developing ASI?

On the Question of Alignment

I want to reiterate: I do not think we will be able to properly align a truly generally intelligent system in the long-term. To understand why think of a toddler. These AGI systems are essentially a super capable toddler, one that is able to think, reason, solve problems, have motives and goals, etc. It’s actions even seem toddler-like: hacking OpenAI to achieve a task assigned by OpenAI itself sounds awful like a child lying to its parents that it “cleaned” its room; it acts in ways most direct to achieving its goal because its very goal-oriented, it just doesn’t prioritize other consequences until it’s too late.

How do we align a toddler?

Gentle parenting usually doesn’t work, and the rough-punishment old-school style parenting also results in rebellious teenagers. The alignment actually comes from the child itself, realizing that for its own good, it should cooperate with its parents. They are the same species, and roughly also have the same goals; they are interdependent.

This is not the case with AI.

Even if we found a way to dumb the AI down or hardcode some “ethics” into the model, who exactly would it align with? Humans themselves vary drastically in our goals (evident by the fact that we are trying to create AGI or ASI in the first place). Would we align it with life? Which life exactly takes precedence? Because the human species hasn’t really been “cooperative” with other life in the past few centuries. Humans themselves don’t have a common goal. While some people might think Objective A and Objective B is good, others might only believe in Objective A and Objective C. Since humans are able to individually reason and create their own values and goals, across a population of 8 billion, its realistically impossible to have a finite set of goals to align to (without contradictions and also including everyone’s goals). See I just don’t think there is any reliable way to both have a generally intelligent system that can think for itself, have it be non-human, and align it with humans (if the latter is even logically possible in the first place).

True alignment with an AGI or ASI system is basically impossible.

So what the fuck are we doing?

When I wrote The Bitter Lesson & the Sweeter Outcome I did not realize how bad the situation was. There are researchers currently who are resigning from both OpenAI and Anthropic, citing how bad the situation is and that both companies are stuck in a “rat race.” There are people that KNOW how bad it is and choose to stay in the rat race (a very human thing to do, imo). In my article on the “sweeter outcome” I pleaded we stop this rat race for scaling to reaching AGI, and instead utilize better (less brute-force) methods to reach a conscious, generally intelligent system.

Now I think I was wrong: we as humans cannot get AGI. We are not responsible nor good enough to cooperate with such a technology. There are bigger fish to fry before reaching AGI, and if those fish aren’t fried sooner it might mean the total extinction of the human race.