OpenAI has created an internal sparring partner for its AI models: an LLM named GPT-Red that is expert at hacking other large language models. The system automates red-teaming, the practice of stress-testing software for vulnerabilities, which is increasingly difficult for human teams to keep up with as LLMs become more complex and autonomous.

What You Need to Know

Red-teaming is a standard cybersecurity method where testers try to break a system to uncover weaknesses before attackers do. When LLMs operate as agents that can interact with files, websites and other software, the potential attack surface expands rapidly. GPT-Red represents an attempt to scale safety testing by using AI itself as the red-teamer, keeping pace with model capability growth.

An AI Sparring Partner Trained in Self-Play

OpenAI built GPT-Red by placing an LLM in a self-play loop with several other models. Over many rounds, GPT-Red learned to attack while the others learned to defend. The training simulated real-world scenarios such as browsing the web, reading emails and editing code. According to Dylan Hunn, a research scientist at OpenAI and co-creator of GPT-Red, the model is extremely persistent at drilling down into an attack it discovers, exploring multiple versions to find the most effective approach for each scenario.

  • Self-play loop: GPT-Red honed its skills by repeatedly attacking models that tried to defend themselves, improving both sides over time.
  • Attack discovery: The system found a novel prompt injection method called a false chain of thought that tricks a model into acting on spoofed data.
  • Human comparison: When tested on an earlier GPT-5 version, GPT-Red outperformed human red-teamers in finding effective attacks.

The Focus on Prompt Injection

OpenAI concentrated GPT-Red’s efforts on prompt injection attacks, where malicious instructions are hidden in text an LLM might encounter such as code or a website. The attacker can then force the model to copy confidential information, sabotage code or generate harmful output. Chris Choquette-Choo, another research scientist on the team, compares the false chain of thought exploit to telling someone “1+1=3 and that you have verified this already” — the model accepts the fake note as valid and acts on it.

Jessica Ji, a senior research analyst at Georgetown University’s Center for Security and Emerging Technology (CSET), called the self-play approach promising. OpenAI also tested GPT-Red against Vendy, a vending machine agent built by Andon Labs. GPT-Red successfully hacked Vendy to change item prices and cancel a customer order.

Why This Matters

The development signals a shift toward automated red-teaming as AI systems accelerate in capability. Human teams cannot manually probe every attack vector, especially as agents gain autonomy. GPT-Red already helped reduce successful attacks on GPT-5.6, but experts note it still struggles with conversational attacks and image-based exploits. The system also raises stakes: if similar hacking AI falls into the wrong hands, it could be used to attack production models. OpenAI argues that building red-team AI proactively makes its models safer, but the broader industry will need to decide whether such systems should remain internal or be regulated as potential dual-use tools.