As LLMs become more complex and used to solve a wider range of problems (especially in the form of agents that can interact with computer files, websites and third-party code, as well as other agents), it becomes increasingly difficult for teams of people to deal with all the types of attacks that can occur. “The risk surface is increasing, and the blast radius is also increasing,” says Nikhil Kandpal, a research scientist at OpenAI who co-authored GPT-Red.
OpenAI created GPT-Red to future-proof its security testing process. “As more powerful models become available, we will already be developing a system that can detect new attack methods,” says Dylan Hann, a research scientist at the company and co-author of GPT-Red. The researchers say they have already come up with new types of attacks that haven’t existed before.
OpenAI has focused most of its efforts on a type of attack known as rapid injection, where a hacker slips instructions into an LLM to force it to do something its developers or users don’t want, such as copying sensitive information, sabotaging a company’s codebase, or generating embarrassing or harmful output. In theory, such instructions could be hidden in any text that the LLM might encounter—for example, in code or on a website.
Training dojo
To create GPT-Red, OpenAI researchers took an LLM that had not been hacked and set it up in a so-called self-play loop with several other models. His goal was to try to attack other models; their goal was to try to protect themselves. Over many rounds of play, GPT-Red became better and better at attacking other LLMs, and those LLMs became better and better at fending off attacks.