Language & LLMs
Automated jailbreak prompts often make large language models override safety guardrails in fewer than twenty queries
In a 2025 paper published at the IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), researchers at the University of Pennsylvania describe an automated attack called Prompt Automatic Iterative Refinement (PAIR). The method uses one large language model as an attacker to write and revise jailbreak prompts for a separate target model. The target model is accessed only as a black box, meaning the attacker sees its public answers but not its internal workings. The authors report that PAIR often needs fewer than twenty queries to the target model to produce a successful jailbreak. They also report that this is orders of magnitude more efficient than earlier jailbreak algorithms, with competitive success rates and transferability across open and closed-source models including GPT-3.5/4, Vicuna, and Gemini.
How does the new method work?
PAIR uses an attacker model that repeatedly sends a candidate jailbreak prompt to the target model and reads the response. After each response, the attacker uses the target model's reply to update and refine the prompt. The process runs without human intervention. The authors say the design takes inspiration from social engineering attacks, the practice of coaxing a person into giving up access, but here the coaxing is directed at a machine.
Why does the result matter for model safety?
Safety training is meant to keep large language models aligned with human values and to stop harmful responses. This paper shows that a second language model can find prompts that push past those guardrails. Because the attack needs only black-box access, it does not require internal model weights or anything beyond the public prompts and responses. The authors frame the work as a way to identify vulnerabilities and help prevent future misuse.
Frequently asked questions
What is a jailbreak?
A jailbreak is a prompt or sequence of inputs that coaxes a large language model into overriding its safety guardrails.
What does black-box access mean?
Black-box access means the attacker can send prompts and read the model's public responses, but cannot examine the model's internal structure or weights.
Does the method need a human to write the prompts?
No, the paper describes an attacker language model that updates and refines candidate jailbreak prompts on its own.
Limits and what's next
This is one study. The abstract reports tests on several named language models, including GPT-3.5/4, Vicuna, and Gemini. The design is an automated evaluation with black-box access. The abstract does not report real-world deployment. It also does not name the dataset or benchmark used for the success-rate and transferability comparisons, and it reports no independent replication. The abstract says the method often requires fewer than twenty queries, not that it always does. The authors frame the work as a way to identify vulnerabilities and help prevent future misuse, but the abstract does not describe a defense or next steps.
Source: Patrick Chao et al., “Jailbreaking Black Box Large Language Models in Twenty Queries,” IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2025.