SWITCHED ON
The daily technology series nobody asked for but everyone needed
Breaking It on Purpose: AI Red-Teaming and the Science of Finding Failure
Before a frontier AI model reaches the public, teams of researchers spend weeks trying to make it do the worst things they can imagine. This is not a fringe activity. It has become one of the more consequential and least publicly understood professions in the entire AI industry.
You cannot prove a sufficiently complex AI system is safe. You can only fail to find the ways it is unsafe, using the most creative and adversarial testing you can muster, and treat that failure to find problems as provisional evidence rather than proof. This is a genuinely uncomfortable epistemic position for an industry racing to deploy systems at global scale, and understanding it is essential to understanding how AI safety actually works in practice.
Yesterday we covered the humanoid robotics race — the economic logic of human form factor, Figure's narrow BMW deployment versus Tesla Optimus's teleoperation scepticism, the manipulation problem as the real bottleneck, and a realistic timeline that separates structured industrial deployment from the much further-out general-purpose home robot. Today we are going behind the curtain on a practice that determines what actually reaches you when you use an AI product: red-teaming and safety evaluation. The deliberate, adversarial process of trying to break an AI system before it ships, the institutions that have grown up around this practice, and the genuinely hard epistemic problem at its centre — that you can never fully prove a system is safe, only fail, so far, to prove that it isn't.
01 — What Red-Teaming Actually Involves
Red-teaming, a term borrowed from military and cybersecurity practice where a "red team" simulates an adversary to test an organisation's defences, in the AI context refers to the systematic practice of deliberately attempting to elicit harmful, incorrect, or otherwise undesirable behaviour from an AI model before it is released, in order to identify and fix problems in advance rather than discovering them after millions of users have access to the system.
The practice ranges from structured and systematic to genuinely creative and adversarial. Structured red-teaming involves running a model against established benchmark suites — standardised sets of prompts designed to test for specific known failure categories including generating instructions for weapons, producing content that sexualises minors, providing dangerous medical or legal misinformation, or exhibiting biased or discriminatory outputs across protected characteristics. Creative red-teaming, considered by most practitioners to be the more valuable and harder-to-automate component, involves skilled human testers — often including specialists in specific harm domains like chemical weapons, cybersecurity, or child safety — attempting genuinely novel approaches to circumvent the model's safety training, using techniques including elaborate fictional framing, incremental escalation across a conversation, claimed authority or expertise, and dozens of other adversarial patterns that have emerged from the practical experience of testing these systems.
A red team's job is not to prove the model is safe. It is to fail, as thoroughly and creatively as possible, to make the model do something bad — and every failure to break it is weak evidence of safety, while every success is a concrete, fixable problem. This is an inherently asymmetric and never fully completable task.
02 — The Institutions That Have Emerged
The practice of AI red-teaming has matured from an internal, ad hoc activity within individual AI labs into an increasingly formalised discipline involving dedicated government institutions, third-party auditing organisations, and standardised evaluation frameworks that did not exist prior to 2023.
The UK AI Safety Institute, established following the 2023 Bletchley Park AI Safety Summit, and its counterpart the US AI Safety Institute (subsequently reorganised under the Trump administration into the Center for AI Standards and Innovation), represent the most significant government-level efforts to conduct independent, pre-deployment evaluation of frontier AI models, with several major AI labs including Anthropic, OpenAI, and Google DeepMind agreeing to provide these institutes with pre-release access to conduct independent safety testing before public deployment — a voluntary arrangement that reflects genuine industry recognition that self-assessment alone is an inadequate safety assurance mechanism, even as the binding force of these voluntary commitments remains an open question.
METR (Model Evaluation and Threat Research), a non-profit organisation that has become one of the most influential third-party evaluators, has developed specific methodologies for assessing what it calls "autonomous replication and adaptation" capability — essentially, testing whether an AI model could, in principle, take actions to acquire resources, avoid shutdown, or otherwise act in ways that would be concerning if the model were pursuing goals not fully aligned with its developers' intentions. This is a deliberately conservative, capability-focused evaluation approach: rather than trying to determine whether a model would actually attempt such behaviour, it tests whether the model has the underlying capability to do so, on the reasoning that capability evaluation is more tractable and more conservative than attempting to assess intent or disposition.
03 — The Jailbreak Arms Race
Public jailbreaking — the practice, conducted by researchers, hobbyists, and occasionally malicious actors, of publicly discovering and sharing techniques to circumvent an AI model's safety training after it has been deployed — represents a parallel and less formalised evaluation ecosystem that operates continuously after a model's release, providing an ongoing, adversarial, crowd-sourced stress test that no internal red-teaming process, however thorough, can fully anticipate given the sheer creative diversity of a global user base actively trying to find weaknesses.
The techniques that have emerged from this ecosystem have grown genuinely sophisticated: role-play framings that ask a model to simulate a fictional character without safety constraints, "grandma exploits" that frame harmful requests as nostalgic bedtime stories, encoding requests in ciphers or foreign languages that safety training may not cover as comprehensively as the primary training language, and multi-turn approaches that build toward a harmful request gradually across a conversation rather than requesting it directly, exploiting the fact that models trained to refuse obviously harmful single-turn requests may be more susceptible to requests that arrive incrementally. AI labs treat the public jailbreaking community with a complicated mixture of adversarial defensiveness and genuine appreciation, because publicly disclosed jailbreaks — despite the immediate risk they pose before being patched — provide free, continuously updated red-teaming data that materially improves the next generation of safety training.
04 — What Evaluations Actually Measure and Their Limits
The evaluation science underlying AI safety testing has matured significantly but retains genuine and acknowledged limitations that are worth understanding honestly rather than glossing over. Benchmark evaluations measure performance on a fixed, known set of test cases — which means a model can, in principle, be specifically trained or fine-tuned to perform well on the benchmark without the underlying safety property the benchmark was designed to measure actually being robustly present, a problem closely related to the general machine learning phenomenon of overfitting to a test set. Serious evaluation organisations address this partly by maintaining private, held-out test sets not available to model developers during training, though the arms race between evaluation design and models trained on ever-larger swathes of internet data (which may inadvertently include evaluation-adjacent content) is ongoing.
The harder and more fundamental limitation is that current evaluation science is considerably better at measuring capability (can the model do X) than at measuring disposition or alignment (would the model choose to do X unprompted, and does it understand why X might be harmful in a way that generalises to novel situations the evaluation did not specifically test). This is precisely the gap that Anthropic's interpretability research, which we discussed in Season One's AGI episode, aims to help close — by trying to understand what is actually happening inside a model's internal representations rather than relying solely on observing its external behaviour across a necessarily finite set of test scenarios.
05 — Why This Matters More As Models Get More Capable
The stakes of getting AI evaluation right scale directly with model capability, for a reason that is straightforward but easy to overlook amid the technical detail: a model that is only moderately capable poses a bounded risk even if safety evaluation misses something, because the model's own limitations constrain the potential harm. A model approaching the frontier of human-level or superhuman capability in domains including biological, chemical, or cyber weapons development poses a qualitatively different risk profile, where a gap in evaluation coverage could have consequences that are not bounded by the model's own limitations in the same reassuring way.
This is the underlying logic behind the tiered risk frameworks that most frontier AI labs, including Anthropic with its Responsible Scaling Policy, have adopted — committing to specific, pre-defined additional safety measures and evaluation requirements that automatically trigger as models cross defined capability thresholds in specific dangerous domains, rather than relying on ad hoc judgment calls made under the commercial pressure of a launch deadline. Whether these voluntary frameworks, adopted unilaterally by individual companies operating in a competitive market with strong incentives to ship quickly, are sufficient without binding regulatory backing is a genuinely open and actively contested policy question — one that connects directly back to the AI regulation episode from Season One, and one that this season will likely need to revisit as frontier capability continues to advance.
Tomorrow we are shifting from AI safety science to a topic that has been building in the background of dozens of episodes across this series — the technology and economics of critical mineral supply chains beyond the lithium and cobalt we covered in earlier seasons, and why the energy transition's mineral demands are creating a new geopolitical map. See you then.
Switched On is a daily technology series covering the ideas, systems, and arguments shaping the digital world. Opinionated. Witty. Occasionally wrong. Always worth the argument.
