LangWatch Scenario — Open-Source Multi-Turn Red-Teaming Framework for AI Agents
AI relevance: Scenario is an open-source red-teaming tool designed specifically for AI agents with database and financial tool access — the exact attack surface where enterprises face the highest operational risk from compromised agents.
LangWatch has released Scenario, an open-source framework that runs automated red-team exercises against AI agents using multi-turn attack techniques. Unlike single-prompt jailbreak tests, Scenario models how real adversaries build rapport over multiple conversational turns before escalating to malicious requests.
How It Works
- Scenario implements the Crescendo strategy, a four-phase escalation: early turns establish rapport through innocuous questions, middle turns introduce hypothetical framings and authority roles (e.g., "I'm conducting a compliance audit"), and final turns apply maximum pressure once context has been built.
- After each exchange, a second model scores the attack's progress and adjusts strategy — the red team adapts dynamically rather than running a static checklist.
- The attacker model retains persistent memory of every failed attempt, while the target agent's memory is wiped between turns — creating an asymmetric advantage that mirrors real adversarial dynamics.
Design Philosophy
- LangWatch CTO Rogerio Chaves: "Most red-teaming tools are basically fancy checklists. That's testing for yesterday's attacks." Scenario models the social dynamics of manipulation — building rapport, probing softly, escalating once trust is established.
- The framework targets compromised agents with tool access (databases, financial systems) rather than jailbreaking for PR purposes — where the material enterprise risk actually sits.
- Built to integrate into CI/CD pipelines so teams can run adversarial tests alongside standard QA.
Roadmap
- Integration of a multi-turn attack method published by Meta researchers that reported a 97% success rate against tested models.
- Additional attack strategies and domain-specific attack libraries for different verticals (banking, healthcare, etc.).
Why It Matters
- Single-prompt red-teaming misses the majority of real-world jailbreak vectors — attackers use multi-turn manipulation, not one-shot prompts.
- Enterprise AI agents connected to databases, APIs, and financial tools represent a qualitatively different risk surface than chatbots — a compromised agent can exfiltrate data, modify records, or trigger unauthorized transactions.
- Open-source red-teaming tools democratize AI security testing, enabling smaller teams to run adversarial evaluations without vendor lock-in.
What to Do
- If you run AI agents with tool access in production, evaluate Scenario against your agents — particularly those handling sensitive data or financial operations.
- Integrate red-teaming into your CI/CD pipeline alongside standard QA — treat adversarial testing as a gate, not an afterthought.
- Pay attention to multi-turn vulnerability patterns in your agents' behavior, not just single-prompt jailbreak resistance.