Why do multi-agent systems fail? The MAST taxonomy, decoded

The failure rate that should scare you

If you’re building multi-agent LLM systems in production, here’s the number that matters: 41% to 86.7% failure rates across seven state-of-the-art frameworks Cemri, M., Pan, L., Yang, S., et al. Why Do Multi-Agent LLM System…. That’s not a bug in one implementation — that’s a systemic pattern. ChatDev fails 41.4% of the time. MetaGPT fails 56.4%. Magentic-One fails 78.6%. OpenManus fails 86.7% Cemri, M., Pan, L., Yang, S., et al. Why Do Multi-Agent LLM System….

These aren’t toy systems. These are the frameworks teams evaluate for production use. And until the MAST taxonomy paper from UC Berkeley (NeurIPS 2025 Datasets & Benchmarks Track), nobody could tell you why they fail — only that they do Full text (v3):.

This post breaks down the UC Berkeley NeurIPS 2025 paper Why Do Multi-Agent LLM Systems Fail? (arXiv:2503.13657), decodes the 14 failure modes with real trace examples, and gives you a concrete tool to diagnose your own systems.

What MAST-Data actually is

The research team built MAST-Data, a dataset of 1,642 annotated execution traces from seven frameworks: ChatDev, MetaGPT, HyperAgent, AppWorld, AG2, Magentic-One, and OpenManus Cemri, M., Pan, L., Yang, S., et al. Why Do Multi-Agent LLM System…. The dataset spans coding, math, and general-agent benchmarks. Each trace was annotated using Grounded Theory — a systematic qualitative method where failure categories emerge from the data itself, not from pre-conceived hypotheses.

The result: 14 failure modes organized into 3 categories, with a Cohen’s kappa of 0.88 for inter-annotator agreement across six expert annotators Full text (v3):. That’s “almost perfect” agreement in the statistical literature, making the taxonomy reliable and reproducible.

The dataset is open on Hugging Face, the full taxonomy is documented on the project site, and the codebase is on GitHub.

The MAST taxonomy, decoded

The 14 failure modes map to three execution stages: pre-execution (system design), execution (inter-agent coordination), and post-execution (verification) Full text (v3):.

Category 1: System design issues (5 modes)

Failures baked into the architecture before a single token is generated.

FM-1.1: Disobey task specification (11.8%). Failure to adhere to the specified constraints or requirements of a task. In ChatDev, agents violated output format requirements, producing code that didn’t match the spec Full text (v3):.

FM-1.2: Disobey role specification (1.5%). An agent behaves outside its defined responsibilities. In ChatDev, the CPO agent terminated the development process without CEO approval — the system’s termination logic was tied to a single agent’s role, not a group decision Full text (v3):.

FM-1.3: Step repetition (15.7%). The most prevalent failure mode across the entire corpus. Agents unnecessarily reiterate previously completed steps. OpenManus exhibits this tendency heavily — the system loops through the same code-generation phase multiple times Full text (v3):.

FM-1.4: Loss of conversation history (2.8%). Unexpected context truncation causes agents to lose recent interaction history and revert to an earlier state Full text (v3):.

FM-1.5: Unaware of termination conditions (12.4%). Agents don’t recognize when a task is complete, leading to unnecessary continuation or premature exits Full text (v3):.

Category 2: Inter-agent misalignment (6 modes)

Failures in agent coordination during execution.

FM-2.1: Conversation reset (2.2%). Unexpected restarting of dialogue, losing context and progress Full text (v3):.

FM-2.2: Fail to ask for clarification (6.8%). Agents proceed with wrong assumptions instead of requesting additional information when faced with unclear data Full text (v3):.

FM-2.3: Task derailment (7.4%). Deviation from the intended objective, resulting in irrelevant or unproductive actions Full text (v3):.

FM-2.4: Information withholding (0.85%). The most insidious mode despite its low prevalence. In AppWorld, the phone agent successfully interacted with a phone UI but withheld the API username from the admin agent, who needed it for backend setup. The system deadlocked through repeated failed logins Full text (v3):NeurIPS 2025 Datasets & Benchmarks Track:.

FM-2.5: Ignored other agent’s input (1.9%). Disregarding recommendations from other agents, leading to missed collaboration opportunities Full text (v3):.

FM-2.6: Reasoning-action mismatch (13.2%). Discrepancy between what an agent reasons and what it actually does — the second most prevalent mode overall Full text (v3):.

Category 3: Task verification failures (3 modes)

Failures in checking whether the task was actually completed.

FM-3.1: Premature termination (6.2%). The system stops before the task is fully done. AppWorld frequently suffers from this, potentially due to its star topology and lack of predefined workflow making termination conditions less obvious Full text (v3):.

FM-3.2: No or incomplete verification (8.2%). A ChatDev-generated chess program passed superficial checks (code compilation) but contained runtime bugs because it never validated against actual game rules Full text (v3):.

FM-3.3: Incorrect verification (9.1%). The system runs verification but the check itself is flawed. HyperAgent exhibits this as a dominant failure mode alongside step repetition Full text (v3):.

The production hook: o1 as a failure judge

Manual annotation at κ=0.88 is great for research, but you can’t hire six PhDs to annotate every trace in production. The paper’s solution: use OpenAI o1 as an LLM-as-a-judge Full text (v3):.

The pipeline:

Collect traces from your multi-agent system (agent IDs, messages, tool calls, termination conditions).
Feed the trace to o1 with a structured prompt containing MAST taxonomy definitions and few-shot examples from the human-annotated subset.
Get back a failure classification with rationale.

Results:

κ=0.77 agreement with human annotators on in-domain traces
κ=0.79 agreement on a separate human triple-annotated subset (21 traces)
The judge generalizes to unseen frameworks (OpenManus, Magentic-One) and unseen benchmarks (MMLU, GAIA) Full text (v3):

Caveat for eval builders: moderate correlations between similar-symptom modes (max 0.63) mean automated evaluators can conflate distinct root causes Full text (v3):. Human review of critical traces is still warranted for high-stakes systems.

Run MAST on your traces

The paper’s team released agentdash, a Python package on PyPI for MAST-based failure diagnosis GitHub repo (MAST + agentdash):.

pip install agentdash

The full toolkit on GitHub GitHub repo (MAST + agentdash): and Hugging Face Hugging Face dataset — MAST-Data: includes:

The 1,642 annotated traces with failure-mode labels
The o1 judge prompt templates
Inter-rater reliability scripts
Framework-specific trace parsers

from datasets import load_dataset

mast_data = load_dataset("mcemri/MAST-Data")
print(mast_data["train"][0])

The practical workflow: run the judge over your production traces, get a failure-mode distribution, and prioritize by prevalence × blast radius.

Fixes are structural, not prompt-based

The paper tested targeted interventions for specific failure modes. The key result: giving ChatDev’s CEO agent final say over termination (addressing FM-1.2) yielded +9.4% task success Cemri, M., Pan, L., Yang, S., et al. Why Do Multi-Agent LLM System….

But inter-category correlation is only 0.17 to 0.32 Full text (v3):. This means failure modes are largely independent — fixing FM-1.2 (disobey role specification) does not fix FM-2.4 (information withholding). They’re different root causes requiring different architectural changes.

This kills the “prompt engineering will save us” approach. You can’t write a better system prompere analyzed across GPT-4, Claude 3, Qwen2.5-Coder, and CodeLlama-7B — a useful corrective for teams assuming “better model = fewer orchestration failures” Full text (v3):. 4. The dataset is static. As frameworks evolve, new failure modes will emerge.

TL;DR

Multi-agent LLM systems fail at alarming rates: 41–86.7% across seven major frameworks Cemri, M., Pan, L., Yang, S., et al. Why Do Multi-Agent LLM System….
The MAST taxonomy organizes failures into 14 modes across 3 categories: system design (5), inter-agent misalignment (6), task verification (3) Full text (v3):.
Top offenders: step repetition (FM-1.3, 15.7%), reasoning-action mismatch (FM-2.6, 13.2%), unaware of termination (FM-1.5, 12.4%) Full text (v3):.
o1 as a judge works: κ=0.77 agreement with humans, generalizes out-of-domain Full text (v3):.
Fixes are structural: +9.4% from the CEO-consensus fix, but categories are nearly uncorrelated (0.17–0.32) — diagnose before you fix Cemri, M., Pan, L., Yang, S., et al. Why Do Multi-Agent LLM System…Full text (v3):.
Get started: pip install agentdash, open dataset on Hugging Face, code on GitHub Hugging Face dataset — MAST-Data:GitHub repo (MAST + agentdash):.

Stop treating multi-agent failures as “the LLM was confused.” Start treating them as structural defects with identifiable root causes. The MAST taxonomy gives you the diagnostic vocabulary. The judge pipeline gives you the tool. The dataset gives you the ground truth.

Your system is failing. Now you can find out why.