Your agent works beautifully on the demo laptop. Then it hits production — where context windows are half-empty, costs triple overnight, and a regex breaks at 3 AM. Sound familiar? Paul Iusztin and Louis-François Bouchard call it PoC purgatory: teams that ship demos, then watch them silently decay in production because nobody addressed the system-design layer underneath.

After two years building and breaking agents, the pattern is clear — the same six silent mistakes show up again and again. None of them are bugs. They’re architectural decisions that work fine at small scale and quietly destroy reliability as you scale.

Here’s the diagnostic checklist.

1. Treating the context window as an afterthought

The default move is to dump everything in: system instructions, full conversation history, tool schemas, retrieval results, few-shot examples. At 2K tokens it works. At 50K it doesn’t.

The problem has a name: lost in the middle. LLMs attend most strongly to information at the beginning and end of the context. Stuff a critical rule in the middle of a 40K-token blob and it gets reliably ignored. You also pay for every token — twice if you’re using extended thinking — and latency climbs linearly with context length.

Why it happens: Context windows grew from 4K to 200K, so teams stopped thinking about context as a managed resource. “The model can handle it” became the default assumption.

The fix: Scope context as working memory, not long-term storage. Before each agent step, inject only what that step needs — relevant history summary, specific tool schemas, current task state. Treat the context window like a whiteboard: wipe it clean each step, write only what matters right now.

def build_step_context(full_history, current_task, tools_for_step):
    """Only load what this specific step needs."""
    return {
        "system": system_prompt,
        "history_summary": summarize(full_history, max_tokens=2000),
        "task": current_task,
        "tools": [t for t in tools_for_step if t.is_relevant(current_task)],
    }

2. Starting with complicated solutions

This one costs the most money. Teams reach for multi-agent orchestration, heavy frameworks, multi-index RAG pipelines, and MCP before validating the basics. Hidden tax: every dependency adds latency, every abstraction layer adds debugging cost, and every integration point is a failure mode.

Concrete example from Iusztin and Bouchard: ZTRON built a multi-index RAG system with OCR preprocessing, embedding pipelines, hybrid retrieval, and agentic RAG loops. Simple document queries took 10–15 seconds. Then they realized the data fit comfortably inside a modern context window. They replaced the entire pipeline with cache-augmented generation (CAG) — precompute document representations, load relevant ones directly into context. Fewer LLM calls, lower latency, fewer failure modes, fewer errors.

Why it happens: Engineering teams conflate capability with production-readiness. A framework demo shows something impressive, so it must be the right architecture. It isn’t.

The fix: Start with the simplest solution that could possibly work. Single prompt first. Single agent with a few tools. Add complexity only when the problem has proven it needs it — and measure the cost of each addition.

3. Building agents when a workflow will do

An agent autonomously decides what to do next. A workflow follows a predetermined path. Most production tasks — data ingestion, report generation, content summarization, ticket routing — don’t need autonomy. They need reliability.

Anthropic’s engineering team draws the distinction clearly: workflows are orchestrated sequences (prompt chaining, routing, parallelization, orchestrator-workers); agents are autonomous loops where the model decides its own next step. The tradeoff: workflows are predictable and debuggable; agents are flexible but non-deterministic.

Use an agent when you need autonomous planning, exploration, or recovery from unexpected states. Use a workflow when the task path is known. Use hybrid routing — a workflow that dispatches to an agent only when the task exceeds predefined scope.

async def route_task(task):
    if task.is_structured:
        return await structured_workflow(task)
    if task.needs_exploration:
        return await agent_loop(task, max_steps=10)
    return await prompt_chain(task, steps=["classify", "process", "respond"])

4. Fragile parsing of LLM outputs

You ask the model for structured output. It returns something close to JSON — but with a missing comma, an extra quote, or a slightly different key name. Your json.loads() call throws. Or worse: your regex silently parses garbage, and bad data flows into production.

LLMs are non-deterministic. Even at temperature 0, different API versions, prompt rephrasings, or model updates can shift output format. String-splitting and regex extraction are time bombs.

The fix: Enforce schema at generation time with structured outputs or function calling. Then validate at runtime with strict type checking. Fail fast on malformed output rather than letting garbage propagate.

from pydantic import BaseModel, Field

class AnalysisResult(BaseModel):
    sentiment: str = Field(pattern=r"^(positive|negative|neutral)$")
    confidence: float = Field(ge=0.0, le=1.0)
    key_topics: list[str] = Field(min_length=1, max_length=5)

def parse_agent_output(raw: str) -> AnalysisResult:
    """Structured output + Pydantic validation = fail-fast."""
    import json
    data = json.loads(raw)
    return AnalysisResult.model_validate(data)  # raises on bad schema

Pydantic sits between the LLM and your system, converting “probably JSON” into guaranteed typed data. If the model hallucinates a field, you catch it immediately — not in a customer-facing dashboard three days later.

5. Forgetting agents need planning

The most common agent loop is pick-tool → get-feedback → repeat. It’s a workflow with randomness, not goal-directed behavior. Without an explicit reasoning step, agents fall into infinite loops, repeat failed approaches, or reach a shallow conclusion without exhausting the search space.

Generic ReAct (Reasoning + Acting) is not a product. It’s a prompt pattern. Real agents need embedded planning — a structured reasoning step before each tool call that articulates the goal, selects the next action based on evidence, and defines what success looks like for that step.

async def agent_with_planning(task, tools, max_steps=15, token_budget=50000):
    history = []
    for step in range(max_steps):
        if total_tokens(history) > token_budget:
            return escalate(task, history)
        
        plan = await llm.generate(
            f"Current goal: {task.goal}\n"
            f"Evidence so far: {summarize(history)}\n"
            f"Available tools: {tools}\n"
            f"What is the next action and why?"
        )
        
        result = await execute_tool(plan.selected_tool, plan.arguments)
        history.append({"plan": plan, "result": result})
        
        if plan.stop_condition_met(result):
            return synthesize(task, history)
    
    return escalate(task, history, reason="max_steps_reached")

Progress checks and stop conditions are non-negotiable: max steps, token budgets, time limits, and escalation paths for when the agent is stuck.

6. Not starting with AI evals from day zero

Most teams have zero automated evaluation. They test manually in a notebook, ship, and hope. Then a prompt change or model upgrade silently degrades performance, and nobody notices until a customer complains.

“Vibe evals” — eyeballing outputs and saying “looks good” — don’t scale. Generic 1–5 helpfulness scores from an LLM judge are nearly useless — a score of 3.7 “helpfulness” tells you nothing about what to fix. AI systems decay. Prompt changes, model updates, distribution shifts — they all erode performance over time.

The fix: Task-specific binary metrics tied to real behavior, from day one. “Did the agent return the correct API call?” is measurable. “Was the response helpful?” is not.

eval_metrics = {
    "tool_selection_accuracy": lambda trace: trace.final_tool == expected_tool,
    "schema_validity": lambda trace: validate_schema(trace.output),
    "max_steps_respected": lambda trace: trace.steps <= MAX_STEPS,
    "correct_answer": lambda trace: extract_answer(trace) == GOLD_ANSWER,
}

def run_eval(agent_fn, test_cases):
    results = [evaluate(agent_fn, tc, eval_metrics) for tc in test_cases]
    for metric, score in aggregate(results).items():
        assert score >= MIN_THRESHOLD[metric], f"{metric} regressed: {score}"

Integrate evals into your dev workflow. Run them on every prompt change, every model update, every config tweak. If a metric regresses, the build fails — the same way a unit test regression would.

These six mistakes aren’t exotic. They’re boring, structural, and nearly universal. The good news: they’re all fixable with deliberate system design. Audit your current agent against this list — you’ll likely find two or three of these silently degrading your production reliability right now.

Sources: Paul Iusztin & Louis-François Bouchard, “Agentic AI Engineering Guide: 6 Critical Mistakes” (Decoding AI) · Paul Iusztin, LinkedIn · Anthropic, “Building Effective Agents” · Liu et al., “Lost in the Middle” (arXiv:2307.03172) · CAG paper (arXiv:2412.15605)

📖 Related Reads

  • ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
  • CodeIntel Log — code quality, debugging, and software engineering benchmarks

Cross-links automatically generated from NiteAgent.

← Back to all posts