This content originally appeared on DEV Community and was authored by Hady Walied
The 3 AM incident that taught me production agents need database-style transactions
The Problem Nobody Talks About
We’re all building AI agents now. LangChain makes it easy to prototype. Claude and GPT-5 are surprisingly capable. Everyone’s shipping agents to production.
But here’s what the tutorials don’t tell you: AI agents fail in ways traditional software doesn’t.
A bug in your web app? Roll back the deployment. A database migration fails? Transactions ensure consistency. But an AI agent that:
- Charges a customer the wrong amount
- Deletes production data
- Sends 1,000 emails to the wrong list
- Makes an irreversible API call
There’s no Ctrl+Z. You’re left manually reversing actions, apologizing to customers, and trying to piece together what the LLM was “thinking” from scattered logs.
I learned this the hard way while building automation for e-commerce systems. An agent processed refunds slightly wrong, and we spent hours manually correcting transactions. That’s when I realized: AI agents need the same safety guarantees as databases.
What Production Agents Actually Need
After deploying agents in real systems, here’s what I found you absolutely need:
1. Automatic Rollback (Like Database Transactions)
When something fails, every previous step should automatically reverse. Not with manual cleanup scripts. Not with post-incident fire drills. Automatically.
# If step 3 fails, steps 2 and 1 automatically rollback
@tool(compensate_with=cancel_hotel_booking)
def book_hotel(hotel_id: str, dates: dict):
return hotel_api.book(hotel_id, dates)
@tool(compensate_with=refund_customer)
def charge_customer(amount: float, customer_id: str):
return payment_api.charge(amount, customer_id)
This is called the Saga pattern (or compensating transactions). It’s battle-tested in distributed systems. But nobody’s applying it to AI agents.
2. Human Approval for High-Risk Actions
Some operations are too risky to let an LLM decide alone:
@tool(requires_approval=True)
def charge_credit_card(amount: float, card_id: str):
"""Requires human approval before executing"""
return payment_api.charge(amount, card_id)
One decorator. That’s it. The framework handles the approval flow, shows you full context, and waits for your decision.
3. Forensic-Grade Observability
When an agent fails (and they will), you need to answer:
- Which LLM call made the wrong decision?
- What was the exact input that triggered the error?
- How much did this cost in API tokens?
- What did the agent do before failing?
You need this information in 60 seconds, not 60 minutes of log spelunking.
# Find the bug in seconds, not hours
agenthelm traces show <trace_id>
agenthelm traces filter --status failed --tool-name process_payment
agenthelm traces export --output incident_report.csv
How I Built AgentHelm
I started with a simple question: What would a database-style transaction look like for an AI agent?
The Core Idea: Compensating Transactions
For every tool that modifies state, you define a compensating action:
from agenthelm import tool
@tool(compensate_with=reverse_refund)
def process_refund(order_id: str, amount: float):
"""Issue a refund to customer"""
return payment_api.refund(order_id, amount)
@tool()
def reverse_refund(order_id: str, amount: float):
"""Rollback: re-charge if something fails"""
return payment_api.charge(order_id, amount)
When the workflow executes:
- Agent calls
process_refund→ succeeds ✓ - Agent calls next tool → fails ✗
- Framework automatically calls
reverse_refund
- System returns to consistent state ✓
Real Example: Customer Refund Agent
Here’s a realistic scenario—processing customer refunds with validation, approval, and notifications:
# Step 1: Verify order exists
@tool(compensate_with=log_verification_failure)
def verify_order(order_id: str) -> dict:
return orders_api.verify(order_id)
# Step 2: Process refund (requires approval if >$100)
@tool(requires_approval=True, compensate_with=reverse_refund)
def process_refund(order_id: str, amount: float) -> dict:
return payments_api.refund(order_id, amount)
@tool()
def reverse_refund(order_id: str, amount: float) -> dict:
return payments_api.charge(order_id, amount)
# Step 3: Notify customer
@tool(compensate_with=send_cancellation_email)
def send_refund_email(customer_email: str, amount: float):
return email_api.send(customer_email, f"Refund of ${amount} processed")
@tool()
def send_cancellation_email(customer_email: str, amount: float):
return email_api.send(customer_email, "Refund was reversed due to error")
What AgentHelm guarantees:
- If verification fails → nothing happens (atomic)
- If refund >$100 → blocks for approval before executing
- If refund succeeds but email fails → refund is reversed, customer is re-charged
- Every step is logged with full LLM reasoning for compliance
The agent either completes all steps or none. Just like a database transaction.
The Architecture
Agent Task
↓
Execute Step 1 → Success
↓
Execute Step 2 → Success
↓
Execute Step 3 → FAILURE
↓
Compensate Step 2 (automatic)
↓
Compensate Step 1 (automatic)
↓
Return to Consistent State
The orchestrator tracks every step and its compensation function. On failure, it executes compensations in reverse order to safely unwind the workflow.
What I Learned Building This
1. Most Agent Failures Are Predictable
After analyzing dozens of agent failures, patterns emerged:
- Rate limits: External APIs hit rate limits
- Partial failures: Step 2 succeeds but step 3 fails
- Bad LLM decisions: Model misunderstands instructions
- Transient errors: Network timeouts, service unavailability
All of these can be handled with: retries, compensations, and structured traces.
2. Developers Want Boring Reliability Over Clever Features
Early feedback: developers didn’t ask for more LLM providers or fancier orchestration. They asked:
- “Can I see exactly why it failed?”
- “Can I prevent it from making this mistake again?”
- “Can I undo what it did?”
Production engineers want boring, predictable, debuggable systems.
3. The Trace CLI Is Surprisingly Powerful
Being able to run agenthelm traces filter --status failed and see every failure with full context is a game-changer for debugging. It’s like having a time-travel debugger for your agent.
The SQLite storage backend means you can use standard SQL tools to analyze patterns:
SELECT tool_name, COUNT(*) as failures
FROM traces
WHERE status = 'failed'
GROUP BY tool_name
ORDER BY failures DESC;
What’s Next
I’m working on v0.3.0 with:
- Plan-driven execution: LLM generates a plan, you approve it, then execution is deterministic
- Enhanced safety: Budget enforcement (token limits, time limits, I/O constraints)
- Multi-LLM support: Adding Claude and Gemini (both excel at tool use)
But I’m not shipping any of this until I validate that transactional safety is actually what production teams need.
Try It Yourself
If you’re deploying agents that touch real systems—payments, databases, APIs—I’d love your feedback.
Install:
pip install agenthelm
Quick example:
from agenthelm import tool
@tool(compensate_with=undo_action)
def risky_action(data: dict):
return api.modify(data)
@tool()
def undo_action(data: dict):
return api.revert(data)
Run:
agenthelm run --agent-file agent.py --task "Your task here"
GitHub: github.com/hadywalied/agenthelm
Docs: hadywalied.github.io/agenthelm
The Bigger Picture
AI agents are moving from demos to production. But most frameworks are still optimized for prototyping, not reliability.
We need to bring software engineering discipline to AI agents:
- Transactional guarantees
- Structured observability
- Human oversight for high-risk actions
- Reproducible execution
AgentHelm is my attempt at this. It’s not the most feature-rich framework. It’s the framework designed to prevent 3 AM production incidents.
If you’re building production agents, I want to hear from you:
- What breaks in production?
- What’s missing from existing frameworks?
- What would make you trust an agent with real business logic?
Drop a comment or open an issue on GitHub. Let’s figure this out together.
This content originally appeared on DEV Community and was authored by Hady Walied