An Autonomous Support Agent That Reads the Ticket, Fixes the Order, and Knows When to Call a Human

An autonomous ecommerce customer support and order management agent for a retailer based in the US, designed, trained and shipped by our AI agent development team.

Agent console showing a resolved ticket beside the live order record

Overview

Our client is a direct to consumer retailer based in the US, selling through its own online store and a few marketplaces. Its support desk had grown at roughly the same rate as its order volume, which sounds reasonable until you sit with the team for a day and notice that most of what they type is copied from another browser tab. We see the same pattern across almost every enterprise ecommerce business we work with: the store scales, and the people answering questions about the store scale with it, one hire at a time.

The tickets themselves were rarely difficult. Customers wanted to know where a parcel was, whether an address could still change, how to send something back, or why their card showed two charges. Almost every answer already existed somewhere in the stack, in the order record, the warehouse system or the carrier feed, and a person was acting as a very expensive cable between those systems and the customer.

The client had already tried an FAQ chatbot. It could recite the returns policy word for word, and it couldn’t do a single thing about an actual return. Conversational AI that can only talk ends up being a slower route to the same human agent, and customers figure that out fast. That gap is why we argued for an agent instead of another round of chatbot development.

What went live is an agentic AI system that reads every incoming ticket across email, live chat and SMS, works out what the customer actually needs, pulls the live order, takes the action if policy allows it, and replies in the brand’s voice. Address changes, cancellations, return labels, partial refunds, replacements for items damaged in transit. When it isn’t sure, or when the money involved crosses a line the client set, it passes the ticket to a person with the summary already written. Underneath all of it sits the part nobody demos: a machine learning and MLOps layer that keeps the agent accurate long after launch day.

The Challenge

Four problems shaped the architecture before anyone wrote code.

1

Most tickets asked something the systems already knew

Most of the monthly volume was order status, delivery estimates and address questions. Each one meant a support agent opening the helpdesk, then the store admin, then the carrier’s tracking page, then writing an answer by hand. During the holiday rush, the queue backed up for hours, and the team spent its days on the questions that needed the least judgment while the ones that needed real thought sat waiting.

2

Answering is easy, acting is where it gets risky

Customers expect address changes to happen before a parcel leaves the warehouse, requiring write access to order management, warehouse systems, and payment gateways. This level of integration often falls under ecommerce development services, not just AI development. Giving an AI agent control over refunds and inventory raises valid security concerns. Businesses need clear rules defining what the agent can access, modify, and approve before deploying it.

3

Policy lived partly in documents and partly in people

The written returns policy covered the standard cases. The exceptions, such as final sale items, VIP customers, parcels marked delivered that never arrived and damaged goods with no photos, lived in the heads of a few senior agents and in a team chat channel full of past decisions. An agent trained only on the written policy would have been confidently wrong on exactly the tickets that matter most.

4

An AI model that is right in March can be wrong by November

Catalogs change, carriers change, holiday return windows come and go, and a new product line brings questions the model has never seen. Accuracy slides a little every week and nobody notices until the complaints do. Without evaluation and retraining built into the system itself, any agent we shipped would follow that same curve.

Our Approach

We designed backwards from the riskiest action, not the most common question.

Every action the agent might take was ranked by how bad it would be if it went wrong. Reading an order status sits at the bottom. Changing an address before fulfillment sits in the middle, because it can be undone. Issuing money sits at the top. Autonomy was then granted tier by tier, and each tier had to earn its way out of human review with data rather than with a good demo.

Discovery started with an AI consulting workshop and months of historical tickets. We clustered them into intents, mapped each intent to the systems it touched, and sat with the support team to write down the unwritten rules. That last part took longer than the clustering, and it was worth every hour.

Delivery ran in two week sprints: data and retrieval first, then the agent and tool layer, then evaluation and the operations console, then a shadow period before anything went live. In shadow mode the agent drafted a reply and a proposed action for every ticket, sent nothing, and the human team graded its work. Those grades became our first labeled evaluation set and, just as usefully, gave the support lead a reason to trust the thing. Rollout then went intent by intent, starting with order status and ending with refunds.

The team was a dedicated squad: a delivery manager, Python developers on the agent and tool layer, an ML engineer, a data engineer, a frontend developer for the console, QA and DevOps. The store integrations went to our ecommerce developers, who already knew the platform’s quirks.

Solution Delivered

A supervisor that routes, and specialists that act

A supervisor that routes, and specialists that act.

A supervisor agent built on LangGraph classifies each ticket and passes it to one of five specialist agents: order status, order changes, returns and refunds, product questions, and escalation. Each specialist gets a short list of tools and nothing else. We’d push almost any client toward this pattern, because one giant agent holding thirty tools makes a great demo and a terrible production system, and when it misbehaves nobody can tell which part of the prompt caused it.

A handoff where the customer never repeats themselves

A handoff where the customer never repeats themselves.

When the agent escalates, the human sees a short summary, the order timeline, what the agent already tried and a suggested reply. The customer is told plainly that a person is taking over. That small detail matters more than most teams expect, and our piece on how AI customer service agents are changing support goes deeper into why the handoff is usually where these systems win or lose trust.

Model routing that keeps cost per ticket sensible

Model routing that keeps cost per ticket sensible.

Not every ticket deserves the largest model. A compact classifier, fine tuned on labeled tickets from the client’s own history, reads intent, urgency and sentiment in milliseconds, and a midsize LLM writes the routine replies. The larger model only comes in for tickets that bundle several problems together, or for customers who are clearly upset. Getting this split right was a big part of the LLM development effort, and it’s the main lever on cost per ticket.

Confidence scores that actually decide something

Confidence scores that actually decide something.

Each proposed action gets a score built from the classifier’s probability, the relevance of what retrieval returned and the result of the policy check. Above the threshold the agent acts and replies on its own; below it, the ticket lands in a human queue with a draft already waiting. Thresholds are set per intent from shadow mode data, so a refund needs far more certainty than a tracking update.

Retrieval that knows which policy is current

Retrieval that knows which policy is current.

The agent answers from a retrieval augmented generation (RAG) layer built over the returns policy, shipping rules, the product catalog, sizing guides and a library of past resolved tickets. Everything sits in Qdrant with metadata for region, product line and effective date, so an expired holiday policy simply can’t be retrieved. Hybrid search, keyword plus dense embeddings with a reranking step, handles the awkward queries, like a customer pasting half an SKU into the chat. Every answer that leans on policy logs the source passage it used.

Guardrails inside the tools, not bolted on afterwards

Guardrails inside the tools, not bolted on afterwards.

Every action is a typed Python function, validated with Pydantic, wrapping the storefront’s admin API, the warehouse management system, the carrier tracking APIs and the payment gateway’s refund endpoint. The policy checks live inside those functions: refund caps, return windows, whether the parcel has already left the warehouse. The model can request an action, but the tool decides whether it happens. Every write carries an idempotency key, so a retried call can never refund the same order twice. Our AI integration work follows this rule everywhere, since a prompt is a suggestion and a function is a contract. Every integration also sits behind its own adapter, so the same agent runs on a Shopify Plus or Adobe Commerce store with only the adapter changing.

Brand voice with hard limits

Brand voice with hard limits.

Replies come from the LLM conditioned on the client’s tone guide and a set of approved examples, then pass a final check for anything the brand never says: delivery dates the carrier hasn’t confirmed, discount codes that aren’t live, promises about stock. It’s the same discipline we apply on any generative AI development project where the output goes straight to customers.

Three Modules Nobody Screenshots

The parts of an agent build that never make the sales deck but decide whether it still works a year later.

An evaluation suite that gates every release

A golden set of anonymized real tickets, each with a known correct outcome, runs in CI against every prompt change, model swap or tool update. If accuracy on refunds drops, the release doesn’t ship, full stop. Honestly, this is the module that separates teams who run AI agents in production from teams who only demo them.

A drift monitor for intents and retrieval

A weekly job compares the mix of incoming intents and the quality of retrieval hits against a baseline. When the client launches a new product line and a new kind of question starts showing up, the monitor flags it before the escalation rate does. Flagged clusters feed a labeling queue, and the intent classifier retrains from there. The reporting borrows from our data analytics practice: a few charts the support lead actually reads, rather than a dashboard nobody opens.

An operations console the support lead owns

Thresholds, refund caps, autonomy per intent, and a kill switch for every action type all live here. During a big sale the support lead can move refunds back to human approval in one click without filing an engineering ticket. Every autonomous action is logged with its reasoning trace, the tools it called and the data that came back, which keeps the occasional “why did it do that” conversation short and factual.

Key Features

Agentic ticket resolution

Agentic ticket resolution

Reads, decides, acts and replies across email, chat and SMS.

Live order actions

Live order actions

Address edits, cancellations, return labels and partial refunds, all within policy.

Specialist agents

Specialist agents

A supervisor routes each ticket to a narrow agent with its own tools.

Policy aware retrieval

Policy aware retrieval

RAG filtered by region and effective date, with every source logged.

Confidence gated autonomy

Confidence gated autonomy

Per intent thresholds decide between sending and human review.

Handoff with full context

Handoff with full context

Summary, order timeline and a draft reply waiting for the human.

Cost aware model routing

Cost aware model routing

A small classifier first, a larger LLM only when the ticket needs one.

Release evaluation in CI

Release evaluation in CI

A golden ticket set runs before any prompt or model change ships.

Drift monitoring

Drift monitoring

Weekly checks on intent mix and retrieval quality trigger retraining.

Console with kill switches

Console with kill switches

Support leads control autonomy and caps for every action type.

What the Research Says About Agents Like This

Independent research points to what agentic support systems built this way are expected to deliver.

80%

Of common customer service issues that Gartner expects agentic AI to resolve without human help by 2029.

30%

Reduction in operational costs that Gartner projects alongside that shift.

14%

More issues resolved per hour when support agents work with an AI assistant, according to an NBER field study of customer support teams.

34%

Productivity gain for newer, less experienced agents in the same study, which is why the hand-off summaries matter as much as the automation.

On this build we track the same ground from day one: autonomous resolution rate, median first response time, cost per ticket, CSAT on agent resolved versus human resolved tickets, evaluation accuracy and uptime. Release testing ran through our software testing and QA practice, including adversarial prompts written specifically to talk the agent into refunds it shouldn’t give.

Technology Stack

LangGraph LangChain Python

Agent orchestration

LangGraph and LangChain, Python

Azure OpenAI for generation urgency

Language models

Hosted GPT models via Azure OpenAI for generation, Llama fine tuned for intent and urgency

Retrieval

Retrieval

Qdrant vector database, hybrid keyword and dense search, reranking

FastAPIPydantic Redis

Backend and tools

FastAPI, Pydantic, Celery workers, Redis

PostgreSQL BigQuery

Data

PostgreSQL for tickets and audit logs, BigQuery for analytics

Commerce integrations

Commerce integrations

Storefront admin API, warehouse management system, carrier tracking APIs, payment gateway refunds

Helpdesk

Helpdesk

The client’s existing helpdesk for ticket intake and human queues

MLflow GitHub Actions Langfuse

MLOps

MLflow for experiments and the model registry, evaluation suite in GitHub Actions, Langfuse tracing

Prometheus Grafana Sentry

Monitoring

Prometheus, Grafana, Sentry

Docker Kubernetes AWS

Deployment

Docker and Kubernetes on AWS

React

Operations console

React

The Part People Underestimate: This Is an MLOps Problem Wearing a Chatbot Costume

Most buyers evaluate an AI support agent the way they would evaluate a chatbot. They try the demo, ask it ten questions and check whether the answers sound good. That tells you almost nothing about month six, when the catalog has turned over, a carrier has changed its tracking format and the summer returns policy has quietly become the winter one. It’s also why packaged support bots tend to stall, and why custom AI software development pays off once an agent has to act inside your own systems.

A large share of this build went into deliberately boring machinery: versioned prompts, a model registry, a golden evaluation set that grows from real escalations, tracing on every tool call, and a retraining loop for the machine learning model behind intent classification that fires when drift crosses a threshold. Every ticket a human resolves after escalation becomes a labeled example for the next training run. That loop is the part of AI and ML work that clients rarely ask for at the start and almost always value most a year in.

There’s a customer data angle too. The agent writes a structured record of every conversation (intent, sentiment, resolution, order value at stake), which is a far richer signal than a helpdesk tag. Getting that record into the CRM is the same problem any CRM software development project exists to solve, and it hands the marketing team something genuinely new to work with. If you’re still sorting out where agents end and generative assistants begin, our breakdown of agentic AI vs generative AI is a decent place to start.

What’s Queued Next

The platform shipped with room left in it on purpose. On the client’s roadmap:

Proactive outreach, where the agent contacts a customer the moment a carrier delay shows up instead of waiting for the ticket. A voice channel for phone support. Return risk scoring at checkout, built on the same predictive analytics thinking many ecommerce teams already apply to demand. Support for more languages. And an assistant inside the client’s mobile app, which sits naturally alongside our AI app development work.

None of this needs rework, because the supervisor and specialist structure was built from the first sprint to take on new agents and new channels.

Conclusion

The client didn’t need a smarter FAQ. They needed something that could do what a good support agent does on a routine ticket, safely, at three in the morning, in the middle of a sale.

The decisions we’d point to as the reasons it works are fairly unglamorous: designing around the riskiest action instead of the most common one, putting policy inside the tools rather than inside the prompt, and treating evaluation and drift as product features instead of something to sort out after launch.

If you run a retail or ecommerce business where the support desk grows every time the order book does, you can hire dedicated developers from us the way this client did, as one squad that stays with the build from the first ticket audit through launch and the retraining cycles after it.

Your Support Desk Shouldn’t Grow as Fast as Your Orders

Let’s talk about which tickets an AI agent could safely take off your team’s plate.

Schedule a Free Consultation

FAQs

The timeline depends more on how many systems the agent has to act on than on how many questions it answers, since each integration (store, warehouse, payments, helpdesk) needs sandbox work, error handling and policy checks of its own. Plan for a few weeks of shadow mode on top of that, where the agent drafts every reply but sends nothing until the team trusts it.
Both use artificial intelligence, but a chatbot answers questions from a script or a knowledge base, while an AI agent can also take actions in your systems, such as changing an address, creating a return label or issuing a refund, and it decides which steps to take based on the ticket in front of it. In practice, the agent resolves the ticket while the chatbot mostly explains how the customer could resolve it.
Yes, as long as the safety lives in code rather than in the prompt. In this build, refund limits, return windows and order state checks are enforced inside the refund tool, every call carries an idempotency key, refunds need a higher confidence score than any other action, and the support lead can switch refunds back to human approval instantly.
Yes. The tool layer is a set of adapters, so the agent itself doesn’t care which platform sits underneath it. The same approach works with Shopify Plus, Adobe Commerce and custom order systems, and the adapter is usually the smallest part of the job.
We use three layers: answers about orders come only from tool results and never from the model’s memory, answers about policy come from retrieval with the source logged, and a final output check blocks unconfirmed delivery dates, inactive discount codes and anything else on the client’s list of things never to say. The evaluation set also includes questions that have no valid answer, so we can confirm the agent admits when it doesn’t know.
Prompts and models are versioned, every change runs against the golden evaluation set in CI, and every tool call is traced. A weekly drift job watches intent mix and retrieval quality, and when it flags a shift, new tickets get labeled and the intent classifier retrains. Without that loop, accuracy decays as the catalog and policies change underneath the agent.
Packaged tools are fine for FAQ deflection. Once the agent needs to act on your own order data, follow your own exception rules and sit inside your own audit requirements, custom AI software development is usually the better fit, and it’s worth running the cost comparison over three years rather than one.
Yes. Some clients bring us in for the agent and evaluation layer while their own team owns the store, and others hire ecommerce developers from us for the integration work alone. Engagements run hourly, as a single dedicated engineer, or as a full squad.

Show Similar Projects

Goods Fulfill

Goods Fulfill

An online store runs on a dropshipping business model for electronics

Rebellious

Rebellious Fashion

One-Stop Solution for overall women’s fashion needs

Proship

Proship

A logistics aggregator app provides a cost-effective shipping experience for ecommerce.