# AI Support Agent Failure Modes and Early Catches
> Read this guide to learn the five failure types an AI support agent shows on a Shopify store, and the check that catches each one early.
Source: https://arbyn.app/blog/ai-support-agent-failure-modes-and-early-catches
Published: 2026-11-13
---
Your AI support agent is probably not failing loudly. It is more likely to answer with confidence, close the ticket, and leave a wrong refund or an invented shipping promise behind it.

The main ai support agent failure modes on a Shopify store are wrong facts, unauthorised actions, invented policies, missed handoffs, and off-brand tone. Each one leaves a trace in your helpdesk and in the Shopify admin. You catch them early by checking those traces on a fixed schedule, with a short sample each week.

## Key takeaways

- The most common AI support agent failures are wrong facts, unauthorised actions, invented policies, missed handoffs, and off-brand tone.
- Sample 20 AI-handled conversations a week and check each one against the Shopify order before you widen the agent's scope.
- Keep refunds, cancellations, and discount changes behind a human approval step, while low-risk address changes on unfulfilled orders can be automated.
- Set handoff triggers in your helpdesk and test them with fake conversations before going live.

## What are the most common ai support agent failure modes?

Five failure types cover most of what store owners see in practice. Each one has a visible symptom, so you can name it and assign it a check. The table below is the working list to keep next to your support queue.

| Failure mode | What it looks like | Early check |
| --- | --- | --- |
| Wrong fact | Agent quotes a delivery window or stock level that does not match the order or product page | Sample 20 order-status answers a week against the order status page |
| Unauthorised action | Agent issues a refund, cancels an order, or applies a discount code it should not touch | Compare agent actions in the Shopify admin order timeline with your refund rules |
| Invented policy | Agent promises a 60-day return window when your policy says 30 days | Check every policy claim against the page you published |
| Missed handoff | Agent keeps replying to a customer who asked for a person, or loops a damaged-item claim back to itself | Count threads with two or more agent replies after a "human" or "agent" request |
| Wrong voice | Replies are stiff, over-apologetic, or use phrases your customers never see from you | Read five replies a week out loud before you approve a wider rollout |

Wrong facts and invented policies look similar from the outside, but they need different fixes. A wrong fact usually means the agent read stale or missing data. An invented policy means the agent filled a gap with a guess, so check that the policy text it can read is complete and current.

## How do you catch a wrong answer before a customer does?

Start with a sample, not a full audit. Pull 20 conversations a week, split across order questions, returns, and product questions. Check each agent reply against the order in Shopify and the policy page it claims to follow.

1. Filter the week's AI-handled conversations in your helpdesk.
2. Pick 20 at random, with at least five from order-status questions.
3. For each one, open the order in the Shopify admin and check the fulfilment status, tracking, and dates the agent quoted.
4. Mark each reply as correct, incomplete, or wrong. Write one line on each wrong one: what the agent said and what was true.
5. If more than two of the 20 are wrong on the same topic, pause that topic for the agent, fix the source text, and only then turn it back on.

The threshold of two in twenty is a starting point for a small store, not a rule from any study. Adjust it to your ticket volume and to what a wrong answer costs you. A wrong delivery date on a gift order costs more than a wrong store hours answer, so you may want a lower threshold for order questions.

### A worked example of a weekly sample

Say a store handles 300 AI conversations in a week, and you pull 20 of them. Eighteen replies are correct. Two quote a delivery window that ignores the carrier's tracking update. That is exactly two wrong, which is at the threshold, so you watch the topic for another week. If a third comes back wrong, you pause order-delivery answers and fix the shipping text first.

Keep a simple log next to the sample. Use four columns: date, conversation link, topic, and result. After four weeks, sort by topic. If order-status answers are wrong in three of the four weeks, the problem is the source data, not the random bad reply.

## How do you stop an agent from taking actions it should not take?

Limit what the agent can do before you worry about how well it talks. Think of the agent as a staff member with a role. Shopify's staff permissions let you grant access by area, and the same thinking applies to any AI tool that acts inside your store. Only give it the actions you have already decided are safe.

Money-moving actions need a human gate. That covers refunds, cancellations, store credit, and discount codes that change what a customer pays. The agent can gather the order details and draft the decision, then hand it to a person to approve. [OWASP's Top 10 for LLM applications](https://genai.owasp.org/llm-top-10/?utm_source=arbyn.app&utm_medium=blog&utm_campaign=ai-support-agent-failure-modes-and-early-catches) lists excessive agency as a risk category for LLM-based systems, which is the reason for the gate. Treat that as the principle, not a checklist.

Changing a shipping address on an unfulfilled order is a different kind of action. It is low risk and easy to reverse, so you can automate it end to end and log it in the order timeline. The agent should confirm the new address back to the customer and note the change on the order. Keep approval steps for money and cancellations, not for a corrected street number.

### A worked example of an action gate

Say a store gets 400 support conversations a month, and 30 of them ask for a refund. Under a simple rule, the agent answers the policy question, checks the order is inside the return window, and drafts a refund for a person to approve. The agent never clicks refund itself. If the refund draft is wrong, you see it in one place, before the money leaves.

Now take the address case from the same month. Say 12 customers ask to change a shipping address, and 9 of those orders are still unfulfilled. The agent updates the address on those nine, writes a timeline note, and sends the customer a confirmation. The three fulfilled orders get a reply that explains the order has shipped and gives the tracking link. Those nine edits run without a person in the loop.

| Action | Who decides | Why |
| --- | --- | --- |
| Change shipping address on an unfulfilled order | Agent, logged in the order timeline | Low risk and reversible with one more edit |
| Answer a tracking or stock question | Agent, from live order data | Read-only, nothing is changed |
| Issue a refund or store credit | Person approves the agent's draft | Moves money and is hard to reverse |
| Cancel an order | Person approves the agent's draft | Stops fulfilment and may trigger a refund |
| Apply a discount code that changes the price | Person approves the agent's draft | Changes what the customer pays |

## What does a missed handoff look like, and how do you spot it?

A missed handoff is the most damaging failure, because the customer has already decided the bot is not helping. Look for threads where the customer asked for a person, or used words like "human", "agent", or "real person", and the agent kept replying. Also look for threads with several agent replies and no change in the customer's problem.

Set the handoff trigger in your helpdesk, not in the agent's memory. The trigger should fire on specific words and on a repeat-question count, for example after the same question comes up three times in one thread. Test it with a fake conversation before you go live, then check it weekly with the same sample you use for accuracy.

1. Write the list of trigger words your customers actually use, such as "human", "real person", and "agent".
2. Set a repeat-question count in your helpdesk. Three is a common starting point, and you can lower it if customers get frustrated fast.
3. Send three fake conversations: one asking for a person, one repeating a question, and one that should stay with the agent.
4. Confirm the thread moves to your team inbox and that the agent stops replying.
5. Count missed handoffs in your weekly sample. Any miss on a live thread goes on the failure log the same day.

> Sample reply: I can see you asked for someone on our team. I have passed this to a person, and they will reply to you on this thread. You do not need to repeat your order number.

Use the handoff reply as written, or adjust it to your voice. Notice that it says what happens next. It does not promise a reply time you cannot meet.

## How do you check that the agent sounds like your store?

Tone failures are quiet, but they erode trust. An agent that apologises three times, or uses a word your brand never uses, reads like a template. Read a handful of replies out loud each week. If you would not say it at the counter, rewrite the source text the agent learns from.

Voice training is a separate job from accuracy. The post on [how to train an AI support agent on your Shopify store's actual voice](/blog/how-to-train-an-ai-support-agent-on-your-shopify-store-s-actual-voice) covers how to build that source text, and it is worth reading before you change the tone settings.

## Which failure should you check first?

Check unauthorised actions first, then missed handoffs, then wrong facts. An unauthorised refund costs money the same day. A missed handoff loses the customer. A wrong fact is often caught by the customer, who replies to correct it, which gives you a signal. Voice problems can wait a week.

Many store owners also set up a weekly review that counts three numbers: actions the agent took, handoffs requested, and wrong answers found in the sample. If one number moves up two weeks in a row, that is your signal to tighten the rule behind it. The [guide on how to calculate ROI on an AI customer support agent for Shopify](/blog/how-to-calculate-roi-on-an-ai-customer-support-agent-for-shopify) shows how to weigh the cost of these errors against the time saved, which helps you decide how much review time to budget.

If you are comparing tools, look for one that logs every action and every handoff in one place. The [features page](/features) and the [integrations page](/integrations) list what is available, and the [compare page](/compare) sets the options side by side. Check whether the log shows the order data the agent read, not just the reply it sent.

## How often should you review the agent's work?

Review weekly during the first month, then move to a schedule that matches your ticket volume. The weekly sample of 20 is the minimum for accuracy. Add a separate look at every refund or cancellation draft, since those are where a single error costs the most.

A practical rhythm looks like this. On Monday, pull the sample and the action log for the past seven days. Mid-week, fix any source text and retest the handoff triggers. On Friday, write the one-line summary for the week and decide whether any topic stays paused. Most small stores can do this in under an hour once the log is set up.

## What should you do when a failure is confirmed?

Write down the failure in one line, fix the source text or the rule behind it, and retest with a few fake conversations before you reopen the topic. Do not just delete the bad reply. Keep the record so you can see whether the same failure mode keeps coming back.

Keep a short list of the wrong answers you have caught. Over a few months, that list becomes your training source for the agent and your checklist for the review. It also shows you which failure mode is growing, which is the number that matters most when you decide how far to widen the agent's scope.

## Frequently asked questions

### What are the most common ai support agent failure modes?

The common ones are wrong facts about orders or stock, actions taken without approval such as refunds, invented policies, missed handoffs to a person, and replies that do not sound like your store. Each leaves a trace you can check in your helpdesk and the Shopify admin.

### How often should I review AI support agent conversations?

A weekly sample of about 20 conversations is a practical start for a small store. Add a separate check on every refund or cancellation draft, and pause any topic where more than two of the 20 are wrong until the source text is fixed.

### Should an AI agent be allowed to issue refunds?

Most store owners keep refunds behind a human approval step. The agent can check the order and the return window, then draft the refund for a person to approve. That way a wrong amount is caught before the money leaves.

### Can an AI agent change a shipping address on its own?

Yes, for an order that has not been fulfilled. It is low risk and easy to reverse, so many stores automate it. Log the change in the order timeline so you can see what happened and correct it with one more edit if needed.

### How do I know if the agent missed a handoff?

Look for threads where the customer asked for a person and the agent kept replying, or threads with several agent replies and no progress on the problem. Set the handoff trigger in your helpdesk and test it with fake conversations first.

## Sources

- [OWASP: Top 10 for LLM applications](https://genai.owasp.org/llm-top-10/?utm_source=arbyn.app&utm_medium=blog&utm_campaign=ai-support-agent-failure-modes-and-early-catches)

Arbyn is an AI support agent for Shopify stores. It answers customers by email and on-site chat. Arbyn never moves money on its own: a refund request reaches you, and a refund or a cancellation you make from the conversation is carried out by Arbyn in Shopify. [See Arbyn in the Shopify App Store](https://apps.shopify.com/arbyn?utm_source=arbyn.app&utm_medium=blog&utm_campaign=ai-support-agent-failure-modes-and-early-catches).
---
## Pricing

Plans and prices are listed at https://arbyn.app/pricing.


## Channels

Arbyn handles **support email** and **on-site live chat**. That is the complete list.

SMS, Instagram DMs, Facebook Messenger, WhatsApp and Voice are on the roadmap and are NOT live. Arbyn does not edit orders or change line items. Money-moving actions never run without the store owner: refunds and returns reach your team as requests; once your team approves one, Arbyn does it in Shopify. You can also cancel an order, send a gift card or issue a discount code from the conversation, and Arbyn does it in Shopify. Shipping address changes can run on their own once you switch that on.

## What Arbyn does on a Shopify order

- **Change the shipping address**: Arbyn does this on its own once you switch that on. Arbyn updates the shipping address on the Shopify order itself, inside the conversation, and tags the order so your team can see it.
- **Cancel an order**: From any conversation you can cancel an order, and Arbyn does it in Shopify. Anything that moves money waits for the store owner. That is a deliberate control, not a missing feature. Arbyn does not cancel an order on its own: you cancel from the conversation, and Arbyn fires Shopify's order cancellation.
- **Issue a refund**: Refund requests reach your team, and from any conversation you can refund and Arbyn does it in Shopify. Arbyn never issues a refund on its own: a refund request is sent to you. From the conversation you can refund, and Arbyn files the refund in Shopify.
- **Apply a discount**: From any conversation you can issue a discount code, and Arbyn does it in Shopify. Arbyn creates a real Shopify discount code from the conversation, and the shopper gets a checkout with the code already on it.
- **Send a gift card**: From any conversation you can send a gift card, and Arbyn does it in Shopify. Arbyn creates the gift card in Shopify when you send it from the conversation.
- **Start a return**: The return request reaches your team. Once they approve it, Arbyn creates the return in Shopify. The AI gathers the order and items and hands your team a one-click approval. Once approved, Arbyn creates the return in Shopify. Returns are never autonomous.
- **Look up a gift card or store-credit balance**: Arbyn does this on its own. "Do I have store credit left?" is a question most support tools answer with a human. Arbyn reads the balance itself, for a verified customer or from the code they give you, and reports the masked card, the balance and the expiry. If there is no card, it says so rather than guessing.
- **Handle a subscription question**: You choose what it does. Arbyn knows which of your products are sold as a subscription, shows that on the product card in the conversation, and sends a subscriber to their subscription management page to pause, skip or cancel. It answers how your subscriptions work from your own knowledge, but it does not read an individual customer's contract, so it will not state their renewal date or status. Most cancels are a customer with product piling up, and the fix is getting them to the page where they can slow the cadence down. Reading the contract itself is on the roadmap.
- **Answer support email and live chat**: Arbyn handles both channels. Arbyn reads every inbound support email and every chat, works out the intent, pulls the live Shopify context, and replies in your brand voice.

Money-moving actions never run without the store owner: refunds and returns reach your team as requests; once your team approves one, Arbyn does it in Shopify. You can also cancel an order, send a gift card or issue a discount code from the conversation, and Arbyn does it in Shopify. Shipping address changes can run on their own once you switch that on.