AI Support Agent Launch Checklist for Observe Mode

An AI support agent in observe mode reads incoming tickets and writes draft replies, but nothing reaches a customer until a person has graded them. Run it that way for one to two weeks, grade each ticket category against what your team would have said, then switch on live replies one category at a time.

Key takeaways

• Run the AI support agent in observe mode first so every draft is read by a person before a shopper sees it.
• Grade drafts by ticket category over at least 50 tickets, and switch each category to live only when it meets a bar you set in advance.
• Address changes on unfulfilled orders are low risk and reversible, so they can be automated once the agent confirms fulfillment status.
• Keep refunds and cancellations under human approval, and keep current policy pages and a grading log, since stale policies and missed thread context cause most early errors.

A launch checklist is only useful if you run it in order. The steps below cover setup, which tickets to start with, how to grade, how to handle address changes, and when to switch a category to live.

What does observe mode actually mean?

Observe mode means the agent sees the same customer messages your team sees, writes a reply, and stores that reply where a person can read it. Nothing reaches the shopper. Your staff keep sending replies as they always did, and they can compare their own reply with the draft side by side.

This is different from a test store or a sandbox. You want real tickets, real order data and real customer wording, because that is where the agent will fail. A fake inbox teaches you very little about how a shopper writes "where is my stuff???" late at night.

Observe mode also lets you see the agent's decision, not just its text. For each ticket you want to know which order it looked up, which policy it quoted, and whether it wanted to hand the case to a person. Those details are where most early problems show up.

How do you set up observe mode before the first ticket?

Set up the agent with the minimum access it needs, then write down what it is allowed to do. Most of the risk comes from permissions you forgot to remove. Start with read access to orders, the order status page and your help content, and add write actions only after observe mode has earned them.

1. Confirm which staff account the integration uses. In the Shopify admin, check the role assigned to it under Settings, then Users and permissions. Shopify's staff permissions page explains that available permissions depend on your pricing plan and sales channels, so check what your role actually includes.
2. Turn off any automatic send option the tool offers, so every draft lands in a review queue instead of the customer's inbox. Confirm it by sending a test message from your own email and checking that nothing goes out.
3. Connect your order data and confirm the agent can see the order number, fulfillment status, tracking number and shipping address for a test order you placed yourself.
4. Load your return policy, shipping policy and FAQ pages. Check that the agent quotes the current versions, not an old PDF still sitting in a help doc.
5. Write a short list of topics the agent may never answer alone. Refunds, cancellations, chargebacks, legal complaints, damaged item claims and anything involving a medical or safety issue belong on that list.
6. Set a daily review slot. Observe mode produces drafts only if someone reads them, so put a 30 minute block on the calendar for the first two weeks.

Write the list down before go-live day, and keep a copy with the grading log. When someone asks six weeks later why a category is live, the answer should be on paper, not in memory.

Which tickets should the agent draft first?

Start with high-volume, low-risk questions where the answer comes from order data and a fixed policy. Order status, tracking requests and "when will I get it" questions are the usual first group. They are frequent, the facts are checkable, and a wrong draft costs you a few seconds of review instead of money.

Leave out anything that moves money in observe mode, even as a draft. You still need to see how the agent handles a refund request, because that tells you whether it will invent a policy under pressure. But refunds should not be the first category you switch on.

Ticket type	Risk if the draft is wrong	Observe mode first?	Go-live rule
Order status and tracking	Low: a few minutes of customer confusion	Yes	Go live once the category clears your bar over at least 50 graded drafts
Shipping address change on an unfulfilled order	Low and reversible while the order has not been fulfilled	Yes	Can be automated once the agent confirms fulfillment status first
Product questions from the store FAQ	Low to medium: a wrong size or material claim	Yes	Go live per topic, after grading each topic separately
Refund and return requests	High: money leaves the store	Yes, drafts only	Keep a person approving every refund or return
Order cancellations	High: fulfillment and payment may already be underway	Yes, drafts only	Keep a person approving every cancellation

The table is a starting point, not a rule for every store. A store that sells made-to-order goods may need to treat order status differently, because the fulfillment step comes later than the customer expects. Adjust the table to your own order flow before you write the go-live bar.

How should you grade the drafts?

Grade each draft on three things: were the facts right, was the policy right, and would a shopper be satisfied. Use a simple three-label scale: send as is, send with edits, or do not send. Track the counts for each ticket type, not one blended score, because an agent can be excellent on tracking and poor on returns at the same time.

Keep a log with the ticket ID, the category, the grade, and one line on what went wrong. The line matters more than the grade. "Quoted 14 day return window, policy says 30" tells you exactly what to fix in the knowledge base, and "missed the second tracking number" tells you the agent needs a rule for split shipments.

Set the bar before you grade anything. Pick a percentage that matches how much a wrong reply costs you in that category, write it in the log, and do not move it after you see the results. A stricter bar for product claims than for order status is a normal choice.

A worked example of grading

Suppose a store gets 400 tickets a month, and 120 of them are order status questions. The owner grades 60 of those drafts during week one and sets a bar of 50 drafts marked send as is out of 60, which is about 83%. If 56 are marked send as is, the category clears the bar with room to spare, but the owner still reads the four misses first.

Suppose three of those four misses are orders split across two shipments. The owner adds a rule telling the agent to list every tracking number on the order and to say which items are in each box. The next batch of 60 drafts goes through clean, and the owner moves that category to live.

The numbers in that example are invented round figures to show the method. Your own volume and bar will differ. The point is to read the misses, not just count them.

Grade the thread, not only the last message

Read the full conversation before you grade a draft. A shopper who writes "same as before" is pointing at an earlier message, and a draft that ignores it looks careless even when the facts are right. Mark those drafts as do not send, and log the missing context as its own failure type.

What address changes need a different check?

A shipping address change on an unfulfilled order is low risk and easy to reverse, so it can be automated. The agent can make the change once it confirms the order is unfulfilled. Your observe mode job is to check that it reads that status correctly and writes the confirmation in the right place.

The check is about the order, not about a manager. Before the agent edits anything, it should confirm the order is unfulfilled. If the order is already fulfilled, the agent should say so and point the customer to the carrier tracking link instead. Grade the drafts for accuracy: did the agent name the right order, did it show the right confirmation text, and did it avoid promising a delivery date the carrier does not give.

Use a reply like this as your standard for the confirmation text, and check that the agent sends the same wording with the order number in it:

Sample reply: Your order #1048 has not shipped yet, so we have updated the delivery address to the one you sent. You will get a new confirmation email for the change. If you want to add or change anything else, reply here before the order leaves our warehouse.

Once a category like this is live, watch the first week of changes closely. Look for orders where the address was changed after the fulfillment status had already moved, since that is the case where the customer will get a delivery to the wrong place.

When should you switch from observe mode to live replies?

Switch a category to live only when three things are true: the grades meet your bar, the failures are understood, and you know what happens when the agent is wrong. Do it one category at a time. Switching everything on the same day makes it impossible to tell which change caused a bad reply.

Keep the rollback simple. You should be able to turn a category back to drafts in a few minutes, and your team should know where that switch lives. Check the feed each morning for the first week after going live, and sample a handful of sent replies rather than trusting the grade alone.

• Set a fallback path: any ticket the agent cannot answer goes to a person with the conversation attached, not a dead end.
• Watch for repeated customers. If the same shopper writes three times, the agent should stop and hand over.
• Review any reply that quotes a policy, a date or a price. Those are the replies that cause refund disputes later.

The AI risk guidance from NIST suggests treating an AI system as something you measure and manage over its life, not something you approve once. The NIST AI Risk Management Framework is written for a wide range of organizations and is voluntary, so you can borrow its habit of ongoing review without adopting the whole framework.

What goes wrong in the first two weeks?

The most common failure is a confident answer built on stale information. The agent reads an old shipping page, quotes a return window you changed last spring, or assumes a discount code is still live. Keep your policy pages current and check them before each review session, not after a complaint.

The second failure is the agent following instructions it should not follow. A shopper can paste text into a message that tries to change how the agent behaves, and a draft built on that text can promise something your policy does not allow. The OWASP Top 10 for LLM applications lists prompt injection as a leading risk for this kind of system, so check a few drafts for replies that follow odd instructions from the customer's message.

The third failure is missing context from the conversation, which the grading section covers. For a fuller list of error patterns and how to catch them early, see AI Support Agent Failure Modes and Early Catches.

Who should own the launch?

One named person should own the observe period, usually the support lead or whoever answers the most tickets. That person runs the daily review, keeps the grading log, and decides when a category moves to live. Without an owner, drafts pile up unread and the agent never leaves observe mode, or it goes live before anyone checks it.

The owner does not need to be technical, but they do need the authority to pause a category when the grades slip. If you are weighing whether this role should manage the AI or be replaced by it, Should Your First Shopify Support Hire Manage the AI, or Replace It? works through that choice.

If you are still choosing between tool types, AI Shopping Assistant vs AI Support Agent: What's the Difference explains which job each one does, which helps you decide what the observe period should cover. You can also check how connections are set up on the integrations page before you grant any access.

How do you know the launch worked?

You know the launch worked when a category is live, the grades have held for two review cycles, and your team spends less time on that category without a rise in reopened tickets. Measure the reopen rate for each live category, since a reply that looks fine but leaves the shopper writing again is a failure too.

Compare the before and after numbers for the same category over the same weeks. If order status tickets dropped from 120 a month to 70 in a worked example, that is 50 tickets your team no longer answers by hand, and you can put a value on that time using the method in How to Calculate ROI on an AI Customer Support Agent for Shopify. Keep the observe period log as well, because it is the best record you have of what the agent got wrong and how you fixed it.

Frequently asked questions

How long should an AI support agent stay in observe mode?

There is no fixed number of days that fits every store. A common approach is one to two weeks, long enough to grade at least 50 drafts in each category you plan to switch on. Stay longer if a category keeps missing your bar, and move faster only when the grades hold steady across several review sessions.

Can the agent send replies while in observe mode?

It should not. In observe mode every draft goes to a review queue, and your staff decide what to send. Turn off any automatic send setting in the tool before the first ticket arrives, and confirm it with a test message you send yourself.

Should refunds be approved by a person even after going live?

Yes, for most stores. Refunds and cancellations move money or stop fulfillment, so a wrong answer is expensive. Keep a person approving those actions, and let the agent prepare the case with the order details and the policy quoted for your team to confirm.

Does a shipping address change need human approval?

Not usually. On an unfulfilled order, an address change is low risk and easy to reverse, so it can be automated once the agent confirms the order is unfulfilled. Keep human approval for refunds, cancellations and other money-moving actions.

What access does the agent need in the Shopify admin?

It needs to read orders, customers and fulfillment status, and only the write access for the actions you have approved. Check the role assigned to the connection under Settings, then Users and permissions, since the role controls what it can see and edit.

What is a good accuracy bar before going live?

Set the bar yourself before grading starts, based on how much a wrong reply costs you. Many store owners pick a stricter bar for product claims than for order status. Grade at least 50 drafts per category so one lucky week does not decide the switch.

Sources

• NIST: AI Risk Management Framework
• OWASP: Top 10 for LLM applications

Arbyn is an AI support agent for Shopify stores. It answers customers by email and on-site chat. Arbyn never moves money on its own: a refund request reaches you, and a refund or a cancellation you make from the conversation is carried out by Arbyn in Shopify. See Arbyn in the Shopify App Store.