Automating Order-Status Questions for a Print-on-Demand Company

Merchants at a print-on-demand company kept asking support the same question: “Where is my order?” It made up about 15% of incoming conversations.

When I compared order-status questions before and after release, deflection improved for those questions and overall support deflection rose by 12 percentage points.

Here, deflection means a conversation resolved without being handed to a human agent. In other words, 12 more conversations out of every 100 were resolved without a handoff. With 10,000 conversations entering each month, that movement represented capacity equivalent to roughly 1,200 fewer conversations needing an outsourced support agent.

The people asking were merchants: businesses using the platform to make and ship products under their own brands. Their customers were the people waiting for those products. The service sat behind Fin (opens in a new tab), an AI support agent inside Intercom Messenger (opens in a new tab). Fin could ask this service for an order explanation, but it could not browse the four underlying systems itself.

The interesting part was not getting a model to write an order update. It was checking that the order belonged to the merchant, deciding what the merchant was allowed to know, and assembling only that information before the model saw it.

The rest of the case study follows those decisions: what the service fetched, what it was allowed to show, and how it stayed within Fin’s response deadline.

The problem was scattered context, not tracking

The roles shaped the access rules

The company did not speak directly to the merchant’s customer through this support experience. The person talking to Fin was the merchant, who could see their own order and its delivery details but should never be able to see another merchant’s order.

A single merchant order could become several packages. Each package could contain several items, but everything in one package came from the same production facility. Merchants knew the order ID shown in the platform. Production facilities and internal operations worked with an internal package ID. That package ID was not a carrier tracking number or a tracking link.

flowchart TD
accTitle: One merchant order can contain several internal packages
accDescr: A merchant order can split into package A and package B. Each package is produced by a different facility before reaching the merchant's customer.
Merchant[Merchant] --> Order[Merchant order ID]
Order --> PackageA[Package A: internal package ID]
Order --> PackageB[Package B: internal package ID]
PackageA --> FacilityA[Production facility 1]
PackageB --> FacilityB[Production facility 2]
PackageA --> Customer[Merchant's customer]
PackageB --> Customer

That object model explains why “Where is my order?” was not one question. One package could still be in production while another had shipped. A delay might come from a machine outage at one facility, a carrier scan, or a technical problem preventing a print file from reaching production.

The answer lived in four places

The context needed to explain an order could be found in:

1. Internal order and package data.
2. Tracking events available through the company's AfterShip (opens in a new tab) integration.
3. Conversations between production facilities and an operations team in a separate Intercom (opens in a new tab) workspace.
4. Jira (opens in a new tab) issues describing technical problems affecting particular orders or packages.

Human support agents could not see all four directly. Support and operations worked in separate Intercom workspaces. Giving the whole support team Jira seats would have widened access and added recurring per-user cost, just to expose one narrow slice of order context. Public carrier pages sometimes exposed less information than the shipping data the company already received through its backend.

The company did not lack information. It lacked one controlled way to assemble the right information for the right merchant.

I checked whether better information would actually prevent handoffs

My three questions for support automation

I like taxonomies, but I now prefer to start with a plain question: what would the system need permission to do?

I sort support work into three practical levels:

1. Explain something. The answer comes from shared information and can be the same for everyone: “How does shipping work?”
2. Look something up. The answer depends on who is asking: “Where is my order?”
3. Change something. The resolution modifies an account, an order, or money: “Cancel this order” or “Issue a refund.”

flowchart LR
accTitle: Three levels of support automation responsibility
accDescr: Support work moves from shared explanations to merchant-specific lookups to changes that affect accounts, orders, or money. The controls get stronger at each level.
Request["Support request"] --> Level{"What does the system need to do?"}
Level --> Explain["Explain something: shared information"]
Level --> Lookup["Look something up: identity and ownership"]
Level --> Change["Change something: approvals and recovery"]

Each level adds a different kind of responsibility. Explaining needs reliable knowledge. Looking something up means checking who is asking and whether they may see the answer. Changing something also needs controls around approvals, duplicate actions, limits, and recovery when only part of an operation succeeds.

Order status sat in the middle. The service needed to read information belonging to one merchant, but it did not need to change anything. That made a narrow, read-only design a good match for the problem.

The important discovery was that specific bad news still helped

Before building, I reviewed order-status conversations that Fin had touched but eventually handed to a person. I was looking for two things: which information the human agent needed, and whether providing that information alone was enough to resolve the question.

A clear pattern emerged. Merchants were often satisfied once they understood the real reason for a delay, even when the answer was a machine breakdown or a known technical problem. Feedback from the human agents handling these cases confirmed the same pattern.

This was qualitative discovery, not an experiment with a neat confidence interval. But it changed the problem definition. The goal was not to make every order sound healthy. It was to replace a generic “your order is being processed” reply with the most specific explanation the merchant was allowed to receive.

Transparency did not mean copying internal conversations into chat. It meant giving merchants enough truth to set expectations with their own customers.

The opportunity estimate was simple

Order-status questions represented roughly 15% of all support conversations. My review suggested that better context could satisfy about 80% of those questions without a human handoff.

The maximum expected improvement was therefore:

15% of conversations × 80% realistically answerable = 12 percentage points

At a normal 10,000 conversations per month, 12 points represented about 1,200 additional deflections. Human handling through an outsourced support provider (BPO) was estimated at roughly €5 per conversation. That was roughly €6,000 worth of outsourced support handling capacity in a normal month. During peaks, volume could double, making the equivalent capacity roughly 2,400 conversations or €12,000. This becomes spend avoided only to the extent that staffing or BPO charges flex with volume; it is not a claim of guaranteed net savings.

The harder-to-price benefit was operational. Every seasonal agent not required for a peak also meant less recruiting, onboarding, training, quality monitoring, and management time. The result was not accepted on deflection alone: customer satisfaction was a guardrail, and the reporting used for the release did not show a decline.

A narrow service sat between Fin and the internal systems

I built the first production version in late 2024. Fin had a configured tool named check_order_status, with instructions describing when to call it and an endpoint pointing to a webhook-triggered workflow in n8n (opens in a new tab), a workflow automation tool. From here on, I’ll call this the order-status service. It had an HTTP interface and behaved like a service, but it was implemented as an automation workflow rather than a separate custom backend application.

The live request looked like this after the package-to-conversation relation index had already been prepared asynchronously:

sequenceDiagram
accTitle: A merchant order-status request becomes one read-only answer
accDescr: After package-ID indexing has prepared the relation table, a merchant asks Fin about an order. Fin calls the service, which checks ownership, collects permitted order, tracking, operations, and technical context, then returns one answer.
participant Merchant
participant Fin
participant Service as Read-only service
participant Orders as Order and tracking data
participant Context as Operations and Jira context
Merchant->>Fin: Ask about an order
Fin->>Service: check_order_status with merchant and order IDs
Service->>Orders: Verify ownership and retrieve order facts
Service->>Context: Retrieve permitted operational and technical context
Orders-->>Service: Order, shipping dates, and tracking
Context-->>Service: Permitted incident context
Service-->>Fin: Read-only order explanation
Fin-->>Merchant: Explain the order status

Two inputs had very different trust levels

An illustrative request body would look like this:

{
  "internal_account_id": "merchant-account-123",
  "order_id": "123456789"
}

This is deliberately clearer than the original production payload, not a copy of it.

The internal account ID was the company's own identifier, stored in Intercom as the contact's externally configured user_id (opens in a new tab), not Intercom's generated contact ID. Intercom accepted it inside a verified Messenger session, and the configured tool passed that account key to the service. The order ID came from the merchant's message and was extracted by Fin, so the service treated it as untrusted input. Intercom's connector inputs are configured by the builder; there is no universal payload that Fin sends to every external service.

Getting through the front door did not grant access to every room

The company's application server generated a signed proof — the HMAC used by Intercom's then-current Messenger Identity Verification (opens in a new tab) — that the session belonged to the logged-in merchant. Intercom used it to accept that identity. A separate secret in the webhook request authenticated the integration calling the service; the matching value was stored through n8n credentials rather than exposed in the workflow. Neither step granted access to an order by itself.

Those checks established who was calling. The service still had to confirm that the requested order belonged to that merchant.

I used one internal lookup filtered by both the trusted account ID and the supplied order ID:

find order
where order_id = supplied_order_id
and merchant_account_id = authenticated_account_id

If the lookup returned nothing, Fin received the same generic response whether the order did not exist or belonged to somebody else. Only a successful match allowed the service to retrieve packages and continue.

The plain-language rule was: knowing an order number is not proof that you may open it. OWASP describes (opens in a new tab) the underlying risk as Broken Object Level Authorization, but the practical defence was simply to check ownership on every request and avoid revealing whether somebody else's order existed.

Four sources became one answer

1. Internal order state and the dates merchants actually needed

The first source was the company's existing internal API. It returned structured facts such as order status, package status, original dates, and revised estimates.

The merchant dashboard already showed original and estimated delivery dates. What it did not show were the original and updated shipping dates. Those dates mattered because a merchant needed to tell their own customer when the white-label production step would be complete and the package would leave the facility. Before this service, a human support agent supplied that information after the merchant contacted support.

This branch needed no model. The information was already structured, so the service passed it through as facts.

2. Tracking events from the existing integration

For packages that had shipped, the second source retrieved tracking history through the company's AfterShip (opens in a new tab) shipping-events integration. It could return carrier updates, location history, and details that were not always shown on the public tracking page.

The alternative was to open carrier tracking pages and extract their rendered contents. That would have meant different behaviour across many carriers, JavaScript rendering, occasional postal-code verification, page-layout changes, and another presentation layer between the service and the underlying data.

My default is to prefer an existing structured API over scraping a customer-facing interface. A frontend is designed for a person, not as a stable machine contract. Browser automation is useful when no better boundary exists, but it should not replace a trusted backend integration the business already has.

3. Production delays hidden in another team's Intercom

The largest information gain came from conversations between production facilities and the company's operations team.

A facility might report a machine outage, power failure, staffing bottleneck, or another delay affecting a group of packages. Those messages lived in an operations Intercom workspace that support agents could not access. A support agent had to ask an operations specialist to investigate, and the merchant waited while the two teams coordinated.

The challenge was joining a package to the right operational conversation.

Production facilities already included internal package IDs in their messages. The IDs followed a predictable nine- or ten-digit pattern, so I used a regular expression — a simple pattern matcher — to extract candidates and then checked that the sender belonged to the facility responsible for that package.

This extraction happened asynchronously whenever a new facility message arrived. It did not run inside the live Fin request. A numeric match was only indexed after the service verified that it was a real package assigned to the facility represented by the trusted sender. By the time a merchant asked about an order, those facility-checked candidate relations were already available.

A smaller model could probably have extracted the IDs accurately, and the model cost would have been small. I still preferred regex because the format was known: it was deterministic, fast to test, cheap to run, and did not add a probabilistic dependency where a simple rule was sufficient. The live 15-second response limit was important elsewhere in the design, but it was not the reason for this particular choice.

I accepted a narrow failure case: a facility inserting spaces inside an ID could prevent a match. I also tracked the point at which growing identifier lengths would require the expression to change. Those were explicit maintenance assumptions, not reasons to replace a predictable rule with a model.

The relation table was a workaround for an API gap

My first plan was to save the extracted package IDs in an Intercom conversation data attribute (opens in a new tab) and search for conversations by that field. The version available at the time could not filter Conversation Search (opens in a new tab) by custom conversation attributes. The company's daily BigQuery (opens in a new tab) sync was too stale for a merchant asking about an order now.

I created a narrow Supabase (opens in a new tab) table containing the relation I needed. Conceptually, it looked like this; the public shape is simplified rather than a schema contract:

conversation_id | package_ids[]

The asynchronous indexing workflow populated that table as production-facility messages arrived. During an order check, the service used the package IDs to find relevant conversations, considered only those still open in Intercom, and prepared only the material it was allowed to use for summarisation. The table was an index, not a second copy of the conversations themselves.

4. Technical incidents in Jira

The fourth source searched Jira, the engineering team’s issue tracker, for issues whose custom fields listed affected order IDs or package IDs. A match could explain a print-file rendering failure, a storage problem, or another technical issue preventing production from progressing.

Those fields were maintained manually by the people investigating an incident. I accepted that an affected package could occasionally be omitted or entered incorrectly. The likely consequence was missing or inaccurate root-cause context, not access to another merchant's order.

That was a different class of risk from the ownership check. I would not accept an imperfect rule for deciding whether a merchant could see an order. I could accept metadata that might occasionally be incomplete when connecting an already-authorised order to an incident, provided the answer did not pretend to know more than the available evidence.

The service decided what the model could see

The service treated authentication, ownership, and disclosure as separate gates. Passing one did not grant access to the next layer.

flowchart LR
accTitle: The service's disclosure boundary
accDescr: A request first passes authentication and order-ownership checks. The service then finds the relevant records, skips sensitive content, redacts unnecessary personal data, summarises only permitted context, and returns one answer.
Request["Fin request: merchant ID + order ID"] --> Auth["Authenticate the caller"]
Auth --> Ownership{"Does the order belong to this merchant?"}
Ownership -- "No" --> Deny["Generic response"]
Ownership -- "Yes" --> Records["Find relevant order, tracking, operations, and Jira records"]
Records --> Sensitive{"Sensitive flag = true?"}
Sensitive -- "Yes" --> Safe["Skip content fetch: use fixed generic message"]
Sensitive -- "No" --> Fetch["Fetch permitted content"]
Fetch --> Redact["Remove unnecessary personal data"]
Redact --> Summarise["Summarise permitted context"]
Safe --> Answer["Return one answer to Fin"]
Summarise --> Answer

Sensitive incidents were stopped before their contents were fetched

Some operational and technical incidents should not be explained to merchants in detail.

The responsible teams could mark an Intercom conversation or Jira issue as sensitive using a custom field. The service checked that field before retrieving the object's full contents. If it was true, the branch stopped and substituted a generic message saying that a known issue might be affecting the order and the relevant team was working on it. The conversation or issue body never entered a model prompt.

The field defaulted to false, so a person had to mark the record explicitly. That is an important limitation: enforcement after classification was deterministic, but classification still depended on human governance. I would not describe the default as fail-closed or claim the design eliminated the risk of somebody forgetting to mark a sensitive incident.

Security and legal stakeholders reviewed the narrow permissions, disclosure rules, and redaction before the model call. The design discussion was not “can the model write a good reply?” It was “which records may this merchant access, what enters the model, and what happens when information should not be disclosed?”

Permitted did not mean complete

Even when an incident was not marked sensitive, the model did not need the entire source object.

The service removed the names, email addresses, and delivery addresses of merchants' customers, along with fields unrelated to the explanation. Employee and facility identities became stable role labels such as Operations Specialist 1 and Production Facility 1.

I would call this redaction and pseudonymisation rather than promise irreversible anonymisation. The useful structure remained — who said what and whether two messages came from the same participant — while unnecessary personal data was removed before the model call.

AI had three narrow writing jobs

The service could make up to three model calls:

1. Summarise permitted operational conversations around cause, action taken, and expected resolution.
2. Summarise permitted Jira issues around technical impact and current status.
3. Combine those summaries with structured order facts and tracking events into one answer for Fin.

The merchant's free-form Messenger message was not included in the prompts that summarised operational or technical sources. Each job received only the context needed for that job. If there was no relevant incident, the corresponding model call was skipped. If a record was marked sensitive, the final step received a fixed generic message instead of its contents.

The model was good at turning uneven context into readable language. It did not authenticate callers, decide order ownership, classify an incident as sensitive, or discover relations between systems.

The 15-second deadline shaped the live request

The live path had to stay inside the 15-second timeout used by Intercom's external-data connector, then called Custom Actions and now documented as Data connectors (opens in a new tab). Intercom's current documentation still lists 15 seconds by default, although eligible Fin Procedures (opens in a new tab) can now receive a longer window.

My target was for at least 95 of every 100 live requests to finish within those 15 seconds. In performance terms, that is a p95 latency below 15 seconds. This was a design target, not a result I can now prove from retained performance data.

That constraint affected the architecture:

• Independent source lookups ran in parallel.
• Model calls were skipped when a source had nothing useful to summarise.
• Structured facts stayed structured instead of being sent through a model.
• Package-to-conversation relations were prepared asynchronously before a merchant asked.

The observed average was about 12 seconds, and the estimated workflow cost was below $0.01 per order.

There was a more elaborate alternative. The service could return immediately, store an in-progress job, finish gathering context asynchronously, and let Fin fetch the result later. That would require persistent job state, correlation IDs, expiry rules, retries, and another set of failure paths.

I intentionally kept that out of scope. The occasional request missing the synchronous deadline could cost a potential deflection. That was preferable to building and operating a system that stored jobs, retried them, and matched a later answer back to the original question before the business value justified it.

Missing context was acceptable; unsafe context was not

The service treated missing extra context differently from failed identity or ownership checks.

If it could not establish the merchant or confirm the order belonged to them, it stopped. If a source of extra context was temporarily unavailable, the final answer used the reliable information still available. The merchant did not receive an implementation report listing which integrations had failed.

In plain terms, a missing tracking update could produce a less complete answer. A failed identity or ownership check produced no order answer at all.

The workflow was monitored after deployment, and execution errors were investigated and fixed as they appeared. Source content was not copied into a new case database, but it remained inspectable in n8n execution history for troubleshooting. That distinction matters: the design avoided another application datastore, but the data was not “memory-only.” I can no longer verify the exact retention and access configuration, so I will not retroactively claim stronger controls than I can support.

What changed after launch

I measured order-status questions separately before and after release. Deflection improved for those questions, while the overall support deflection rate rose by 12 percentage points. That overall movement happened to match the earlier 15% × 80% opportunity estimate. The alignment is useful context, not proof that this implementation caused every change in the overall rate.

The same service was also made available to human support agents through Slack (opens in a new tab). Instead of checking a dashboard and tracking link, asking operations for context, and looking for a related technical incident, an agent could request the same read-only explanation where the team already worked. The Slack version checked the support agent through its own access rules; it did not pretend that the agent was a merchant using Fin.

The merchant-facing result was more deflection without a fall in the CSAT reporting used as a guardrail. The internal result was faster access to context. The operational result was capacity to handle seasonal peaks — especially ecommerce Q4 — with less temporary hiring, onboarding, and training than the same volume would otherwise require.

What I would build differently today

Some implementation choices were products of late 2024. The underlying boundaries still hold.

I would use Intercom's current Messenger security

The original implementation used Intercom's HMAC Identity Verification because that was the available Messenger security mechanism. Intercom introduced JWT-based Messenger authentication in May 2025 (opens in a new tab), and its current JWT documentation (opens in a new tab) now recommends moving from user hashes to JWTs. Today I would use JWTs for expiry, stronger control over signed attributes, and alignment with Intercom's current recommendation.

I would first test whether the relation table was still necessary

Intercom has since added Conversation Search filtering by custom conversation attributes. A new implementation might be able to store and query the package relation directly in Intercom instead of maintaining a separate index.

If a separate index were still useful, I would also evaluate n8n's native Data tables (opens in a new tab) before reaching for Supabase. n8n announced Data Tables in September 2025, with version 1.113.1 as the patch release that made the feature available, after this service was built. The announcement (opens in a new tab) describes them as persistent storage inside the workflow platform. They could fit a small normalized index, although I would keep a real database if I needed stricter access boundaries, uniqueness guarantees, or heavier concurrent use.

I would make the human classification dependency more explicit

The sensitivity flag prevented marked content from being retrieved, but an unset field meant “not sensitive.” For a new system, I would revisit whether higher-risk incident types should require an explicit disclosure classification before any contents could be used.

That could add friction for operations and engineering teams, so it is not automatically the right answer. The important part is to name the trade-off correctly: deterministic enforcement does not remove the need for reliable human governance.

I would review execution logging as part of the data design

The original service left source material inspectable in workflow execution history. In a current implementation, I would explicitly decide which fields may be logged, who can inspect executions, and how long that data should remain available. Troubleshooting access is useful, but it is still another place where sensitive operational context can exist.

I would keep the model's job small

Current agent runtimes could expose the capability through more interfaces and orchestrate more steps. I still would not give a merchant-facing agent broad access to Intercom, Jira, tracking providers, and production data.

The durable architecture is one narrow capability that receives authenticated context, checks access at the order level, fetches only necessary data, applies disclosure rules before model calls, and returns one task-complete answer.

This was a read-only service. Refunds, cancellations, and other changes need a different design around approvals, idempotency, monetary limits, audit trails, and recovery from partial writes. That deserves its own case study.

The most useful AI system I built for support worked because AI had the smallest job in it.