Evidence & Returns: How AI Changes a Business Result

The before

Why the comparison point decides what a result means

Every result is a difference between two situations. Change the 'before' and the same AI can look remarkable or ordinary. The shopper and chat on this page are a teaching reconstruction, not a real participant, record or the chatbot's wording.

One buying questionInformation is missing at the moment of a decision.

Two different responsesAn unavailable-support reply, or an AI chatbot answer.

A buying decisionCompare what customers actually do next.

A reconstruction of the comparison. The illustrated question is not a research transcript.

How the comparison was set up

Researchers randomly assigned 44,614 consumers to a chatbot group or an unavailable-support group in September–October 2023. This separates the tested response from customers choosing which service to use. The comparison supports the specific intervention in that setting.

What it did not compare

With no live answer as the before, the result says nothing about AI against a capable human team. A business that already answers questions well starts from a different place and may see a different, possibly smaller, change at the same moment.

Finding your own before

Application, not a study finding: the same shape shows up in a quote request left until morning, an after-hours call that rings out, or a booking form nobody reads until Monday. Write down what happens today, who handles it and what the customer does next, before changing anything.

Before judging any AI result, including your own, name the exact 'before' it was measured against and check that it matches your business today.

Retail experiment · working paper v6 ↗

The result

What a sales lift can and cannot tell you

A measured sales gain is genuine evidence. Reading it well means knowing what else the study found, what exactly was tested, and what still stands between a sale and profit.

Answer a real questionReduce friction before checkout.

Observe the purchaseCheck whether the business result actually changed.

Carry the extra workFulfill the order and retain margin after costs.

The purchase result is research evidence. Delivery and financial capture are the next business questions.

One experiment among seven

That result was one of seven separate experiments in the study. Some other uses showed smaller gains, and two advertising tests showed no statistically detectable sales gain, which is not proof of zero. Different conditions and timing mean the results cannot be added up or ranked as one table of AI returns.

What actually transfers

What was tested is one particular chatbot against no live answer, so the study does not separate 'any answer' from this chatbot's design. What reasonably transfers is the shape: a customer with intent, friction, and a timely answer. The size of the lift stays with this setting.

Can delivery keep up?

Application, not a study finding: extra demand lands on whoever packs, ships, installs or serves it. If that team cannot keep up, orders can wait, mistakes can rise and refunds can follow, and the sales gain leaks away before it reaches margin. Check delivery capacity before you widen the test.

If you test an answer at a buying moment, decide in advance which sales you will count: orders delivered and kept, not chats or clicks.

Retail experiment · working paper v6 ↗

The team

More issues resolved per hour: where does the time go?

An assistant can make room for more work. The financial question is where that room goes next: additional profitable work, spending that really stops, or service you value before its money effect is known.

AI suggestsThe agent can use, edit or ignore a reply.

People actThen decide where released capacity can do useful work.

More profitable workCapacity put to useSpending avoidedA cost that really stopsBetter serviceMoney effect still to establish

Possible business paths, not three financial outcomes measured by the support study.

Uneven gains, human control

Gains were uneven, and the highest-skilled agents showed small declines in quality. The assistant was introduced in stages rather than randomly assigned, so the evidence rests on differences in timing. The average is real, but it hides who benefited most and whose work needed closer review.

Capacity or cash?

Application of general finance, not a study result: fixed salaries do not shrink because a tool frees time. Released hours become money in two main ways: additional profitable work the team now handles, or spending that genuinely stops, such as overtime, contractor hours or a hire you no longer need.

Count each hour once

If a freed hour serves an extra customer, its value is the margin on that work; the same hour cannot also be claimed as wages removed. Faster or better service still matters and belongs on the record, labeled as a service benefit until a financial effect is actually shown.

Before rolling assistance out, decide what the released time is for: more profitable work, a specific cost you will stop, or service you value but won't price yet.

Customer support · QJE 2025 ↗

The judgment

Advice is a suggestion; the result comes from the change

A suggestion is only the beginning. A business result depends on the action taken and the setting it lands in. Read the average alongside the differences underneath it, then bring your own business knowledge to the decision.

Advice offeredA possible change to consider.

Owner choosesFit it to real customers, margins and the current situation.

The authors interpret differences in actions as part of the mechanism. This is not a separate causal test of a tactic.

What the average does and doesn't say

Researchers randomly offered access to the adviser and compared revenue and profit between groups. No statistically distinguishable average gain was established. That leaves room for different individual outcomes; it does not prove every business gained nothing or that AI advice always fails.

Tailored versus generic

The authors associate stronger performance with changes tailored to the business, and note that weaker performers more often chose generic moves such as cutting prices or advertising more. That is their interpretation of the pattern, not a separate test of any single tactic. Treat it as a lens, not a rule.

Advice versus action in context

Application, not a study finding: an AI suggestion knows only what it has been told. Check it against your customers, margins, capacity and constraints before acting. A suggested price cut, for instance, lowers the margin on every sale, not just new ones. Your knowledge turns a plausible idea into a fitting one.

Treat AI advice as a first draft. Act on changes you can explain in terms of your own customers and margin, and try them small before committing.

Business adviser · Management Science 2026 ↗

The bridge

From a measured change to profit you can count

Follow the change to the money it actually creates or the spending it genuinely avoids. Then compare that benefit with the full cost of producing it.

SalesMore completed ordersCan the business deliver them?Margin retainedAfter product, delivery & returns

CapacityMore room to workHow is that capacity used?Profitable work or spending avoidedEach released hour counted once

Financial benefit demonstrated↓ Compare over the same period

Then account for setup and running costsA common period. Each benefit and cost counted once.

A qualitative business reasoning map. No visitor ROI amount or forecast is being calculated.

The sales lane

Start with the extra orders, keep only those delivered and not returned, then subtract product, delivery and other variable costs, once. What remains is delivered margin. Cost the delivery here and nowhere else, so it is never taken off a second time further down the comparison.

The capacity lane

Start with released time and follow where it actually went. Count additional profitable work completed, or spending that genuinely stopped, choosing one path for each hour. Service, accuracy and customer confidence go in a separate column, visible and valued, until you can show their effect on money.

Same period, full cost, real timing

Set the combined benefit against everything the system cost over the same period: setup, adoption, running and review. A quarter of benefit against a week of cost flatters the result. Getting paid sooner improves cash flow but is not extra profit. Unknown costs stay unknown, not zero.

Before calling any AI change a return, fill in both columns for the same period and mark every figure you haven't confirmed as unknown.

What does it take to build, adopt and keep it running?

Your moment

Test one moment, then decide what comes next

Pick a specific work moment, name who would act differently, and decide how you will tell whether the rest of the business can carry the change.

The visible beforeWhat happens today?

The specific changeWho does what differently?

The next decisionWhat would justify continuing, adjusting or stopping?

Use the study mechanisms to sharpen a question about your own work, not to import a result.

Choose a moment you can already see

Prefer a moment where today's outcome is already visible and the path to money is short: a sale your team can deliver, or a cost that will genuinely stop. A clear before makes any change visible; a short path makes the result easier to judge honestly.

Run a small, honest test

Choose a fixed period and change one thing. A comparable group gives a stronger comparison; an earlier period can help, but seasonality or staffing changes may explain the difference. Decide who reviews AI output, what delivered value means, and which costs to record.

Expand, adjust or stop

Look at who gained, not only the average; uneven results ran through every case on this page. Expand when the benefit holds after full cost. Adjust when it helps but delivery or billing cannot keep up. Stop when it doesn't hold, and keep what you learned.

Choose one moment you can observe. Name its before, decide who owns the change, and agree what would justify continuing, adjusting or stopping.

Where do these findings come from, and how were they measured?

Retail experiment · working paper v6 ↗Customer support · QJE 2025 ↗Business adviser · Management Science 2026 ↗

Sources

Three studies: how each was run and how to read it

External research, not BH Technology client results. Each study asked a different question with a different method; read them side by side, never as one score.

A sales resultOne specific retail chatbot comparison.

A capacity resultAI suggestions inside a human support team.

Mixed financial resultsAn AI business adviser used by different owners.

Three different settings and outcome measures. They are not a combined ROI estimate.

Three methods, three strengths

Random assignment compares groups allocated by chance. A staged rollout compares changes around different introduction times and needs its own checks. Read each design on its own terms; a productivity measure and a sales measure cannot be added into one financial return.

Reading 'no detectable effect'

'No statistically detectable' or 'distinguishable' means a difference could not be reliably separated from chance in that setting. It is not proof of zero, and it does not mean nobody benefited. It does mean no dependable average gain was shown.

Versions and dates matter

Working papers change between versions: earlier listings of the retail paper carry a different title and abstract, so this page cites version 6. The support study's published figure differs from an earlier working-paper version. Quote the version you read, with its date.

What this page relied on

This page draws on published abstracts and targeted passages, including the retail study's comparison tables, not full readings or independent replication. Scenes and conversations are illustrations, and the financial reasoning is general application. The linked papers remain the authority for every figure.

Authors' working paper, version 6, revised 29 June 2026

Generative AI and Sales Productivity: Field Experiments in Online Retail

Read the original research ↗

One of seven separate experiments; sales, not profit or a human-service comparison.

Peer-reviewed, Quarterly Journal of Economics, published 4 February 2025

Generative AI at Work

Read the original research ↗

One firm, staged rather than randomized rollout; productivity, not payroll savings or cash.

Peer-reviewed, Management Science, published online 10 July 2026

The Uneven Impact of Generative Artificial Intelligence on Entrepreneurial Performance: Evidence from a Field Experiment in Kenya

Read the original research ↗

Effects varied by business; the tailored-versus-generic explanation is the authors' interpretation.

Open the linked paper before reusing any figure, and always quote it with its comparison point, its setting and its date.

Full cost

What it takes to build, adopt and keep running

The license is only the most visible cost. A fair comparison counts the work to build, adopt and keep the change useful, over the same period as the benefit.

Get it working

Data cleanup & access
Setup & integration
Workflow redesign

Help people use it

Training & practice
Management attention
Adoption time

Keep it useful

Licences, usage & infrastructure
Review, exceptions & corrections
Monitoring & upkeep

Which costs apply, and how large they are, depend on the actual implementation. Unknowns stay visible.

Setup

Connecting the system to the tools you already use, cleaning and structuring the data it needs, setting permissions and access, and redesigning the workflow around it. This work comes before any benefit appears, and it belongs in the same comparison as the gain it makes possible.

Adoption

Training, the slower weeks while people learn a new routine, and the management attention needed to steer it. These costs are real even when no invoice arrives, because they come out of the same hours the change is meant to improve.

Operating

Licenses, usage charges and infrastructure; human review of AI output; handling exceptions, corrections and escalations; and upkeep as tools, models, prompts and data change. Mark unknown amounts clearly, and show how they limit the confidence of the financial comparison.

List every cost line before the test begins, name who owns each one, and record actual amounts across the same period as the benefit.

One question. A different business result.

A question before checkout.

One buying moment.Two different paths.

One question, and nobody there to answer it

When AI answered, more shoppers went on to buy

More capacity. People still in control.

Same AI adviser, different owners, different results

Follow the gain until it shows up as profit

Pick one moment and follow it through

One buying moment.
Two different paths.