MendDocs
MendDocs
Concepts

The forecast

How Mend sorts quiet regulars into three groups with the BG/NBD model, the published benchmark it reproduces, and an honest backtest on two years of real orders.

In plain words

Mend uses a well-known customer model called BG/NBD (beta-geometric, negative binomial distribution), from Fader, Hardie and Lee (2005). It rests on two ideas:

  • While someone is still a customer, they buy at their own steady pace. Some customers buy every few weeks, some twice a year.
  • After any purchase, there is some chance they have quietly left for good.

The model learns from the whole shop how those paces and those chances to leave vary between customers. Then, for each customer, it looks at three facts from the shop's own order records: how many days they came back to buy, when they last bought, and how long they have been a customer. From those it estimates how many purchases that customer will make in the next 13 weeks.

A quiet regular has bought on 3 or more days and has been silent for 60 days or more. Mend splits them by that estimate into three groups. Under 0.1 expected purchases, they are likely gone. At 1 or more, they are likely to come back anyway. Everyone between is worth a nudge, and only that middle group is listed and paid, biggest spenders first.

It reproduces the published model

Fader, Hardie and Lee published the model's four parameters fitted on a sample of 2,357 CDNOW customers. Mend's own fit on the same sample gives the same numbers at the precision the paper reports:

ParameterPublishedMend
r0.2430.243
alpha4.4144.414
a0.7930.793
b2.4262.426
log-likelihood-9582.4-9582.4

The landing page draws the same fit live, from the shop summary the engine stores.

The backtest on real orders

The demo shop's history is UCI Online Retail II: two years of a UK gift-ware wholesaler's orders, December 2009 to December 2011. The backtest runs on the original dates:

  1. Pick a test date. Fit the model on every order before it.
  2. Look at the next 91 days and note who bought again.
  3. Score how well each ranking separates customers who came back from customers who did not, with AUC: the chance that a customer who came back is ranked safer than one who did not. 0.5 is a coin toss.
  4. Compare with the simplest ranking a shop owner would use: days since the last order (recency).
  5. Repeat on six test dates three months apart, from 9 Jun 2010 to 9 Sep 2011, with a 95 percent range from 1,000 resamples.

The fit on 9 Jun 2010 did not converge, so that date is reported and left out, leaving 5 test dates.

Who came back on their own, by group

This is the test of who Mend nudges. At each test date Mend scored the quiet regulars, put each one in a group by the forecast, and checked who bought again within 91 days with no cashback at all.

Test dateLikely goneWorth a nudgeLikely back
9 Sep 2010none124 of 200 (62 percent)131 of 176 (74 percent)
9 Dec 20100 of 1 (0 percent)89 of 298 (30 percent)72 of 181 (40 percent)
9 Mar 20111 of 9 (11 percent)289 of 826 (35 percent)179 of 325 (55 percent)
9 Jun 20112 of 24 (8 percent)295 of 1,019 (29 percent)118 of 203 (58 percent)
9 Sep 20116 of 58 (10 percent)574 of 1,252 (46 percent)153 of 203 (75 percent)
All five9 of 92 (10 percent)1,371 of 3,595 (38 percent)653 of 1,088 (60 percent)

About 1 in 10 of the likely gone came back, so cashback there is rarely answered. About 6 in 10 of the likely back came back without any, so cashback there mostly pays for a purchase that was coming anyway. The middle group is where cashback has room to change something, and the held-back group in every campaign measures whether it did. The likely-gone group is small at every date and empty on 9 Sep 2010, so its share rests on 92 customers in all.

Among quiet regulars: a tie

Quiet regulars are the pool a campaign picks from. The forecast won 3 of 5 test dates. The mean AUC difference is -0.005, with a 95 percent range of -0.024 to +0.016. That is a tie with ranking by most recent order. As a strict ranking inside this pool the forecast is no better than recency, so Mend uses it to split the pool into groups and ranks the middle group by spend instead.

Test dateCustomersCame backForecast AUCRecency AUCDifference95 percent range
9 Jun 201011769did not converge0.550
9 Sep 20103762550.5770.654-0.077-0.149 to -0.003
9 Dec 20104801610.5680.605-0.038-0.096 to +0.018
9 Mar 20111,1604690.6350.593+0.042+0.013 to +0.071
9 Jun 20111,2464150.6710.658+0.013-0.012 to +0.036
9 Sep 20111,5137330.6920.656+0.036+0.017 to +0.054

Other rankings were tested under a rule fixed before the results were seen, and none beat recency among quiet regulars: purchase frequency won 0 of 5 (mean -0.228), an average of ranks 0 of 5 (mean -0.106), and the model's own chance that a customer is still active 1 of 5 (mean -0.018). Mend keeps the forecast: it is level with recency inside the pool, better across the whole shop, and its three groups come back at clearly different rates.

Across all customers: better than recency

Over every customer with an order before the test date, the forecast won 5 of 5, with a mean AUC difference of +0.052 and a 95 percent range of +0.048 to +0.057.

Test dateCustomersCame backForecast AUCRecency AUCDifference95 percent range
9 Jun 20102,7751,347did not converge0.655
9 Sep 20103,3511,9410.7080.652+0.056+0.043 to +0.068
9 Dec 20104,2851,3270.7780.713+0.066+0.054 to +0.078
9 Mar 20114,5631,6040.7880.733+0.054+0.045 to +0.064
9 Jun 20114,9441,5940.8240.777+0.048+0.040 to +0.055
9 Sep 20115,2562,2890.8010.763+0.038+0.031 to +0.045

Who came back, by tenth

On 9 Sep 2011, Mend split all 5,256 customers into tenths by the forecast and checked who bought within 91 days. The safest tenth came back 92 percent of the time. The most at risk tenth came back 10 percent of the time.

TenthCustomersCame back
1, safest52692 percent
252678 percent
352662 percent
452652 percent
552643 percent
652634 percent
752525 percent
852522 percent
952517 percent
10, most at risk52510 percent

What the forecast is not used for

Mend uses the model to sort customers into groups, never to promise how many orders the shop will get. On this seasonal shop, the model's totals for the whole shop were off by between -23 and +91 percent, so Mend does not show them.

The live numbers behind these tables are served by GET /api/shop/summary and drawn on the landing page.