The backtest

Before you act on a risk score you should know whether it would have caught the churn you already had. The backtest answers that, on your own history, after every sync. It is the first thing to look at once the first sync finishes.

What it does

For every customer whose last subscription ended in the last twelve months, the account is rebuilt as it looked on each of the 90 days before it ended, from what your billing provider does record: when the cancellation was requested, when invoices were issued, paid or failed, when charges were refunded or disputed, when the discount or trial ended. The rules are then run on each of those days exactly as they run today.

The result is one table and one sentence:

• Per rule: how many of the churned accounts it flagged, and the median number of days before the end that it first fired.
• The headline: for example, “Would have flagged 8 of 13 churned accounts (60% of the $6,805 a month lost), a median of 30 days early.” That example is from the demo data. Yours will differ, and the point is that you can see it before you trust the list.

What it can’t measure

Stripe doesn’t keep the history of a card’s expiry date, so the card expiring rule can’t be replayed. It is reported as not measurable rather than counted as a miss. Likewise, the rebuilt days have no delinquency history. Days rebuilt this way are drawn dashed on the account’s risk timeline so you can tell them from days the product actually scored.

The same replay, used three more ways

• The risk timeline. On each sync, the 90 days before an account was first scored are rebuilt with the same replay, so every account’s chart is full from day one.
• The renewal forecast. Every account is scored as it looked every 30 days across the year, and the backtest counts how often each risk level was gone by its next renewal. Once a level has 50 such renewals, its observed churn rate (pulled toward the default by 20 pseudo-renewals, so a handful of cases can’t swing it) replaces the default odds in the revenue forecast. The Revenue page says which levels are fitted and on how many renewals.
• Your own rules. While you edit a rule in the Rule lab, it is replayed the same way: the churns it would have caught and how early, the false alarms (accounts flagged at a monthly checkpoint that were still paying 90 days later), and the share of flagged accounts that left within 90 days next to the share for everyone.

Reading the numbers

A rule that flags many churns a long way out is useful. A rule that flags many churns a day before the end is not, because there was nothing left to do. A rule that flags many accounts that stayed is noise, and that is what the Rule lab’s false-alarm count is for. Mute rules that don’t earn their place under Settings; the backtest will tell you what you gave up.