Episode 19 - How to Fast-Track Retention Performance Improvements

Did someone forward this to you? If so, click to subscribe 

📰 TOP STORY

How to Fast-Track Your Retention Experiments 

Any experiment leads to knowledge and applied knowledge leads to an uplift in retention performance.

Even a failed experiment leads you towards success. 

To optimise retention, you must experiment. You need to identify what works and what doesn't across all aspects of your product experience, customer service experience, and CRM Marketing. 

A program of ongoing structured testing across all retention moments of truth, whether onboarding, service handling, usage drops, the cancel journey, offer pricing or any other critical moment (see episode 15 for how to identify moments of truth), is essential to maximising retention outcomes. 

We use statistical significance to measure how likely a test will be repeated (using free calculators like this one) by comparing your treatment group to your control group. 

Shachar Meir, who spent seven years as a Data Director at Meta, told me that if you achieved less than 95% statistical significance in your Facebook or Instagram UX experiment and wanted to roll out the test, you would be laughed out of the room. 

This makes sense for a brand like Meta, with 3.1 billion users. 

Because at large testing volumes, slight differences in outcomes between your control and treatment group mean you have a high chance of statistical significance. 

Let’s take an example 

• 1m records in your treatment group 

• 100k records in your control group 

• A 0.22% impact in your treatment group and 0.2% impact in your control group = 95% confidence that your retention experiment is a success; a 10% improvement is statistically significant. 

Many organisations regard Amazon, Meta, Apple, Netflix, Spotify, and Google as the gold standard in all aspects of their retention operations, and in many respects, they are (I have referred to them all in this newsletter). 

However, if your business has a smaller audience than these giants, you need more considerable differences between the test and control groups to ensure high statistical confidence levels. 

Let’s take another example;

If you have only 10k records in your treatment group and 2k in your control group, you would need to achieve a 0.44% result in your experiment vs. 0.2% in the control group to achieve 95% confidence— more than double the impact of the control.

So if even you are a large business (but not a tech giant) and running experiments on specific moments of truth for sub-cohorts, adopting the Meta threshold of 95% +, your experimentation culture is likely to

• Be slow because it will take a long time to achieve the magic 95% 

• You will rule many tests to be failures because they don't hit 95% confidence 

This will limit your opportunity to improve retention. 

You will hit analysis paralysis. 

And you won't improve. 

To circumvent this, adopting an 80% confidence level will rapidly accelerate your testing outcomes and enable you to fast-track retention performance improvement. 

In the same example, if there are 10k records in the treatment group and 2k in the control group, and we use an 80% confidence level, then a result of 0.35% in the treatment group versus 0.2% in the control group would indicate a successful experiment. This is a 38% lower difference than if we were to use a 95% confidence level.

Sure, sometimes a test that you thought worked, because it hit 80% confidence, will later fail at rollout. 

But you will be right eight times out of ten, and with a reduced confidence threshold comes increased speed to execution. 

More tests will be successful. 

And your organisation will improve retention more rapidly. 

More rapid improvements in retention can significantly impact profits (see episode 13 of this newsletter for how retention can double profits). 

Of course, if you roll out an experiment following an 80%+ confidence score and it later turns out to be wrong, you roll back.

Because almost all retention initiatives are what Jeff Bezos calls Type 2 experiments. 

Type 2 experiments are easily reversible and include things like:   

• Any CRM Marketing initiative 

• Changes to a product onboarding journey 

• Incentives to drive engagement or referral 

• Changes to a cancellation journey 

• Service differentiation, e.g. providing different service levels to high-value customers vs low-value customers 

• Offer pricing 

Type 1 experiments make such a fundamental difference to your business and product that they involve at least a 180-degree flip in operations and would require massive investment to reverse (of course, such changes impact retention, but these will be org-wide changes vs. run by your retention teams).  

Final Thoughts 

There are three critical principles for running effective retention experiments: 

1. Iterate continuously to maximise your impact across all retention moments of truth 

2. If you achieve 80%+ confidence, roll out the type 2 experiment 

3. If you know the answer, do not experiment! 

The last one is important. Knowing the answer is more complex than thinking you know the answer. You must be sure. As Jeff Bezos says, outsized returns come from betting against conventional wisdom. 

🤖 AI Feature

This week we dive into a brilliant article by Henrique Centieiro on Medium.

Analysis shows that ChatGPT models are becoming 5X smarter with each passing year. “This means that by 2027/28, ChatGPT and other LLMs—large Language Models—will be as smart as (all) the top PhDs worldwide (combined)”. This explosive growth will redefine industries and revolutionize science in ways we can’t even fathom. The countdown to superintelligence has begun, read more on Medium here or if you want the details on the decade ahead, read this situational analysis.

Until next week,

Tom

P.P. S. There is still time to complete our survey.

One subscriber will win $30 Amazon gift card once we reach 25 submissions.

Daily News for Curious Minds

“I stopped watching the news, so sick of the bias. Was searching for an alternative that would just tell me WHAT happened, with NO editorializing. I found it. It’s called 1440. It assumes you are smart enough to form your own opinions.”

Sign up now!

CRM Marketing 2.0 Course Waiting List Closes Soon!

The waiting list for the CRM Marketing 2.0 course will close on the 8th November. 

My first course gained over 1140 students and was the top-rated CRM Marketing Course on Udemy.

There are only 40 14 waiting list places remaining for CRM Marketing 2.0 pre-order, so if you want to take advantage of a $80 saving & pre-order bonuses and guarantees, join the waiting list now (Full price $149, waiting list price $69).