Sample Ratio Mismatch (SRM) is a thing, a real thing that optimization programs need to concern themselves about. At a high level, SRM means that the variants (experiences) of an A/B test have a traffic distribution (sample) that differs in a statistically significant way from what you expected the allocation of traffic or the sample to be.
This blog post and this blog post are great as they dive into what it is, how scary it can be, and what to do about it. Please do check out both those posts as they drive home just how important of a topic this is for optimization programs.
Sample Ratio Mismatch and MiaProva
About 5-6 weeks ago, MiaProva quietly released a new Alert to our array of alerts we provide for testing programs. This Alert will notify MiaProva customers that use Adobe Target automatically when SRM has been detected. This approach automatically applies a Chi-Squared calculation to every single A/B Activity currently Live in their Adobe Target accounts.
When MiaProva detects these issues, we will immediately notify stakeholders via email and highlight the Alert in the Live Dashboard and the daily Aggregate email that summarizes all Activity-related Alerts.
In the last month, we have had SRM detected a handful of times across many hundreds of Live Activities from different organizations. With SRM, the most important thing is identifying the root cause of the mismatch. Two out of the five are still being investigated, but the other three were quickly diagnosed. The statistically significant issues were:
- During QA, an internal team became members of one of the variants and maintained that membership when the test went live. Incorrect targeting used in the Activity setup further exacerbated this SRM issue.
- A bot had gotten into one of the variants.
- Improper use of the Advanced settings upon Conversion, the selected Action below, combined with the audience criteria, caused an imbalance of sample distribution.
Had MiaProva not detected and alerted people to this key issue, organizations could have found themselves in a situation where they were making decisions on statistically invalid test results. Furthermore, because MiaProva detected SRM, the organizations could remedy the underlying causes, thus preventing additional reporting issues.
Below represents what SRM alerts look like in context to Live Activities that are from Adobe Target. MiaProva also provides an SRM calculator with each Live Adobe Target Activity that is an A/B Activity. Automated Personalization, Experience Targeting, and Auto-Allocate Activity types have logic to promote differences in sample distribution across Activity variants. MiaProva provides this service for all A/B Activities, Multivariate Activities, and Adobe Recommendations.
And lastly, here is what the calculator looks like and how users can adjust their Expected Visitor distribution as needed.
Leave a Reply