Conservative assumptions, resulting in larger group sizes.There are many assumptions we can use to calculate group sizes, and I am going to group them into two approaches: Put on your thinking hat there are a lot of numbers involved. 50/50 split is often used when trying to minimize the total experiment sample size, and 90/10 split is used when trying to minimize one of the groups. I provide numbers for 50/50 and 90/10 splits, both of which are very common. Approximately, what percent of the total targets is going to be your control group.The larger the effect size you need, the easier it is to prove, thus requiring a smaller control group.
To determine this parameter, you need to figure out potential profit from additional sales, the cost of your marketing campaign (including measurement and creating additional versions), and the breakeven point in additional sales where your marketing campaign pays for itself. The effect size you need to justify the marketing expense.For most marketing campaigns, smaller response rate need smaller groups. The expected response rate in your groups.The metrics used to measure outcomes should be applicable to both groups.Įffect size (incremental lift, uplift) – the difference in the response rate between test and control groups, expressed as percentage points.įor example, if your control group had a 3% conversion rate and the treated group had a 5% conversion rate, then 3% and 5% are response rates, and the effect size was 2 percent points (pp). The rate refers to the percentage of the group that completed the action. Response rate – a collective term for the types of outcomes you will be measuring and comparing, such as purchase rate or conversion rate. Holdout group is a type of control group that is withheld from marketing treatment altogether. Control group is usually randomly chosen from all campaign eligible targets. Test (treatment, marketed, campaign) group – a group of targets that is being exposed to a specific version of the marketing campaign.Ĭontrol group – a group of representative targets that is exposed to a different version of the marketing campaign. either a customer responds or does not and that you are evaluating the difference in the percent response. This type of result implies that your response is binary, i.e. I am going to define “results” as the difference in a percent metric such as purchase or conversion rate between your test and control groups. The goal is to produce statistically significant results. Now you need to figure out how many targets you need for the test, particularly, the number of targets in the test and control groups. So, you have decided to measure your marketing campaign against a control or holdout group.Ĭongratulations! You are on the right path. TL/DR: for most marketing experiments, I recommend having 10,000 targets in your smallest group.
0 kommentar(er)
