“You cannot improve what you cannot measure” Peter Diamandis.

Importance of sample size:

One of the most important aspects of marketing is to measure effectiveness of a specific campaign in achieving the stated goals. The most common method is to use statistical testing on the campaign results to determine if a difference is evident in the outcome. In its simplest form this involves comparing samples between targeted and control populations.

However, the behavior of human beings is very complex and individual reactions to any set of circumstances will vary. Whatever is measured, therefore, there will be variation between individuals. If there is only a small sample, then people with ‘unusual’ reactions can skew the results, but in larger samples this will not happen. Sample size needs to be big enough to overcome the effect of “outliers”. In general, larger sample sizes increase the chances of finding a significant difference. In some cases this might not be possible or become cost prohibitive.

“How then should one determine the optimum sample size needed for credible test results?” The best answer to the question is, ‘”It depends.” It depends on whether you are using multiple test groups of equal size or two groups of unequal size, for example a larger test group and a smaller control group. It depends on your tolerance for errors. It depends on the amount of unexplained variation (or “noise”) in your test. In fact, it depends on several parameters in your campaign.

The issue boils down to 2 key questions;

– How many data points would we need to randomly sample to detect that the difference between the two groups is statistically significantly different from zero? Or

– How many data points would we need to randomly sample to approximate the true difference with sufficient precision?

Depending on the type of test you’re conducting, there are up to four types of numbers you need to know to calculate the necessary sample size (n):

1) Power: or how much you can rely on the test to detect differences between groups if they really exist, that the marketing campaign actually had an effect versus not;

2) the size of the difference that you need to detect;

3) Significance: the level of error you can tolerate if you make the wrong conclusion about the effectiveness of the treatment you are testing. Indicates the desired level of statistical significance, and

4) The standard deviation, the amount of noise or variance in the results within a group, more noise bigger the sample size required

 

(Note: Since the sample size determination will need to be customized for each instance, please contact us for a specific use case)

Relevance of a control group:

mcol_pie_chart
A control group is a group similar to the experimental group, but where the factor thought to be causing an effect is removed.

In order to isolate the effect of a marketing campaign, the target population needs to be compared with a group of non-targeted customers, and this latter group should be as similar as possible to the targeted group, for example:

  • Similar gender balance
  • Similar demographics
  • Similar geographical/weather influences
  • Similar background and life experiences
  • Same time frame of the campaign

It is very unlikely that all of these criteria can be met, but if they could and targeted population out-performed the non-targeted ones, then there could be no doubt that the campaign influenced the outcome. Without any control group, there are too many other possible factors to be confident in your conclusion.