Should the control group be the same size as the experimental group?

Should the control group be the same size as the experimental group?

The size of the control group, or any test group for that matter, depends on the size of the total population. If the experiment is run on a population size of only 100 participants, a 5% control group would be only 5 individuals, which would certainly diminish the significance of the results.

How many control groups should an experiment have?

A true experiment (a.k.a. a controlled experiment) always includes at least one control group that doesn’t receive the experimental treatment. However, some experiments use a within-subjects design to test treatments without a control group.

How will you determine sample size?

How to Calculate Sample Size

  1. Determine the population size (if known).
  2. Determine the confidence interval.
  3. Determine the confidence level.
  4. Determine the standard deviation (a standard deviation of 0.5 is a safe choice where the figure is unknown)
  5. Convert the confidence level into a Z-Score.

How to calculate proportion of sample to experiment group?

To get the most powerful test we want to minimize the variance. If you have a total number of samples N and a proportion p of them are in the experiment group then n E = N p and N C = N ( 1 − p). The variance is σ 2 ( 1 N p + 1 N ( 1 − p)) = σ 2 N ( 1 p + 1 ( 1 − p)).

How big should the sample size be for a control group?

You should not allocate less than 20% of the sample to the control condition, save for situations when you are looking for large effects (e.g., 8 point lifts) and/or using large samples (e.g., 15,000 participants).

What is the true variance of the experiment group?

Suppose that the true variance (not sample variance) of the experiment group is σ E 2 and that you have a sample size n E. Likewise the control group variance and sample size is σ C 2 and n C. From your samples you will be examining X E ¯ − X C ¯ to test the hypothesis μ E − μ C = 0.

When to reduce the size of the control group?

Minimal losses in power occur when we shrink the control size to 40% [of the sample]. A 25% to 30% range is a good compromise, as this exposes 70% of the sample to the treatment, yet still does not harm power terribly.