Test New Email Concepts vs. Your "Control"

Your incumbent email creative-approach and offer are known as the "control."

When sending new concepts to a sample of your email list, simultaneously send your control to a similar sample. This enables an "apples-to-apples comparison." The fancy Latin term for this is ceteris paribus, literally translated as "with other things being the same," and usually translated as "all other things being equal."

Ceteris paribus is critical to email testing; otherwise, you may reach spurious (i.e., incorrect) conclusions.

For example, let's imagine that your incumbent email offer achieved a 5% response rate last week. This week, you tested a new email offer without sending a control sample, achieving a 4% response rate. Can you safely conclude that the incumbent offer outperforms the new offer? No. The difference in performance could easily be due to factors unrelated to the effectiveness of the new offer. For instance, the performance difference could be due to:

Testing helps you make better decisions for roll-outs if "all things are equal" except the factors being measured. That's why new concepts need to be "split-cell" tested versus your control email.

For instance, to split-cell test three new concepts against a control, you could sort your mailing list alphabetically and then populate each sample group with every fourth record, as shown:

To avoid over-mailing any portion of your list when sorting alphabetically, don't start your record selection on "A" every time. Instead, vary the starting points. Better still, if you have the capability, sort the mailing list "randomly" rather than alphabetically.