There are many slogans around tests and affiliate marketing which is why only tests allow your campaigns to progress effectively. If everyone seems to agree on their usefulness, there are often very practical issues regarding their implementation, their application or their interpretation.

Let's embark on a special masterclass about testing and analysis of statistical data!

Testing : definitions and basic principles.

Before we even get to the heart of the matter, we must first quickly go back to the very concept of ab testing. It is based on a simple principle, measuring the performance of a web page by presenting two versions of this page to a homogeneous population.

Take an example, does the color of a button affect user behavior? You create a page with the green button, and another page with an orange button. You then present either one of the pages to your Internet users and measure a significant difference, and therefore a stronger performance of one of the pages compared to the other. Simple, right?

Simple in principle, of course, but we have already mentioned a number of conditions that will be at the heart of your concerns if you adventure yourself on testing.

First of all, the notion of homogeneity of populations. Indeed, if you offer different page versions to different populations, you will not be able to draw relevant conclusions about the relative performance of your modifications, which may be due to the population and not to the page itself. We are touching a first example of bias, and there are thousands!

Bias and statistical significance.

There are many biases, starting with the representativeness of your population. For example, if you present version A of your page to your mobile visitors and version B of your page to people on desktop, you are acting on the representativeness of your sample. In this case, you neglect the impact of your user's context on their behavior. Do you buy the same way and the same products on your home computer the products you would buy on your phone in a public transport? The answer is most probably no.

A bias is therefore any element that can influence the behavior of the sample you are studying, whether it is a difference in traffic source, devices, hours of the day, socio-demographic characteristics, ... or any other element implying that you do not have a uniformity in the population who will see version A or version B of your page.

But the biggest and most dangerous bias does not concern the population observed, but the observer himself. We have all spent time on a landing page thinking that inevitably it was going to be a hit, we launch it, a few clicks, a conversion. We think : "Bingo!" But then, nothing more… This is where the real problem of the representativeness of your test is.

Indeed, if contextual biases are very important to take into account, the worst that can be done is to rely on statistics that are too weak to draw conclusions. It is at this stage that we can use the notion of statistical significance. Who better than Fisher himself to explain what statistical significance is: “In statistics, the result of studies which relate to population samples is said to be statistically significant when it seems to reliably express a fact to which one is interested, for example the difference between 2 groups or a correlation between 2 data. In other words, it is then very unlikely that this apparent result is in fact misleading if it is not due, for example, to an incorrect sample (in), too small or otherwise unrepresentative (especially if the population is very diverse). This reliability generally results in values, differences in values, or relationships between values, sufficiently high or on the contrary low”.

Simply put, you have to make sure the results are reliable. For this we regularly use a very useful word to shine in society but often misunderstood "Bayesian inference". We are not going to enter a mathematics course here but we will retain that “Bayesian inference produces a statistical probability which is interpreted as the degree of confidence to be granted to a hypothetical cause. Source” So it is thanks to this system that we can trust our different tests. Generally, you wait until you reach 95% reliability before making a decision. We will interpret this 95% as the fact that there is a 95% chance that version A is actually more efficient than version B in our test.

Solutions for easy and meaningful testing.

So far we have done a lot of theory, it’s time to move on to practice. These problems of bias, significance and of tests set-ups have been encountered by almost all companies on the web, which is why dozens of solutions exist.

First of all, there is a very simple solution, that is producing a second version of your landing page, and using your favorite tracker, distribute 50% of your traffic to version A then version B. To see if your test is relevant, you just have to use one of the many online tools to calculate the strength of your test. We could name Neil Patel’s for example or VWO’s.

If you use the google suite, you can also use google optimize which is a real all-in-one and free solution.

We could still list hundreds of products more or less well made and especially more or less expensive, but what will above all make your tests successful is the preparation of these.

Create tests that really impact your performance: Hypotheses.

You now know how to create tests and avoid their pitfalls, but what do you test, and more particularly, why?

It’s fundamental to a successful test: hypotheses. Indeed, a successful test is above all a test where you learn. For this, solid hypotheses must be formulated.

Take the example of the title of your landing page. It reflects your marketing angle, but is that angle really the best? You will then need to make hypotheses about what the best angle is and test them. Marketing performance is above all marketing in which you do not seek to confirm a hypothesis but in which it is the mathematical confirmation of the hypothesis that forges your opinion.

This is how most successful companies have adopted the motto of being in continual testing, whether it’s for advanced UX functionality or simply earning a few tenths in conversion.

The impact of a test can be considerable on the revenue generated by a campaign. It is therefore important to understand and measure the tests if you want to take full advantage of them. Let’s say you’re passing your campaign’s lead to click ratio from 4 to 5% through your tests. The revenue from this campaign will therefore be increased by 20% and the margin you generate on it at least as much.

The corollary therefore of assumptions is the definition of performance, or the key points of performance indication that they allow to improve.

Ab testing VS Multivariate Testing

Of course, as you optimize, your hypotheses will become more complex and soon testing only one element within your operation will not be enough.

Indeed, the prerequisite of an ab testing is to change only one element between all versions to avoid any bias related to other variables. But if your interest lies in the analysis of several elements, a / b testing will be limited.

For example, if you want to test the color of a button and the text it contains, you’ll have to make at least the following variations :

• Color A, Text 1
• Color A, Text 2
• Color B, Text 1
• Color B, Text 2

Testing two factors with each two values already leads to 4 versions, since if you change two elements simultaneously you will not be able to test the relative contribution of one and the other.

It’s for this reason that a new generation of tests has appeared, multivariate testing. These tests solve the issues caused by ab test:

The first one is related to the resources and costs of ab tests. Due to the principles of significance and uniqueness, they require a great deal of traffic and therefore represent a certain cost, because if some tests prove to be more efficient, others on the contrary will obviously be much less efficient.

Multivariate testing allows, under conditions, to perform tests that test important combinations of factors and values. They therefore reduce not only the costs of the tests but also the time they take to be carried out.

This type of tests is also present in the tools that we presented to you previously. Even if they were originally designed by companies with highly advanced means, they are now accessible to everyone, in particular thanks to Optimize. It will therefore allow you to further increase your performance by testing more hypotheses at a reduced cost, but its benefits do not stop here.

Multivariate testing allows you to test not only the impact of each factor but also the impact of each combination. Indeed, the combined effects of several factors can have an impact on the success of your campaign that exceeds that of each factor in a unique way. Let's take an example, let's say you are testing a Title, a CTA (call to action) and an image. If you do successive ab tests you get the following results:

• Title 1 : 2% conversion
• Title 2 : 3% conversion
• Title 3 : 2,5% conversion
• Image 1 : 3% conversion
• Image 2 : 3,2% conversion
• CTA 1 : 3,5% conversion
• CTA 2 : 3% conversion
• CTA 3 : 2,2% conversion

According to these results, you therefore think that the best possible combination is Title 2, Image 2 CTA 1. However, to say that is not right. By sequentially analyzing the elements, you assumed each time that there was no interaction. But if your title is strongly correlated with CTA, it is likely that a coordinated title and CPA will perform better than the sum of their two performances. We can therefore imagine that a title such as “Get your free quote in 10 minutes” and a CTA “Send me my quote” will work much better together than individually. Although this example is self-explanatory, it remains very representative of the strength of the interactions within your landing page and their potential to improve your performance.

So it’s also a notable development that the ability for multivariate testing to measure these interactions and therefore take your marketing a step further.

If a tip should emerge from this article, do not neglect the power of tests - especially multivariates - they are the basis of the job of marketer and will make your campaigns powerful and constantly optimizing campaigns.

After a/b testing vs multivariate testing for can also read the article :
how to create an efficient landing page, check out our article here.