Multivariate Testing

August 21, 2019

---

In this ever progressing world, change has become a necessary mode of survival for organizations, big and small. Updating and upgrading your website or software products with different configurational changes, layouts, settings, privacy policy, and more has an effective move in the direction of improving its quality, SEO ranking, and subsequently increasing the users engagement. But, whether these changes work or not? Our article-multivariate testing is all about that.

What is Multivariate testing?

An important software testing technique, Multivariate testing (MVT) has garnered great significance among organizations around the world. It is a method in which different elements of a web page are tested individually, to incrementally improve the conversion rates and to get meaningful results.

The idea during this method of testing is to create different combinations of the elements of the target or landing page and observe which combination works the best and helps the organization reach their conversion goal.

It is simply a hypothetical process of analyzing the various combinations of elements and selecting best of them to optimize different pages, websites and apps.

Example:

We have a website with three elements; an image, a text box and a submit button. For each of the given element, we may have two different variations (colour, size, shape, text, image, etc.) as stated below:

Image(I)= I1 & I2

Text Box(TB)= TB1 & TB2

Submit Button(SB)= SB1 & SB2

Now, the possible combination resulting from each different variation in the element would be:

• I1 + TB1 + SB1
• I2 + TB1 + SB1
• I1 + TB2 + SB1
• I2 + TB2 + SB1
• I1 + TB2 + SB2
• I2 + TB2 + SB2
• I1 + TB1 + SB2
• I1 + TB1 + SB2

Thus, possible number of combinations comes out to be 9.

What's the need of Multivariate Testing?

A web page is composed of various set of elements, such as buttons, header section, sidebar etc. For example, an e-commerce website will have menu options, which will enable a user to make their own choice and order the item they selected. These different versions might be like this:

Case 1:

A user selects his choice of menu and clicks ORDER NOW

Case 2:

User makes his choice, a pop up is shown with his name and BUY NOW button.

There could be various versions of a single task or activity. Each set is tested to see how much traffic each version attracts. Based on the highest rate of traffic generated for a website, that particular version is chosen as a final option.

A diagrammatic representation of multivariate testing process.

Types of Multivariate Testing:

Performed on the basis of the traffic driven to the multiple versions of a web page, there are mainly three important types of multivariate testing, which play a major role in finding the best version of a web page, site, etc. These three types of multivariate testing are:

1. Full Factorial Testing: The most preferred and recommended multivariate testing, full factorial testing tests every possible variation and combination of a web page/site by equally driving traffic to them, until the best version is selected. The accuracy of this type multivariate testing is extremely high as there are no assumptions made by the team during full factorial testing and it is entirely based on the number and statistics, which ensures its reliability.
2. Practical Factorial Testing: During this type of multivariate testing, only fraction of all the variations and combinations are exposed to the website traffic and a static mathematical calculation and analysis is conducted by the team for the remaining combinations to reach the best conversion rate. Though, practical factorial testing requires less time, it is not reliable and accurate as multivariate testing, as only fraction of the variations are tested by the team here.
3. Adaptive Multivariate Testing: A new type of multivariate testing, in adaptive multivariate testing the real time response of the visitors on the web page is evaluated to determine the version of the various variations and combinations.

Process of Multivariate Testing:

By running a multivariate test, an organization can create a website or web page with most engaging and influencing elements, which will help them generate statistically significant traffic. Therefore, to get best results, it is crucial to run successful multivariate test, which can be done with the assistance of the following process:

1. Problem Identification: Before initiating the makeover/improvement of a website, web page, or an specific element, it is important that the team identifies the issue and digs into the data to find the level of engagement of the traffic with that specific element.
2. Formulate a Hypothesis: Once a thorough evaluation of the component or web page is finished, the team prepares a strategy to boost traffic engagement and conversion rates. With this hypothesis, they can implement best methods and reach their desired results.
3. Create Variations: During this stage of the process, the team prepares different versions of the component or web page, as multivariate test multiple versions and selects the one that is most beneficial.
4. Decide the Sample Size: After creating variations, it is vital that the team determine the details related to each sample, such as the number of visitors it needs to generate. These details will determine the success of the testing and will convey whether the target has been achieved by the sample or not.
5. Test All Components: Before driving traffic to the pages, the team tests all the components of the web page to ensure their accuracy. From landing pages and call to action buttons to links for ads, etc. the team makes sure all the elements are intended and do not become a hurdle in getting expected results.
6. Drive Traffic: Finally, after thorough evaluation, traffic is driven to the all the versions of the web page. This, though, is a beneficial process, can be time consuming, as a single test of multivariate testing requires a colossal amount of traffic.
7. Result Analysis: After the culmination of testing, the results of all the versions are analyzed and evaluated by the team and the version that offers expected results or generates more traffic is selected as the final option.
8. Learning: At the end of the process, the lessons learnt from the evaluation of a component is recorded for future reference.

How to Calculate the Number of Combinations required for Multivariate Testing?

No. of possible combinations to perform multivariate testing is calculated based on the number of variations in the elements of a particular section on a website. Mathematical formula for the same is

Merits of Multivariate Testing:

• Such type of testing helps a great deal in tracking the number of users visiting the website. Activities like shares, clicks, sign ups etc. are various ways to keep track of users visiting a page.
• Takes lesser time in comparison with its counterpart A/B testing.
• Multivariate is a more organised way of conducting tests by creating different versions of the same page.
• It lets us choose the best after analysing the various alternatives.
• In terms of design, font, and layout change, this type of testing is more flexible.

Demerits of Multivariate Testing:

Apart from offering various advantages to the team of testers, as well as the organization, multivariate testing also offers few disadvantages, which are stated below:

• To reach a significant statistics, multivariate testing requires excessive web traffic.
• It is complicating to setup test runs for this testing.
• It requires more number of variable version combinations to run tests and reach a result.
• It can be a time consuming process.

Ab Testing vs Multivariate Testing:

AB Testing	Multivariate Testing
1. A/B Testing, also known by the name of split testing, involves the evaluation of two different versions of a website. The versions may be different in one or more than one aspect such as designing, structure, layout and other similar elements.	1. Multivariate is a complex and a detailed form of A/B testing which involves the testing of each element and object of each particular section of a website as well as a web page along with the different possible combinations.
2. It is used to evaluate and record the user-response and behaviour on two different web versions.	2. It is used to assess user reactions but with different possible combinations of each and every section and element of a website.
3. It is used to evaluate and record the user-response and behaviour on two different web versions.	3. MVT is used to determine a few key variables and to create their versions & combinations.
4. It requires relatively less traffic for test execution.	4. It helps optimize an existing web page or website without doing a significant investment in redesigning it.
5. It tests two or more radically different pages.	5. It tests different content, while simultaneously determining most effective combinations.
6. A/B testing is best suited for redesigning a web page with different ideas, which may help improve the conversion rates for a business.	6. A tester can use segmentation for more granular optimization.

What to Choose: A/B Testing or Multivariate Testing?

As stated above, A/B testing is used to compare between two different versions of the website on various parameters, whereas multivariate testing is more refined level of A/B testing with the testing of each different sections and their elements in all different possible combinations in the same website version.

Therefore, if your site has less visitors or traffic then you the A/B testing approach, otherwise multivariate testing is a better option for a website with relatively large number of visitors.

Furthermore, A/B testing is useful in initially testing the home page or landing page in comparison to multivariate which is quite better for testing the existing and already optimized home page or landing page.

Conclusion:

In the end, we can conclude that though time consuming, multivariate testing is an effective and efficient way of testing various aspects of a website, web pages and its numerous elements, to improve conversion rates and to generate better results. By driving traffic to these components, it helps track the user engagement with a particular element, which further helps create an optimized website/web page.