10 Examples Of Winning Experiments That Drove Revenue And Performance Gains

0:06

Vipul: Hello and welcome to ConvEx, where we are celebrating experiment-driven marketing. My name is Vipul and I lead Co-marketing at VWO. Want to fix leaks in your precious conversion funnel? Try VWO.

Today joining us all the way from Pennsylvania is Craig Smith, who is the founder and CEO of Trinity Insight, a full service agency that specializes in internet and e-commerce optimization. I welcome you here with us, Craig.

0:35

Craig: Thank you very much for having me 

Vipul: Now before I let Craig start with his wonderful presentation, I just want to inform you that you can join ConvEx’s official networking group on LinkedIn and ask your questions there from this presentation. With that Craig. The stage is all yours. 

Craig: All right, thank you so much. I’m going to share my screen now and the presentation today. So really what we’re doing today is going through our 10 best experiments that we’ve been able to achieve over the last 12 months or so, as well as going through a variety of the key aspects of a conversion optimization methodology, So your business or brand can be best positioned to enhance the usability and make your site more effective, and obviously the convert more customers.

As a quick introduction about me: I have 20 years of experience in digital and e-commerce. I started as a strategist at eBay Enterprise before starting Trinity in 2006, and I’m extremely passionate about helping brands grow their businesses through data and optimization. And my goal for today is really that you come out of this with a plan of attack. You’re going to have clear examples of different types of experiments that you can run on your website to make it more effective, and that hopefully my presentation will be your favorite from today.

2:01

As for an overview, again, we’re going to be talking about our methodology and with the goal of providing for you the same framework that we execute for Fortune companies and Enterprise retailers, allowing your brand to see exactly the steps that you need to take to leverage data, research, design, and technology in unison so that you’re going to drive growth in your business. And then again, I think what you’re going to find is the most compelling point or points of today’s presentation is the clear examples and ideas, our best performing experiments. There’s 10 of them; each one of these experiments drove significant revenue and value and impact, and I hope that you’ll find this very interesting. Now, starting out though, I think it’s important to first look at – what is the methodology that you should embrace within your brand to optimize your website.

2:55

This is exactly the framework that we go through with retailers and brands, and I think in any business it can be leveraged. Really this is a first phase of researching and planning the effort you’re going to execute. When we start out, there’s a deep dive into Google Analytics, or whatever your platform is, to make sure that you have data integrity – this is critical. You need to make sure that you’re capturing all of the page level session data with appropriate analytical tracking because that’s going to be very important to decipher where you have opportunity. You also want to look at the website itself from a standpoint of usability and your page templates. If you’re an ecommerce retailer, for example, your homepage, your category page, your product page, there’s your cart, your checkout, you need to go through and someone in your team, hopefully, UX specialist can provide a heuristic analysis into what things could potentially be changed. And then the final piece of this research and planning phase is defining the revenue impact of what these changes are going to look like in terms of if we hit certain goals, what is the revenue impact to our brand?

And this is an example of that. You can see from a customer of ours if we at certain progression points are able to move the needle to a certain effect, what is it going to be to the overall revenue of the brand. And the reason you want to do this is because it sets up the business case. It provides clear data into performance and what your goals should be. And after all this is done, you then move to the testing program; and this is obviously where VWO is a critical piece of that puzzle where you’re designing the experiment, which is the creative, the conceptual design […] you’re launching in VWO, you’re running and collecting data, and ultimately you analyze and conclude, and then move forward into an implementation phase or, as I’ll mention later, hosting it into the cloud. So this is kind of, again, that initial research and planning steps to go through with the right data, your ideation, your data definition, and then moving into the designing, running, and analyzing of your information in your testing program.

5:13

Just a couple of quick snapshots and some examples. So from a standpoint of planning a test, you want to make sure that you have ideas and insights into what percentage of traffic is going to get through the test. In this example, it’s a hundred percent of all only desktop visitors. You want to also define what a goal is going to look like, what’s your test population, your expected launch, and your duration. All of this is going to be really important so you can define exactly what it is going to be going on in your experiment. You then want to create the concepts, the variations of what you’re going to construct. Here is a detailed list of what you’re going to change and either in an A/B split test, or A/B/C/D split test environment, or multivariate environment, you want to exactly show the creative and construct the creative for what you want to change.

6:01

You then obviously build it out on VWO, and we love you VWO. We’ve been a partner with VWO for over a decade now, and use it to essentially serve all of these template variations. But really, the key discipline here and the key component is the Javascript and CSS. We’re heavily manipulating the structure, the appearance of the webpage without the user, obviously, ever noticing. And then lastly, it’s really important when you’re constructing your optimization program and your efforts, you do a lot of quality assurance because browsers are different and you want to make sure that your new page, new presentation is going to render perfectly in every type of mobile browser, every type of desktop browser and make sure that you have the right quality assurance so you have approval.

As the test goes on you want to construct reporting into your performance. You can do this within Visual Website Optimizer. This is an example of how we tie in VWO to Google data studio, but again have a 48-hour validation period at the end of your test. Now moving on to the next part of the presentation. This is our top performing tests. This is what I think you’re gonna get the most value from today. I’m showing you some examples from customer data that’s going to really get that ideation exercise kick-started in your organization. And let’s jump right In.

The first one is a value proposition test. So what we’re doing here, we have a hypothesis that if we incorporated the value proposition of the brand we were optimizing throughout the entire web experience, we were going to get a higher rate of lead submissions, more interactivity with the site, and utilizing global information bars to present the key components of this service. In this case, they’re free, home, rental price analysis, the certified tenant match and the property match piece; included with that, you know a call to action. You can see here, the result was 45.70% increase in form submissions overall and the site traffic by incorporating this global value proposition. Obviously, this is a massive result to the business, drove the call center and sales to a higher degree of activity, and hopefully, and I’m sure it is dramatically impacting revenue on the back end. 

The next test that I’d like to talk about is a quote form test that we constructed for a business to business brand. What we did here is really focus in on this form that was presented all throughout the site. We look through […] presentation, condensed the required fields, reordered the fields with the hypothesis that people are going to convert at a higher rate when it’s just an easier step. And, then this is a big part of their business and by having it as the main driver of leads, we knew that we had to get at least a couple variations to make sure that we’re optimizing. And you can see one of our, I think it was the second experiment here, we were able to get a 28.4% increase in form submissions again, empowering inside sales team and an outside sales team with a lot more fresh leads, a lot more interactivity with the business that’s going to increase the enterprise profit and performance.

9:31

The next one is coming from an e-commerce environment where this customer was looking for alternative ways to present packages. The control variation that you can see presented the packages in this type of tab base breakdown in which users have to click on the tabs to see the different types of packages that were presented. You know, our thesis was that we were going to increase the clarity of the content part by presenting it all in one single presentation. As you can see, there’s a pretty significant design change here where we presented them all in one single presentation, made it a horizontal content layout versus a vertical content layout, and we felt this would drive a greater level of conversion. And, what you can see here is the result was a 19.42% increase in progression rate. Now, this is an important concept to understand.  Progression rate is the percentage of people that go from this page template to the next page template in their funnel. It’s really important that you measure your experiments on progression rates, or a single goal in a page versus the overall conversion rate because, theoretically, if your progression rate goes up 19%, and the rest of your funnel holds true, your conversion rate is going to also increase it 19 percent. But you don’t want to measure that conversion rate in a vacuum in a specific page template test. So, in this case, they had almost 20% more people click on that book now or view details button that increase them down the funnel. So a really effective test for this brand.

11:20

The next test that I want to talk about is check-out value proposition, and this can be almost any type of conversion funnel where you’ve got that last mile, you have someone almost at the conclusion of when you’re going to convert them. In this case, what we noticed was that a lot of people were abandoning at this final step, when they had to, you know, apply or start their trial, use a credit card – that was not happening to the degree for which we thought it could be achieved. What we did here is place the value propositions in close proximity to the order summary box. And, by placing an increased emphasis on them, we expected to see an increase in users starting the trial, which is ultimately the conversion they wanted, the customer. And you can see here, we had a 54% increase in trials by incorporating these value propositions in close proximity to the order summary. And that’s the key thing here to take away: think through your value propositions, what’s important in your business and your brand, and incorporate them near that final step, that final click that you need to take on an e-commerce. We’ve done many experiments where we’ve re-emphasize security, we’ve re-emphasize shipping, we re-emphasized returns at this step, and you want to make sure that you do the same thing at your final step.

12:51

The next test I want to talk about is a mobile check-out presentation test. And this again is specific for ecommerce, but really holds true again for any type of business in lead gen as well. What we did here is we redesigned the check-out presentation to really just follow more modern design trends and best practices. With our goal of increasing overall conversion on mobile devices, we looked at the data: the desktop conversion was trending upward but, the mobile conversion was a laggard. And we knew that this page was not performing. And you can kind of see here in the control and variation examples, It wasn’t drastic changes. It was making the form boxes bigger. It was not using black text outlines. It was making a call to action clearer and more easily clicked upon with thumb or finger, and this test saw strong lifts in conversion immediately after launch. The styling to the more modern presentation but the primary focus of the changes were just making this check-out flow more intuitive and user-friendly. But again, a great impact is it got a 7.5% lift in check-out conversion.

14:06

Here’s another mobile checkout presentation that I thought was important to show and this one got an 11.2% increase to add to cart rate. The hypothesis of this experiment was that we really wanted to provide users with easy access to the product details from the buy box. What we did, we used VWO to incorporate the overview, reviews, and knowledge component higher up as a jump point, to get people to the content that are going to prove that is going to provide the knowledge they’re looking for. In this example, the winning variation again, incorporate those links above the product image and those links anchor directly to the corresponding details down the page. This led the users to the information they wanted faster, and more prominently above the fold which again helped them in making the conversion decision. And this again drove an 11% increase to the add-to-cart rate.

15:05

This was another very interesting experiment. The way I titled it was product page: old versus new. But you can see what we’re doing here, we’re presenting the personalization form and its entirety, and allowed this brand which sells personalized children’s books the ability to mirror a previous function that they had in their business years ago. So our customer came to us and showed us this page, we noticed it wasn’t performing, and we went back and looked at old versions of the website in the Wayback machine, worked with the customer in conducting some primary and secondary research into these pages, and incorporated a new presentation for personalizing a book. You can see, in the top version up there, it works as a control. This was the existing page where you had this vertical layout in which the user had to include the child’s information in this step-based sequential module.

16:03

The variation takes a different approach. We have a simpler create your book component that’s drop-down,  or […] presented in a module that allows someone to put in the information, click add to bag, and start the process. And, what we found was, the updated personalization process and page layout was much more effective than the existing design. This drove a 21% increase to overall conversion and probably a higher percent increase to the page progression rate as we talked about before.

16:36

Another test that we ran for an ecommerce retailer was to provide intuitive design, to incorporate the best practices to simplify the fields. In this example, we’ve condensed the billing address. We’ve made it so that the billing address can associate to the shipping address on a click versus showing both fields. When you showed both fields, and you make the user type in their address information a second time, it presents dramatic fatigue, which can cause people to abandon. We wanted to not have that in the presentation and/or in the site presentation. And use VWO to again simplify the check-out, to make it easier, to make the call to action a little more cleaner, to include another place order button into the page. And, you can see here, we had nearly 12% increase in overall conversion by incorporating this cleaner, more optimized, more user-friendly and intuitive checkout process.

17:44

Here’s another great example relating to urgency. If your brand and your business has any type of urgency that they can incorporate into the call to action within your funnel, I highly recommend doing it. If you’re an informational product, maybe it’s a countdown timer. In this example, which is ecommerce, we incorporated stock messaging for products with 10 or less items in stock. And, our thoughts were, this would dramatically increase the effectiveness of our pages. And you can see that the control version on left, the variation on the right. The variation included stock messaging that had a data connection to pull in that stock number and it had a dramatic increase to conversion. In this example we were able to have a 30% increase in the conversion rate by including the stock number right above that call to action, right above that add to cart button.

18:40

So the takeaway here: think through, what elements you can bring in to your conversion action that’s going to give some type of urgency, either time or inventory, that’s going to make the user have more of a need to act now.

19:00

And this next example is showing an order summary. This example is that redesigning an order summary will increase clarity and increased conversion. In this example here, what we did was really make aesthetic changes, and aesthetics are going to be a key part for your program and how you execute conversion optimization. Functionality is very important, speed is very important, but aesthetics are also very important. In this example, we really didn’t change much functionality in the page. We changed the aesthetics, incorporated, you know, a different layout for how we wanted to present the brand and the calls to action for this informational product. And what we achieved was a 25% increase to the overall conversion rate by including the order summary on the right side, and providing users with the check-out the experience that they would likely experience on an ecommerce store.

19:55

So now let’s talk about your brand. If you would like a free user experience analysis of your website, we love to give you one. Email me at [email protected]. I’ll be happy to align you with one of our user experience specialists who will construct the document for you and go through it with you, and share their findings and perspectives. And out of that document you’ll get a lot of new ideas to how to execute a conversion program on your site. And again, this is a hundred percent free and you can also request this at trinityinsight.com. Just go there, follow the conversion optimization user experience pathway, and you’ll have the ability to request it there. So, with that I want to thank you for your time today. I really appreciate VWO for giving me the chance to speak. But, I’m going to now open it up for any questions and any further commentary.

20:55

Vipul: That was really an insightful presentation, Craig. Quite amazing examples that I’m sure that the audience would love to replicate in their respective online businesses. 

Craig: Thank you.

Vipul: Okay. So before I take up the questions, I just wanted to highlight this one test that you mentioned in your presentation earlier, about the mobile check-out rates that showed an uplift of 7.58% in the mobile checkout rates. Small examples there that the businesses need to take care of and improve conversion rates, isn’t it? 

Craig: Yeah, and you know, I always say this to folks that I meet who might not be familiar with conversion optimization – it’s very much an iterative step by step effort, it is not a silver bullet, it is not something in which you’re going to just have three tests and be done. It is very much like, I’ll make this as a kind of a joke analogy, it’s like baseball – you’re gonna get a single, you’re gonna get a single, you’re gonna strike out, you’re gonna strike out, you’re doing a single and then boom you hit that home run; and you get that winning experiment that really is one of the ones we showed which will dramatically affect your business. 

Vipul: Yeah, and then when you mentioned ‘step by step’, you reminded me of the importance of having a proper hypothesis in place for a successful conversion optimization program. 

Craig: Yeah. Yeah, it’s you need to go into each experiment with a thesis and you also need to go into each experiment with clear data. The data is going to tell you exactly what is happening. Now research and the tools that VWO provides, heatmaps, recording, surveying, that’s gonna tell you why it’s happening. But the data is going to tell you what’s happening. Both are really really important. You cannot just go into a testing program and just jump in – Oh! I wanna test the homepage. Okay. Test it for what? Do you [know] is the problem progression into a product page or category page? Is the problem abandonment? Is the problem just lack of clarity and the bounce rates really high? So there’s a variety of things to consider and that’s why you need to make sure you have clean data that’s going to tell you the approach to go.

22:53

Vipul: Yeah, that makes a lot of sense. So, I think I can now move on to the questions that I have for you. So the first question is – what are the most critical components of a conversion optimization workflow that you outlined?

Craig: Yeah! I think two components that are really important. One is, I mentioned before, quality assurance. Before you launch an experiment, you need to make sure that it is rendering perfectly on all browsers and device types. Because, if you don’t do that, if you don’t spend the time and the resources to make sure it takes place, what’s going to happen is you’re going to launch the experiment and maybe it’s not working in a certain version of Explorer, or may not [be] working in a certain version of Safari and your data is polluted.

23:41

So the experiment is going to be null and void because you’re going to get potentially, you know, calls in your call center or contact form saying hey your website’s broken or and it’s just not the approach you want to have. So to do the quality assurance is very important. And the second piece I would say is that data engineering discipline where you can ensure you have great right data, but also do the custom segmentation. That’s going to tell your business where the key lagging areas of your website are. Because every website is different, every website has different areas of the site that have monetary opportunity, they needed to be documented and you need to model out at a low, mid and high effect. – like I said single, double, home run – what is the revenue impact that we’re going to see? What is the lead flow impact that we’re going to see? And that is going to make your whole process much much more effective.

24:40

Vipul: Perfect. So my next question to you is – What are the key disciplines that are needed for a brand to reach full potential in UX optimization? 

Craig: Yeah, I would say there’s two disciplines that, you know, outside of the data piece that we just talked about that you need to make sure are in your organization or you have access to it to some degree. The first is Brand Aesthetics  – the designer, the person who’s constructing, what we call, a conceptual design, the new look and feel of a page, that’s going to be tested and experimented against the control version. And, you want to make sure that you know, this is part of the reason why folks hire us is because you want a third party perspective or to someone who’s really fresh to your organization, who hasn’t been with the company for 10 years, because it’s going to give you new ideas and new ways to present your brand value proposition, as mentioned earlier. The second discipline that I think it’s really important is Javascript.

25:38

The web, internet browsers, HTML5 – it’s JavaScript-driven. Every website you see is JavaScript-driven. And with the proliferation of JavaScript and how great it is to work with VWO, you can essentially make any type of change you want in a presentation with the right JavaScript chops. So, JavaScript is a technology, and more specifically, jQuery is a technology that needs to be leveraged, and that discipline needs to be embraced within a testing program.

26:11

Vipul: Right! So, that makes me curious about the conversion challenges or obstacles that your customers face. And also, how do you help overcome them?

26:22

Craig: Yeah, great question. So the first [thing] I would say is executive oversight. Too often, we get situations in which the CEO, the VP of marketing, they have their favorite design. They have the design that they might’ve approved six months ago or a year ago that we feel is underperforming but they might have this legacy bias, and they don’t want to change for change’s sake. Now, that sometimes can add levels of friction in an organization to have an experiment or an optimization effort for a certain page, get traction. The way to overcome that is data. And it’s that slide I showed earlier that says – this is what we can be making if we make these changes and hit our goals. When that data is presented to an executive team, I’ve never seen any instance in which that hesitation from the executives is still persistent. So that is the way to overcome it. 

You know, there’s a funny saying in the up position world the HiPPO – the highest and informed paid person in the room. They get on your side and become an ally for you when you have the data and the revenue models that paint the vision of what you’re trying to achieve. The second challenge, and we use VWO heavily overcoming it, is the implementation after experimentation. So let’s give a hypothetical example, let’s say we run a check-out page optimization test, and it wins, and it outperforms the control version by 20%. In that instance, we’re giving high fives, we’re all excited, we know we’re going to make more money for the customer at that point.

28:17

It’s now up to the customer to implement the new version, but what frequently happens is technology backlogs, alternative priorities, code freezes – things that delay the actual implementation of the new presentation, what we call hard coding the new presentation. In this instance, what we used VWO for is to render via the cloud. We then say – okay, you can’t implement it for six months. No problem. We’re going to render that winning variation via the cloud using VWO to all of your traffic, so you get the economic and monetary benefits from that experiment. It is now to a point where VWO obviously and the web, in general, the cloud is so fast that there literally is no difference between cloud rendering and hard coding from a standpoint of speed.

29:09

So I have some customers that literally are changing their check-out and cart pages every other month, and only do it be the cloud, and look to the cloud as their lever to achieve what I call agility, speed. It’s giving marketers the ability to get ideas in the wild faster and get it out to their customer base with that same level of speed. So those are the two main obstacles – executive, kind of friction, potentially that could overcome with data; and implementation bottlenecks that could overcome using the cloud and VWO.

29:47

Vipul: That’s right. So my next question is basically something that I’m personally curious to know more about, is what books are you currently reading? 

Craig: What I’m currently reading right now? I just finished it, I just got back from vacation yesterday morning, and I just finished the book called 10% Happier. It’s all about meditation and the story of an ABC news anchor who embraced Buddhism and meditation as a way to essentially relieve anxiety and present an alternative way of living. And, it was recommended to me by a friend and I loved it – it was a great great book.

30:23

Vipul: That sounds like an interesting book. My last question is – how can the audience connect with you?

30:31

Craig: Yeah, so you can connect with me directly and I would recommend [to] email me at csmith@trinityinsight.com – that’s insight singular. And you can also go to our website at www.trinityinsight.com. Send us a contact request or a hello message, and we’ll reach out. And, like I mentioned, we do provide free user experience review documentation. To give you some ideas, it’s a 10 to 20-page document that I can guarantee your audience will get a lot of value from. So, feel free to request that for your website and we can show you what potentially should be changed to optimize your conversion rate. 

Vipul: Absolutely. And if you want a free UX analysis of your website, do reach out to Craig directly on his email address or just visit trinityinsight.com. With that we’ve reached the end of the session. Thank you so much, Craig for your time and effort that you’ve put into creating this wonderful presentation, and enlightening our audience. 

Craig: Thank you for having me. It was a pleasure.