Structured CRO Program Helped RuneScape Increase Purchases

About 50% of users on the RuneScape website arrive as direct traffic. The other half of the traffic comes from referrals, social media, and email marketing campaigns.

Commenting on this traffic mix, Rob Marfleet, UX Specialist at Jagex, responsible for User Experience and CRO across the payment flow on the website (the payment gateway and its preceding pages), said, “This is mainly down to RuneScape enjoying a very loyal user base, with many players having played for several years.”

Rob appreciated the need to test and optimize various pages on the website and had a CRO team to assist him in optimizing conversions. This team included Jagex’s Dave Parrott, Payments Services Director and Nastassja Gilmartin, Payments Manager, who helped in identifying testing opportunities and analyzing test results.

Disha Ahuja, Client Success Manager at VWO, helped the team utilize the VWO platform to its full potential. A team of designers and developers facilitated implementation of winning test variants on the RuneScape website.

The CRO team focused on optimizing high-potential pages, that is, those that were closest to the payment gateway and required minimum effort in optimization. The Treasure Hunter page on the website was one such high-potential page that the team chose to optimize. The Treasure Hunter page let users buy keys to unlock treasure chests in the game. The treasure chests contain items that can be used within RuneScape.

Rob explained, “Treasure Hunter activity is an optional mini-game within RuneScape- keys are earned through play, but can also be gathered in bundles that are purchasable on the site.”

Optimizing conversions from the Treasure Hunter page was thus important.

This is what the original page looked like:

On clicking Continue on the Treasure Hunter page, users were directed to the Payment page from where they could choose from multiple treasure chest packages.

The RuneScape CRO team followed a structured process that involved the following steps and the use of VWO tools such as heatmaps, session recordings, and A/B testing:

• Setting a goal
• Finding opportunities for optimization
• Creating hypotheses
• Developing variations
• Analyzing test results

Setting a Goal
The goal of the optimization campaign was to grow revenue by increasing the number of purchases.

Finding Opportunities for Optimization
The CRO team studied a heatmap of the Treasure Hunter page and found that a significant number of users were clicking the Get Keys section on the page—a section that was not clickable. This led them to infer that users either wanted direct access to the keys or wanted to search for further information.

Heatmap of the original page

Next, the team analyzed session recording sessions on the page and observed that many visitors on the Payment page returned to the Treasure Hunter page. The team thus realized that the Treasure Hunter page probably did not offer users sufficient information about the treasure chest packages.

Creating Hypothesis
Based on the above insights, the team hypothesized that providing details about treasure chest packages on the Treasure Hunter page would lead to more conversions on the Payment page.

Developing a Variation
To test the hypothesis, the team created a variation of the Treasure Hunter page that included a new section highlighting 4 treasure chest packages. Here’s how it looked:

An A/B test was run to objectively determine which page, original or variation, performed better.

The test ran from August 15, 2016, to September 13, 2016.

The variation outperformed the original page and increased the number of purchases by almost 10 percent.

Test Result Report on VWO

Buoyed by the success of the A/B test, the RuneScape CRO team believed that further optimization was possible. The team realized that the offer of 4 treasure chest packages left users spoiled for choice. The team hypothesized that recommending one of the packages to users would help them choose better and, consequently, increase conversions.

Based on this hypothesis, the following variation was created:

The variation featured a Recommended package. This variation was pitted against the winning page from the first A/B test.

The variation won and further increased the number of purchases by almost 6%.