Comments on: A/B Test Statistical Significance Calculator [Free Excel]

By: Dawn Stoner

Dawn Stoner — Thu, 30 May 2019 20:47:19 +0000

This spreadsheet is awesome! Finding confidence intervals in rivals’ tools is increasingly a challenge and this solves the problem beautifully.

By: Adam Lundquist

Adam Lundquist — Mon, 22 Apr 2019 07:45:44 +0000

In reply to Henry. I second this request - I am looking for one that tests 4 variables

By: How Long to Run A/B Tests: the Known and Unknown | Articles in IT and more

How Long to Run A/B Tests: the Known and Unknown | Articles in IT and more — Mon, 13 Jul 2015 14:39:01 +0000

[…] tip goes to VWO's Excel calculator, which I modified to generate the curves […]

By: Paras Chopra

Paras Chopra — Wed, 18 Jun 2014 11:38:57 +0000

@Yael: yes, you’re right.

By: Yael

Yael — Thu, 12 Jun 2014 07:14:54 +0000

Hi Paras,
regarding you answer about revenue significance – as far as I know the formula assumes normal distribution. So if the revenue is not normally distributed (which is probably right for most commerce sites), you can’t use this formula. Isn’t that right?

By: Henry

Henry — Wed, 28 May 2014 05:59:46 +0000

Hi Paras, this is a great tool to get statistical confidence in A/B testing. Thank you for sharing!
Just wondering if you have multivariate significance calculator as well? Or can you let me know where I can get it?

By: Paras Chopra

Paras Chopra — Thu, 02 Jan 2014 09:42:48 +0000

In reply to Dannie. @Dannie: you use the same formulae, but plugin in real average and standard deviation that you get from revenue figures. For conversion rate, we calculate it using the formula p*(1-p)

By: Paras Chopra

Paras Chopra — Thu, 02 Jan 2014 09:41:20 +0000

In reply to Sanjay.

No, both values are p values, but in the excel file in C18 =(control_p-variation_p) and in the calculator it is (variation_p – control_p).

So in the Excel file, the p value for the given values is 0.957435466 ~ 95.7% and in JS it is 1 – 0.957435466 ~ 0.043 ~ 4.3%

So both is right, though it seems as if the order control_p-variation_p is more common.

By: Sanjay

Sanjay — Thu, 19 Dec 2013 19:40:52 +0000

The p-value mentioned here (https://vwo.com/ab-split-significance-calculator/) doesnt actually seem to the be the p-value. It seems to be calculating confidence.

For example, in the excel image above the follow values in control (visitors: 2000, conversion: 134), and variation (visitors: 3000, conversion: 165), the p-value is 0.95.

The same values when used in the calculator give a value of 0.043. It therefore needs to referred to as confidence instead of p-value