Find solution
How do you rate the IDF?

How can I verify A/B results in a seasonal business whose conversion rate is a yearly curve?

  • Our peak converting month is Dec at 4.81. From there is goes downhill until July when it hits 2.55%. Then it goes back up until Dec. This effects every sampling of conversion rates I do. My experiment has now been live for a week. I can see its average conversion rate. I can compare it to the week before when the experiment was running, and the rate is slightly higher. If I take a long run average, it all depends on when i take it. If I take it 2 months before the test, of course the conversion rate is going to be higher. If I take it from last year at the very same time, yes the conversion rate is higher, but the conversion rate is higher from last year period. With or without any optimization. (due to running ads etc..) So the question remains. How can I accurately verify a result? Is it enough to say that it's conversion rate increase from last year over this week it more then the overall average conversion rate from last year? And that it is a bit more during the experiment week?

  • Answer:

    Clearly to give the best advice we would need a bit more context, but I have included some general thoughts below. To me it sounds like you should be looking at two different numbers to assess the relative success of your test. First, you should be looking at the conversion rates of your variations side by side in the current time period to determine which is converting best. Secondly, you can then compare the side-by-side winner to your conversion rate in the same time period of the previous year. When looking for a winner for this specific test, it will clearly be determined by which variation is performing better in the current time period. By also looking at the year-over-year comparison you can better understand how your changes are affecting your overall performance. Generally what this means is that you will select the same winner no matter what, but if your year-over-year performance is declining it might suggest that a wholly different test should be run to get your conversion rates trending upward again. As for accuracy, because your most important metric is the side-by-side in the current period, as long as you have sufficient conversions for statistical significance you can be assured of the accuracy of the test.

Cobi Druxerman at Quora Visit the source

Was this solution helpful to you?

Other answers

The only way to really get true results is running a test concurrently. Conversion rate is defined be so many parameters that would make the comparison from previous period of times (even a week ago) difficult. E.g. Changes in mixture of traffic resources - different quality of traffic, changes to your site, current promotions and many other external effects. Besides, in order to make sure that the results are significant (meaning most likely to be repeatable in the future) you can use a Significance Calculator like this one: http://commercesciences.com/tools/ab-test-significance-calculator?a=quorao I strongly suggest you would read the following posts:  - http://blog.commercesciences.com/ab-testing-pitfalls-which-will-lead-you-astray/?a=quorao  - https://www.linkedin.com/pulse/article/20141029231252-95126344-a-snake-named-conversion-rate?a=linkedino

Omri Yacubovich

If you are engaged with Seasonal business and want to apply A/B strategy and with the help of it decide the business strategy then I must say you can apply it after the season. The reason behind it you cannot take a chance with your business when your business is on Top. What you can do is when there is not a much demand in your business you can try something different in marketing in your products or services even you can offer some discounts and benefits to users etc and you need to do it for amount of time. I mean it is not a process for one or two day in business. You need to stuck with it atleast for month or quarter to get idea, is it working or not? If you got good response from people in any strategy you can modify, improve or target with specific target audience and increase your business.

Dharmik Rajput

You cannot be absolutely sure. Even with A/B testing you cannot be absolutely sure. The one way to get really close to be sure is use a real A/B experiment on the same data at the same time on the same audience. If you did not do that take the winnings and move on. New experiments await you :-)

Dennis Van Der Heijden

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.