Achieving Product Market Fit (PMF) is one of the huge goals of an early stage startup. There have been countless posts written on this subject and I'd highly recommend spending some time reading them if you're at this stage of your startup.
Tools of the Trade
There are other services like Optimizely we have also used but I like having more fine grained control over the tests.
Tips and Tricks
Below are a few helpful tips that could help save you some time and frustration as you are getting started with A/B testing.
If you're using page caching in rails, make sure your A/B test is factored into the cache key or all users will see the same variation regardless of what the Split gem thinks.
The easiest way to do this is to set an instance variable in your controller for the Split value then use that in your view and add it to your cache key. For example:
# Controller action def index @ab_show_video = ab_test('ab_show_video', 'true', 'false') end
# Cache key in the view - cache ['report', @ab_show_video] do
# The actual test - if @ab_show_video == 'true' # Show the video - else # Show some text
By default, the Split gem doesn't allow you to run multiple tests on the same page. While you can run into problems testing multiple things on one page and it definitely affects the number of subjects you need to reach statistical significance, you can run multiple tests on one page by adding the following to your split initializer:
# config/initializers/split.rb Split.configure do |config| config.allow_multiple_experiments = true end
A/B Testing the Bounce Rate
While the split gem makes it easy to track conversions on your application (i.e. user signups) we also want to be able to test how a change affects our bounce rate in Google Analytics.
We use Google Tag Manager in addition to Google Analytics which complicates things a little bit. The configuration and set up of your datalayer variables in GA and GTM go beyond the scope of this article but you can find some good information on that here and here.
For the application side of things, you can set the datalayer variables to the values from your split test:
Finally, within GA you can create a dashboard full of widgets for each test to easily monitor your bounce rate on each test. The setup of a widget looks like:
And then your dashboard can look like this:
Hopefully this is helpful to you and your company when it comes to chasing the elusive Product Market Fit.
Feel free to reach out if you have questions or feedback: @timsegraves