Achieving Product Market Fit (PMF) is one of the huge goals of an early stage startup. There have been countless posts written on this subject and I'd highly recommend spending some time reading them if you're at this stage of your startup.

One of the tools of the trade for testing different theories in your product is A/B Testing. Today we're going to look at how we A/B test different changes in the Revaluate Rails application.

Tools of the Trade

The main tool we are using currently is the easy to use Split Gem by Andrew Nesbitt. The README on the GitHub page provides all the details you need to get started with the gem.

Split Gem Screenshot

There are other services like Optimizely we have also used but I like having more fine grained control over the tests.

Tips and Tricks

Below are a few helpful tips that could help save you some time and frustration as you are getting started with A/B testing.

Caching

If you're using page caching in rails, make sure your A/B test is factored into the cache key or all users will see the same variation regardless of what the Split gem thinks.

The easiest way to do this is to set an instance variable in your controller for the Split value then use that in your view and add it to your cache key. For example:

# Controller action
def index  
  @ab_show_video = ab_test('ab_show_video', 'true', 'false')
end

# Cache key in the view
- cache ['report', @ab_show_video] do

# The actual test
- if @ab_show_video == 'true'
  # Show the video
- else
  # Show some text

Multiple Tests

By default, the Split gem doesn't allow you to run multiple tests on the same page. While you can run into problems testing multiple things on one page and it definitely affects the number of subjects you need to reach statistical significance, you can run multiple tests on one page by adding the following to your split initializer:

# config/initializers/split.rb
Split.configure do |config|  
  config.allow_multiple_experiments = true
end

A/B Testing the Bounce Rate

While the split gem makes it easy to track conversions on your application (i.e. user signups) we also want to be able to test how a change affects our bounce rate in Google Analytics.

We use Google Tag Manager in addition to Google Analytics which complicates things a little bit. The configuration and set up of your datalayer variables in GA and GTM go beyond the scope of this article but you can find some good information on that here and here.

For the application side of things, you can set the datalayer variables to the values from your split test:

- content_for :header do
  :javascript
    window.dataLayer[0]['abTest1'] = '#{@ab_test_1}';
    window.dataLayer[0]['abTest2'] = '#{@ab_test_2}';
    window.dataLayer[0]['abTest3'] = '#{@ab_test_3}';

Finally, within GA you can create a dashboard full of widgets for each test to easily monitor your bounce rate on each test. The setup of a widget looks like:

AB Test Bounce Rate Setup

And then your dashboard can look like this:

AB Test Bounce Rate Display

Hopefully this is helpful to you and your company when it comes to chasing the elusive Product Market Fit.

Feel free to reach out if you have questions or feedback: @timsegraves