The 10 Best A/B Testing Books for Practitioners of All Levels

Last Updated on January 10, 2020 by Alex Birkett

A/B testing is an important skill for anyone to learn, whether you’re a marketer, product person, designer, or an analyst.

It’s also a great framework for managers, as Elliot Shmukler has noted. Once you learn about A/B testing, you look at decision making, probability, innovation, and risk differently. It’s an attitude shift as much as a tactical application in marketing or UX.

But as simple as the concept is (pit one online experience – the control – against a new one – the variant), the execution is filled with frustration and nuance.

I’ve written extensively about A/B testing on this blog, so you can definitely check that out. Additionally, if you prefer courses, CXL Institute has the best ones.

However, as a heavy reader, I’ve gone through dozens and dozens of books directly and indirectly related to A/B testing. Some are really good, more aren’t. This list contains the really good ones.

The Best 10 A/B Testing Books for Practitioners of All Levels

  1. The Innovator’s Hypothesis
  2. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
  3. Statistical Methods in Online A/B Testing
  4. Statistics for Experimenters
  5. Bandit Algorithms
  6. The Drunkard’s Walk
  7. The Black Swan
  8. Antifragile
  9. Don’t Make Me Think
  10. Lean Analytics

Note: I’m using affiliate links here, so I’ll make money if you buy these books. A win/win really as you’ll get some great knowledge and I’ll get some tiny percentage from Amazon!

1. The Innovator’s Hypothesis

By Michael Schrage

The Innovator’s Hypothesis is one of the better A/B testing books that focuses on the actual strategy behind experimentation.

First off, it puts forth an excellent business case for experimentation. If you need executive buy-in, this is the book to read. It’s short, and it gets into the meat of the matter pretty quickly.

Ideas are overrated, and good ideas are unpredictable. You’re best off making it cheap and easy for employees to run and analyze experiments. It mitigates the risk of bad ideas and decisions, and perhaps more importantly, allows everyone to capture the upside of great ideas that otherwise wouldn’t have been tried.

This is one of my overall favorite books on experimentation, as it doesn’t rely on overly industry-specific terminology like the CRO specific books. It’s also not incredibly technical, so everyone should be able to grok this one.

2. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

By Ron Kohavi, Diane Tang, Ya Xu

This is my favorite A/B testing book that specifically covers digital experimentation – controlled experiments on the web or on your product/application. It’s not actually released yet at the time of writing, but should be soon in 2020.

It’s excellent for many reasons:

  • Ronny Kohavi is a trusted authority, having built up Microsoft’s experimentation platform and program to a few hundred team members.
  • The book reaches technical depths when it comes to statistics and also on the sections describing proper testing platforms.
  • However, it also covers high level case studies on why to invest in proper experimentation in the first place.

The scale at which companies like Microsoft, booking.com, and Google are running experiments is incredibly impressive, and the cultures they have built around testing are equally inspiring. This book cracks open the secrets of these big companies, and it doesn’t shy away from advanced and technical topics. True experts and practitioners will love this one.

3. Statistical Methods in Online A/B Testing

by Georgi Georgiev

Statistical Methods in Online A/B Testing is my favorite book specifically centered on the statistical aspects of experimentation. It’s super comprehensive, though it’s not impossible to read as a layperson (you will, however, have to read some sections quite slowly or re-read them to truly grok).

This book is hyper-focused on digital experiments, particularly so in the field of ecommerce conversion rate optimization. As such, it won’t be as broadly applicable as others on this list, but if you’re reading this article, it’s very likely you’re interested in the specific applications for conversion rate optimization.

If you want to dive deeply into the statistics of A/B testing, you really just need this book. It’s got the depth of a textbook, but it’s actually readable. And Georgi Georgiev really knows his shit.

4. Statistics for Experimenters

By George E.P Box

Now, if Georgi’s book were like the accessible version of a statistics textbook, this is the actual textbook – it’s super dense but filled with technical information useful to serious analysts. I’m going to wager that few people will want to read and finish this one, but the most serious experimenters, analysts, and data scientists really should pick up a copy of this one.

You don’t have to read it through like you would a novel, but you should have a copy for reference. It’s wildly comprehensive and practical, albeit very dense for the layperson.

5. Bandit Algorithms for Website Optimization

By John Myles White.

I like to think of bandit algorithms both as a good framework for decision making and optimization and as a literal technical solution for certain optimization problems.

The multi-armed bandit problem is generally described like this: you have a set of slot machines, and they have varying reward systems. Some give out more money over time than the others, but you don’t know which ones they are. What’s your ideal strategy for figuring out which machine gives the highest rewards, preferably while limiting the amount of time spent pulling suboptimal machines?

Bandit algorithms (of which there are many flavors/types) are used to balance exploration (the pulling over many different levers) with exploitation (pulling the optimal arm more frequently). In online optimization, this means adapting the traffic allocation between the control and the variant(s) in real time as the algorithm learns more about which variant is optimal.

This book is the best explanation as well as manual on technical implementation of bandits.

6. The Drunkard’s Walk

By Leonard Mlodinow.

The Drunkard’s Walk is an entertaining book that covers a ton of different topics on probability and randomness.

If you don’t have the patience (or desire) to sit through a textbook, but you still want to intuitively grasp difficult topic matter in statistics and probability, this is an awesome book. The writing and storytelling is phenomenal.

I read this over the course of a few flights and have recommended it to several people in conversion optimization. I haven’t heard any negative reviews back yet.

7. The Black Swan

By Nassim Nicholas Taleb.

I’m currently re-reading all of Nassim Taleb’s books, and I can say with full confidence that this one is one of the most influential books I’ve ever read. It has had a deep and lasting impact on how I look at experimentation and conversion rate optimization, but it has also impacted how I look at life and decision making more generally.

You won’t regret picking this one up. It will introduce you to all sorts of useful ideas and heuristics (one of my favorites being the narrative fallacy).

8. Antifragile

By Nassim Nicholas Taleb.

I could have put all of Nassim Taleb’s books on this list, but I’m just listing The Black Swan and Antifragile because they have direct applications to A/B testing.

Antifragile in particular has had an influence on how I build CRO systems and how I build strategy around optimization and decision making. We operate in informationally-opaque arenas for the most part, and experimentation can help us capture the upside to randomness (or the optionality involved in experimentation). With this in mind, it becomes apparent that you’re often best off a) lowering the cost of running experiments (which by nature increases the ROI) b) being open (and welcoming) surprising results, even if (and especially if) they don’t conform to your previous world-views and c) testing unintuitive and wide ranging ideas, without cornering yourself into predisposed patterns.

Doing this can have big wins over the long course of an experimentation program.

9. Don’t Make Me Think

By Steve Krug

This is the best book I’ve read on usability and user experience. UX and experimentation go hand-in-hand, of course. Some of your best insights for conversion rate optimization opportunities will come from running user tests. This book is an awesome introduction and manual for how to do it right.

10. Lean Analytics

By Alistair Croll and Benjamin Yoskovitz

Lean Analytics covers tons of analytics and data topics, mostly aimed at startups. While analytics (like user testing) isn’t necessarily the same thing as A/B testing, it’s something absolutely necessary to know about. How are events logged? How can you discover optimization opportunities? What does cohort analysis have to do with it? This book helps you answers some of these questions and many more.

Analytics is a big subject, but this book will give you a good overview and practical guide to getting started.

Conclusion

Books are great, and you should buy everything on this list and read each book twice. But at a certain point, you just have to start diving in and learning by doing.

I’d particularly recommend reading the statistics-heavy books here; that’s the type of knowledge where you really do benefit from an academic and theoretical underpinning. Other than that, N.N. Taleb’s books will help you in areas beyond A/B testing, but they’ll definitely help you with testing and strategy as well.

This is just a short sampling of all the books that have helped me with A/B testing, but I feel if you read all of the above, you’ll be far beyond most people who are running A/B tests today.

Alex Birkett
Alex Birkett is a product growth and experimentation expert as well as co-founder of Omniscient Digital, a premium content marketing agency. He enjoys skiing, making and experiencing music, reading and writing, and language learning. He lives in Austin, Texas with his dog, Biscuit.

4 Responses to “The 10 Best A/B Testing Books for Practitioners of All Levels

  • Jacob Jarnvall
    4 years ago

    Great list Alex! If you haven’t read it yet, I believe you should take a look at “Hacking Growth: How Today’s Fastest-Growing Companies Drive Breakout Success” by Sean Ellis & Morgan Brown.

    • Alex Birkett
      4 years ago

      I read and loved Hacking Growth. I might have to reconsider adding it to the list, though I don’t remember any particularly novel sections on A/B testing that aren’t covered by others on this list

  • Nate Medina
    4 years ago

    Thanks for the write-up, Alex. I’m curious if you know when “Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing” will be published? You’ve piqued my interest in that one. Thanks!

    • Alex Birkett
      4 years ago

      Good question! I’m hearing around March 2020, so a couple months out. I think it’s on pre-order on Amazon now