I Took Udacity’s Free A/B Testing Course by Google: Here’s What I Learned

Image by Author

I am a data scientist with a computer science background.

1. Google Cybersecurity Certificate – Get on the fast track to a career in cybersecurity.

2. Google Data Analytics Professional Certificate – Up your data analytics game

3. Google IT Support Professional Certificate – Support your organization in IT

When I first entered the field, I struggled to pass data science interviews due to my lack of knowledge of underlying math and statistics concepts.

One of those concepts was A/B testing.

While I excelled at the coding portion of the interview, I’d often freeze up when asked questions on inferential statistics and experiment design.

To bridge this knowledge gap, I took the free A/B testing course on Udacity, taught by senior data professionals at Google.

In this article, I will break down what I learned in the course and explain how you can use it to build an understanding of inferential statistics.
 

What Is an A/B Test and Why Do We Need It?

 
Let’s say your company wants to launch a new type of candy alongside the existing dessert being sold.

The hypothesis is that new candy will drive more users to stores worldwide, leading to an increase in total sales.

Before launching this new candy in all of its stores, the company needs to understand whether it’s profitable to do so.

It must first test this hypothesis on a few stores, and only expand production if there indeed is an increase in sales.

This is where A/B testing comes in.

In simple terms, A/B testing compares two versions of something to figure out which is better.

An A/B test typically comprises the following components:

• Variant A: In our dessert store example, this is the variant that does not sell the new candy. It is also known as the control group.
• Variant B: This is the variant for which a change is applied. In our case, this includes the group of stores that offer the new candy. It is called the treatment group.
• Hypothesis: This is a clear statement of what you expect to happen. Here is the hypothesis in our example: “Stores that sell the new candy will see higher average sales than the stores that don’t.”

The hypothesis statement can further be broken down into a null and alternate hypothesis, which will be covered in this free course.

What You’ll Learn in Udacity’s Free A/B Testing Course

1. Overview of A/B Testing

You will learn how A/B testing is being used at large companies like Netflix, Amazon, and Google.

I found this insight valuable since you get to learn about the practical application of A/B testing from instructors who work as statisticians and engineers at Google.
 

2. Inferential Statistics and Experiment Design

The course then covers everything you need to know about experiment design — from the metrics you’d like to measure, to concepts like confidence intervals and statistical significance.

You will learn:

• How to construct a confidence interval
• The different statistical distributions (normal, binomial)
• Best practices to design the null and alternate hypothesis.

This section of the course is demonstrated with a simple real-world example of increasing website click-through rates, which I found interesting.
 

3. Policy and Ethics for Experiments

This lesson will cover the ethical considerations involved in conducting experiments — understanding whether participants of the experiment are being exposed to risk and obtaining user consent.
 

4. How to Size an Experiment

You will learn to determine an appropriate sample size for the experiment, which involves measures like statistical power, significance levels, standard deviation, and the Minimum Detectable Effect (MDE).

If these concepts sound foreign to you, don’t worry!

I went into this course with little understanding of the above statistical topics but was able to follow along easily thanks to the additional learning material and notes provided.
 

5. How to Analyze the Results of an Experiment

Here, you will learn about the types of metrics that must be constant across all groups in your experiment. These are called invariants.

You will also learn to analyze whether the result of an experiment is statistically significant — for both single and multiple metric experiments.
 

6. Turning Results into Actionable Insight

In this section, you will learn to use the results of an A/B test to make a business decision.

For example, if you find that stores selling the new candy do experience a significant improvement in sales, what’s the next step?

Do you roll out multiple batches of this new candy to all stores worldwide? Or do you start with a single state or country?

Perhaps you’d like to iterate on the experiment based on the data you’ve gathered.

This section of the course is heavily focused on the business impact of A/B testing and aids with the decision-making process once the test has been conducted.
 

7. Final Project

In the final project, you will be provided with real data from an actual experiment that was run by Udacity.

Using this data, you’ll have to answer a set of questions on experiment design — you will be asked to calculate metrics like the standard deviation, experiment duration, and sample size.

After answering all the questions about the dataset, you’ve got to make a final recommendation as to whether you would launch the experiment given the analysis conducted.
 

Takeaways

I went into Udacity’s A/B testing course expecting it to be math-heavy and hard to grasp.

To my surprise (and delight), it was more business-centric and focused on the practical implementation of A/B tests.

If you want to start performing A/B tests and would like to understand how to define a hypothesis, choose a sample size, and other experiment parameters, this course will help you get up and running quickly.

I also recommend the course for anyone who wants to deepen their understanding of statistical inference and experiment design, as knowledge of these concepts will help you ace data science and analytics interviews.

&nbsp
&nbsp

Natassha Selvaraj is a self-taught data scientist with a passion for writing. Natassha writes on everything data science-related, a true master of all data topics. You can connect with her on LinkedIn or check out her YouTube channel.