Implementing Reliable A/B Tests
Dive deep into the implementation methods, calculations, setup, and analysis of A/B tests with a focus on engineering and statistical practice.
Access to: session recordings, curated resources and exclusive events
Designed to help make organizations
Data Driven
Product Leaders
Program managers and product managers that are focused on metrics like growth and revenue, and prioritization decisions
Data Scientists
Data Scientists and Data Science managers who help map strategic decisions to actionable experimental designs and then interpret the results in a trustworthy manner
Engineering Leaders
Engineering managers, directors, VPs, and CTOs who want to make their organizations data-driven with metrics and A/B tests
Designed for Data Scientists, Data Engineers and Software Engineers
who want to deploy high velocity and reliable randomized experiments
- Reduce bias in A/B testing and increase confidence in your resultsÂ
- Navigate organizational constraints to optimize for experiment velocity and reliability
- Setup technical infrastructure to enable reliable and scalable experiment analysis
- Have minimum 1 year of experience at a data-driven company working with product data or doing data scienceÂ
- Have knowledge of basic statistics (rates, averages, prior exposure to a/b and hypothesis testing)
- Be proficient with SQL and a statistics package such as Python or R
Live Cohort
Assess the key operating principles you need for a successful experimentation platform at your organization. In this session, we will:Â
- Showcase experiment workflow, engineering components, and data architecture
- Discuss how to make effective tradeoffs in the context of your organization and technical support
- Use case studies at Facebook and Netflix to discover how their infrastructure impacted their business
The culprit of a variety of data quality issues is a mismatch between expected sample ratio and observed sample ratio (SRMs). In this session, we will:Â
- Demonstrate different types of SRMs and how to detect and fix them
- Review examples of randomization gone wrong at Skype, Microsoft, and Shopify
- Learn frameworks to prevent SRMs and avoid wasted experiment time
- Compare the complications of experiments run on servers vs browser/mobile apps
Metrics play a key role in aligning experiments with strategic goals. In this session, we will:Â
- Discuss and apply the principles for creating reliable a/b testing metrics
- Demonstrate how to calculate metrics from “raw” data and SQL queries/Python code to create a table for analysis
As you analyze your experiment data, it’s key to clearly separate the signal from the noise. In this session, we will:Â
- Apply three tests to check the reliability of data, and calculate uplift, p-values, and confidence intervals
- Learn from case studies at Netflix, Google, and Airbnb about how to avoid incorrect conclusions due to peeking and multiple comparisons
Not everyone has the traffic of today’s most successful experimentation platforms. In this session, we will:Â
- Discuss how to make confident decisions, despite smaller sample sizes
- Review why and how to do a power analysisÂ
- Apply statistical techniques to accelerate experiment results
Take your skills to the next level and accelerate your program of experimentation. In this session, we will:Â
- Discuss how to run multiple experiments at the same timeÂ
- Apply more advanced statistical techniques to increase iteration speed
- Learn from case studies at Spotify and Stitchfix about how to experiment faster, running multiple experiments simultaneously using experiment coordination and holdbacks, and how to combine multiple treatments into one experiment using multi-arm bandits
Team?
Sphere offers a range of subscription packages that provide discounts on all courses in our library. We help upskill employees at some of the world’s best companies. Learn more about pricing options here or book a time to talk to one of our staff below.
Book a free consultation
Learn live from a world-class
Instructor
Learn live from a world-class
Instructor
Kyle is a Staff Data Scientist at Stripe, leading the development of their in-house experiment platform over the last 4 years. Previously, he taught an experimentation and causal inference course at the University of San Francisco's Department of Economics. His previous work experience includes data science at Twitter, economic experiments at the University of Lausanne, and economics research at the Federal Reserve Bank of Boston. Kyle has a PhD in economics from the California Institute of Technology and BA in economics from the University of Memphis.
Learn live from world-class
Instructors
Kyle is a Staff Data Scientist at Stripe, leading the development of their in-house experiment platform over the last 4 years. Previously, he taught an experimentation and causal inference course at the University of San Francisco's Department of Economics. His previous work experience includes data science at Twitter, economic experiments at the University of Lausanne, and economics research at the Federal Reserve Bank of Boston. Kyle has a PhD in economics from the California Institute of Technology and BA in economics from the University of Memphis.
Recommended by
Industry Experts
Join a diverse and experienced
Community
This cohort gives you access to a rich community of like-minded professionals from some of the best businesses in the world. Even after the course ends, you will continue to learn and build with each other.
Exclusive Content
to advance your business
Get access to exclusive content through live sessions, meetups and our Student Portal (even after you finish the cohort). Ask questions and get personal feedback directly from your instructors and others taking the course.
Still have questions?
We’re here to help!
Do I have to attend all of the sessions live in real-time?
You don’t! We record every live session in the cohort and make each recording and the session slides available on our portal for you to access anytime.
Will I receive a certificate upon completion?
Each learner receives a certificate of completion, which is sent to you upon completion of the cohort (along with access to our Alumni portal!). Additionally, Sphere is listed as a school on LinkedIn so you can display your certificate in the Education section of your profile.
Is there homework?
Throughout the cohort, there may be take-home questions that pertain to subsequent sessions. These are optional, but allow you to engage more with the instructor and other cohort members!
Can I get the course fee reimbursed by my company?
While we cannot guarantee that your company will cover the cost of the cohort, we are accredited by the Continuing Professional Development (CPD) Standards Office, meaning many of our learners are able to expense the course via their company or team’s L&D budget. We even provide an email template you can use to request approval.
I have more questions, how can I get in touch?
Please reach out to us via our Contact Form with any questions. We’re here to help!
.svg)
.svg)