🚨 End of Year Deal 🚨 Purchase a course through an individual or team subscription and get $200 towards your next course | Expiring Jan 1 2023

Designed to help make organizations

Data Driven

Product Leaders

Program managers and product managers that are focused on metrics like growth and revenue, and prioritization decisions

Data Scientists

Data Scientists and Data Science managers who help map strategic decisions to actionable experimental designs and then interpret the results in a trustworthy manner

The motivation and basics of A/B testing (e.g., causality, surprising examples, metrics, interpreting results, trust and pitfalls, Twyman’s law, A/A tests)
Cultural challenges, humbling results (e.g., failing often, pivoting, iterating), experimentation platform, institutional memory and meta-analysis, ethics
Hierarchy of evidence, Expected Value of Information (EVI), complementary techniques, risks in observational causal studies

Engineering Leaders

Engineering managers, directors, VPs, and CTOs who want to make their organizations data-driven with metrics and A/B tests

Safe deployments
Triggering, especially in evaluating machine learning models
The benefits of agile product development

Designed for Data Scientists, Data Engineers and Software Engineers

who want to deploy high velocity and reliable randomized experiments

You will:
  • Reduce bias in A/B testing and increase confidence in your results 
  • Navigate organizational constraints to optimize for experiment velocity and reliability
  • Setup technical infrastructure to enable reliable and scalable experiment analysis
You should:
  • Have minimum 1 year of experience at a data-driven company working with product data or doing data science 
  • Have knowledge of basic statistics (rates, averages, prior exposure to a/b and hypothesis testing)
  • Be proficient with SQL and a statistics package such as Python or R

About Kyle's

Live Cohort

Randomized experimentation is a core practice at data-focused organizations. But reliable experimentation is not trivial! To achieve that, a rigorous statistical approach and careful engineering must work hand-in-hand. 

In this course, you will learn the nuts and bolts of how to design, implement, and analyze experiments, with a focus on reliability and velocity. We will examine randomization implementation, logging, sample ratio mismatches, and data set construction. On the analysis side, we will focus on metric choice and construction along with the most common statistical techniques. Topics will include types of hypothesis tests, p-values, confidence intervals, multiple hypothesis testing, and p-hacking.

This course is less about the "why" and more about the "how" of experimentation. A key theme will be navigating experimentation choices and tradeoffs while facing the constraints of your organization, technical resources, and data availability. We will look at examples of real experiments and use simulated data and code to get "hands on" with problems and solutions. At the end of the course, you will be equipped to run reliable, randomized experiments end-to-end - from design, to implementation, to statistical analysis, and ultimately to a confident decision.

Session 1: Setup Infrastructure for Your Experimentation Platform

Monday, April 3, 2023
4-6 pm PST

Assess the key operating principles you need for a successful experimentation platform at your organization. In this session, we will: 

  • Showcase experiment workflow, engineering components, and data architecture
  • Discuss how to make effective tradeoffs in the context of your organization and technical support
  • Use case studies at Facebook and Netflix to discover how their infrastructure impacted their business
Live with

Session 2: Detect, Fix and Prevent Sample Ratio Mismatches (SRMs)

Thursday, April 6, 2023
4-6 pm PST

The culprit of a variety of data quality issues is a mismatch between expected sample ratio and observed sample ratio (SRMs). In this session, we will: 

  • Demonstrate different types of SRMs and how to detect and fix them
  • Review examples of randomization gone wrong at Skype, Microsoft, and Shopify
  • Learn frameworks to prevent SRMs and avoid wasted experiment time
  • Compare the complications of experiments run on servers vs browser/mobile apps
Live with

Session 3: Create Reliable A/B Testing Metrics

Tuesday, April 11, 2023
4-6 pm PST

Metrics play a key role in aligning experiments with strategic goals. In this session, we will: 

  • Discuss and apply the principles for creating reliable a/b testing metrics
  • Demonstrate how to calculate metrics from “raw” data and SQL queries/Python code to create a table for analysis
Live with

Session 4: Analyze and Report the Experiment Data

Thursday, April 13, 2023
4-6 pm PST

As you analyze your experiment data, it’s key to clearly separate the signal from the noise. In this session, we will: 

  • Apply three tests to check the reliability of data, and calculate uplift, p-values, and confidence intervals
  • Learn from case studies at Netflix, Google, and Airbnb about how to avoid incorrect conclusions due to peeking and multiple comparisons
Live with

Session 5: Make Small Sample Sizes Count

Monday, April 17, 2023
4-6 pm PST

Not everyone has the traffic of today’s most successful experimentation platforms. In this session, we will: 

  • Discuss how to make confident decisions, despite smaller sample sizes
  • Review why and how to do a power analysis 
  • Apply statistical techniques to accelerate experiment results
Live with

Session 6: Use Advanced Testing Techniques

Thursday, April 20, 2023
4-6 pm PST

Take your skills to the next level and accelerate your program of experimentation. In this session, we will: 

  • Discuss how to run  multiple experiments at the same time 
  • Apply more advanced statistical techniques to increase iteration speed
  • Learn from case studies at Spotify and Stitchfix about how to experiment faster, running multiple experiments simultaneously using experiment coordination and holdbacks, and how to combine multiple treatments into one experiment using multi-arm bandits
Live with
Interested in sending your

Team?

Sphere offers a range of subscription packages that provide discounts on all courses in our library. We help upskill employees at some of the world’s best companies. Learn more about pricing options here or book a time to talk to one of our staff below.

Book a free consultationCheck Expense Approval At Your Company

Learn live from a world-class

Instructor

Learn live from a world-class

Instructor

Kyle is a Staff Data Scientist at Stripe, leading the development of their in-house experiment platform over the last 4 years. Previously, he taught an experimentation and causal inference course at the University of San Francisco's Department of Economics. His previous work experience includes data science at Twitter, economic experiments at the University of Lausanne, and economics research at the Federal Reserve Bank of Boston. Kyle has a PhD in economics from the California Institute of Technology and BA in economics from the University of Memphis.

Learn live from world-class

Instructors

Kyle is a Staff Data Scientist at Stripe, leading the development of their in-house experiment platform over the last 4 years. Previously, he taught an experimentation and causal inference course at the University of San Francisco's Department of Economics. His previous work experience includes data science at Twitter, economic experiments at the University of Lausanne, and economics research at the Federal Reserve Bank of Boston. Kyle has a PhD in economics from the California Institute of Technology and BA in economics from the University of Memphis.

Recommended by

Industry Experts

No items found.

Join a diverse and experienced

Community

This cohort gives you access to a rich community of like-minded professionals from some of the best businesses in the world. Even after the course ends, you will continue to learn and build with each other.

Exclusive Content

to advance your business

Get access to exclusive content through live sessions, meetups and our Student Portal (even after you finish the cohort). Ask questions and get personal feedback directly from your instructors and others taking the course.

Still have questions?

We’re here to help!

Do I have to attend all of the sessions live in real-time?

You don’t! We record every live session in the cohort and make each recording and the session slides available on our portal for you to access anytime.

Will I receive a certificate upon completion?

Each learner receives a certificate of completion, which is sent to you upon completion of the cohort (along with access to our Alumni portal!). Additionally, Sphere is listed as a school on LinkedIn so you can display your certificate in the Education section of your profile.

Is there homework?

Throughout the cohort, there may be take-home questions that pertain to subsequent sessions. These are optional, but allow you to engage more with the instructor and other cohort members!

Can I get the course fee reimbursed by my company?

While we cannot guarantee that your company will cover the cost of the cohort, we are accredited by the Continuing Professional Development (CPD) Standards Office, meaning many of our learners are able to expense the course via their company or team’s L&D budget. We even provide an email template you can use to request approval.

I have more questions, how can I get in touch?

Please reach out to us via our Contact Form with any questions. We’re here to help!

Book a time to talk with the Sphere team

Join us for a next generation

Learning Experience