🚨 End of Year Deal 🚨 Purchase a course through an individual or team subscription and get $200 towards your next course | Expiring Jan 1 2023
📣 This is Ronny Kohavi's last class on Sphere. Make sure to join while you have the chance!

Designed to help make organizations

Data Driven

Product Leaders

Program managers and product managers that are focused on metrics like growth and revenue, and prioritization decisions

Data Scientists

Data Scientists and Data Science managers who help map strategic decisions to actionable experimental designs and then interpret the results in a trustworthy manner

The motivation and basics of A/B testing (e.g., causality, surprising examples, metrics, interpreting results, trust and pitfalls, Twyman’s law, A/A tests)
Cultural challenges, humbling results (e.g., failing often, pivoting, iterating), experimentation platform, institutional memory and meta-analysis, ethics
Hierarchy of evidence, Expected Value of Information (EVI), complementary techniques, risks in observational causal studies

Engineering Leaders

Engineering managers, directors, VPs, and CTOs who want to make their organizations data-driven with metrics and A/B tests

Safe deployments
Triggering, especially in evaluating machine learning models
The benefits of agile product development

Designed For Data Scientists and Machine Learning Engineers

who want to use causal inference to help guide business decisions

You will:
  • Recognize when ignoring causality can lead to poor decision-making inspired by real-world examples experienced by companies like Instacart and Meta
  • Use a variety of causal inference tools common in the industry (including DoWhy and EconML)
  • Learn practical advice for how to check the robustness of causal analyses and how to explain the results to business stakeholders
You should:
  • Be proficient with Python, be familiar with R
  • Know how to apply college-level probability and statistics, such as expected values, confidence intervals and linear regression
  • Possess basic familiarity with supervised machine learning

About Rob

Live Cohort

If we want to use data to guide our decisions, we need to predict how our decisions will cause the world to change. Causal inference, the study of cause-and-effect, can be used to guide decision-making by predicting the outcomes our decisions will cause. A/B testing is a reliable and widely used approach for estimating causal effects and guiding decisions. However, for some important business decisions, we are unable to run randomized experiments for practical, ethical, or legal reasons.

This course will focus on causal inference approaches that can be done on observational data — meaning it does not come from a randomized experiment. You will learn to recognize situations where causal inference can be applied and evaluate the tradeoffs between different approaches to estimation.

Additionally, we’ll cover traditional statistical approaches along with recently developed techniques that use machine learning for causal inference. Along the way, you will also learn to recognize situations where careless use of machine learning models can confuse correlation with causation and generate bad decisions. Several case studies based on real problems faced by large tech companies will guide our conversations, and learners will practice implementing causal inference techniques on simulated data.

In traditional software development, CI/CD automates many tasks, including testing, building and deploying software. But CI/CD for ML is a different beast. Testing and deployment of ML can be triggered by many event types, and observability and logging requirements are materially different for ML.Today, no single tool can facilitate end-to-end CI/CD for ML. The process of testing, building and deploying ML requires a symphony of tools and glue code to create an integrated CI/CD system. To offer an entry point that many data scientists and engineers are familiar with, we’ll teach you how to integrate GitHub with other ML tools to build custom CI/CD automations for ML that will increase your engineering efficiency and prevent errors from being released to production.

Session 1: Solve Business Problems With Causal Inference

Monday, February 27, 2023
1-3 pm PST

Every causal inference initiative starts with identifying and describing suitable business problems. During the session, we will: 

  • Define causal inference and distinguish it from machine learning predictions
  • Learn how to use causal inference to drive business impact 
  • Recognize the key challenges to doing causal inference with observational data by exploring a case study on the impact of Amazon Prime on customer spending
Live with

Session 2: Classical Causal Inference: Regression, Matching and Propensity Scores

Thursday, March 2, 2023
1-3 pm PST

If we want to understand how Amazon Prime affects a user’s spending, we need to adjust for any factors that cause members to behave differently than non-members. In this session, we will: 

  • Use matching-based estimators to compare similar users who received different “treatments” (e.g., one Amazon Prime member compared to a non-member with similar characteristics)
  • Estimate propensity scores and use them for causal inference
  • Combine propensity scores and outcome models to make “doubly robust” estimates
Live with

Session 3: How Effective Were My Facebook Ads? A Sobering Tale of the Limitations of Causal Inference

Monday, March 6, 2023
1-3 pm PST

As we select between many relevant control variables, we may also unknowingly introduce bias. Double/debiased ML is a powerful solution to avoid/ reduce this bias. In this session, we will: 

  • Learn how to combine machine learning models to produce causal estimates using an approach known as double/debiased machine learning
  • Understand the limitations of causal inference approaches by evaluating a situation where causal inference approaches failed (even with Facebook-scale data)
Live with

Session 4: Geo Experiments and Causal Inference With Aggregate Data

Thursday, March 9, 2023
1-3 pm PST

For many important business decisions, it is not feasible to change policies at the level of individual users. Instead the change must be made at a larger level of aggregation such as a whole city, or state, or country. In this session, we will:

  • Understand the challenges of doing causal inference when there are spillover effects such as when studying social networks like Facebook or multi-sided marketplaces like Uber
  • Use synthetic controls and difference-in-differences to estimate the causal effect of a change to the algorithm that matches riders and drivers
Live with

Session 5: Heterogeneous Treatment Effects: Going Beyond Averages

Monday, March 13, 2023
1-3 pm PST

By estimating “heterogeneous treatment effects,” we can help create personalized policies that vary treatment decisions based on whom they will work best for. In this session, we will:

  • Contrast approaches for estimating user-specific causal effects  with estimating the population average effect by examining a case study about targeting advertising to drive incremental sales
  • Apply ML-based techniques for HTE: s-, t- and x- learner
  • Distinguish causal effect estimation from causal decision making
Live with

Session 6: Cutting-Edge Causal Inference

Thursday, March 16, 2023
1-3 pm PST

The last session is reserved for cutting-edge topics that will zoom in on recent developments in causal inference. In this session, we will:

  • Learn how we can evaluate the robustness of causal inference estimates by quantifying their sensitivity to unobserved confounders
  • Understand how to apply causal inference approaches for continuous variables like price
  • Examine the tradeoffs between exploration and optimization
Live with
Interested in sending your

Team?

Sphere offers a range of subscription packages that provide discounts on all courses in our library. We help upskill employees at some of the world’s best companies. Learn more about pricing options here or book a time to talk to one of our staff below.

Book a free consultationCheck Expense Approval At Your Company

Learn live from a world-class

Instructor

Learn live from a world-class

Instructor

Rob Donnelly worked at Microsoft Research on statistical models of advertiser behavior that were used to predict how advertisers would respond to changes to the ad auction format. Then he studied economics at Stanford Graduate School of Business while doing research that combined machine learning and causal inference to better understand customer behavior. Susan Athey, one of the leading researchers in the area of machine learning and causal inference, was Rob’s advisor.

After earning his PhD, he worked on economic models of competition and network growth as a research scientist at Facebook and on advertising, pricing, and marketplace growth as a senior economist at Instacart. Now he is a machine learning scientist at a small startup called Arena-ai, which is focused on reinforcement learning for automating high-frequency business decisions.

Learn live from world-class

Instructors

Rob Donnelly worked at Microsoft Research on statistical models of advertiser behavior that were used to predict how advertisers would respond to changes to the ad auction format. Then he studied economics at Stanford Graduate School of Business while doing research that combined machine learning and causal inference to better understand customer behavior. Susan Athey, one of the leading researchers in the area of machine learning and causal inference, was Rob’s advisor.

After earning his PhD, he worked on economic models of competition and network growth as a research scientist at Facebook and on advertising, pricing, and marketplace growth as a senior economist at Instacart. Now he is a machine learning scientist at a small startup called Arena-ai, which is focused on reinforcement learning for automating high-frequency business decisions.

Guest Lectures by

Industry Experts

Sean Taylor
Data Science Manager, ex-Lyft

Until recently, causal inference has been challenging to learn due to a  lack of practical examples, rapidly evolving techniques, and few experienced practitioners. Rob Donnelly’s combination of expertise in the latest methods with exposure to real applications at some of the most innovative companies makes him an ideal teacher.

Susan Athey
Professor of Economics @ Stanford Business School

Rob Donnelly brings a great mix of insights to his work.  He has coauthored with top ML scholars and brought together different approaches into applications. Recently, he has been at the frontier of these applications in industry, building insight into what works well to answer questions and change products.

Tilman Dreup
Director of Machine Learning Engineering @ Instacart

Rob is a uniquely talented communicator with the ability to break down and present even the most complex technical topics. I cannot recommend him enough to anybody intent on learning about and applying state-of-the-art methods.

Join a diverse and experienced

Community

This cohort gives you access to a rich community of like-minded professionals from some of the best businesses in the world. Even after the course ends, you will continue to learn and build with each other.

Exclusive Content

to advance your business

Get access to exclusive content through live sessions, meetups and our Student Portal (even after you finish the cohort). Learn to recognize situations where causal inference can be applied and to evaluate the tradeoffs between different approaches to estimation.

Still have questions?

We’re here to help!

Do I have to attend all of the sessions live in real-time?

You don’t! We record every live session in the cohort and make each recording and the session slides available on our portal for you to access anytime.

Will I receive a certificate upon completion?

Each learner receives a certificate of completion, which is sent to you upon completion of the cohort (along with access to our Alumni portal!). Additionally, Sphere is listed as a school on LinkedIn so you can display your certificate in the Education section of your profile.

Is there homework?

Throughout the cohort, there may be take-home questions that pertain to subsequent sessions. These are optional, but allow you to engage more with the instructor and other cohort members!

Can I get the course fee reimbursed by my company?

While we cannot guarantee that your company will cover the cost of the cohort, we are accredited by the Continuing Professional Development (CPD) Standards Office, meaning many of our learners are able to expense the course via their company or team’s L&D budget. We even provide an email template you can use to request approval.

I have more questions, how can I get in touch?

Please reach out to us via our Contact Form with any questions. We’re here to help!

Book a time to talk with the Sphere team

Join us for a next generation

Learning Experience