Accelerating Innovation with A/B Testing
Learn how to design, measure and implement trustworthy A/B tests in this live course with world-leading experimentation expert Ronny Kohavi.
Access to: session recordings, curated resources and exclusive events
Designed to help make organizations
Data Driven
Product Leaders
Program managers and product managers that are focused on metrics like growth and revenue, and prioritization decisions
Data Scientists
Data Scientists and Data Science managers who help map strategic decisions to actionable experimental designs and then interpret the results in a trustworthy manner
Engineering Leaders
Engineering managers, directors, VPs, and CTOs who want to make their organizations data-driven with metrics and A/B tests
Designed for Product Leaders, Data Scientists and Engineering Leaders
who want to make their organizations data-driven with metrics and A/B tests
- The motivation and basics of A/B testing (e.g., causality, surprising examples, metrics, interpreting results, trust and pitfalls, Twyman’s law, A/A tests)
- Cultural challenges, humbling results (e.g., failing often, pivoting, iterating), experimentation platform, institutional memory and meta-analysis, ethics
- Hierarchy of evidence, Expected Value of Information (EVI), complementary techniques, risks in observational causal studies
Live Cohort
A/B tests, also called online controlled experiments, are used heavily at companies like Airbnb, Amazon, Booking.com, eBay, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yahoo!/ Oath, and Yandex. These companies run thousands to tens of thousands of experiments every year, sometimes involving millions of users and testing everything, including changes to the user interface (UI), relevance algorithms (search, ads, personalization, recommendations, and so on), latency/ performance, content management systems, customer support systems, and more. Experiments are run on multiple digital channels: websites, desktop applications, mobile applications, and e-mail.
The theory of controlled experiments dates back to Sir Ronald A. Fisher in the 1920s, but running experiments at scale is not simply a theoretical challenge, but mostly a cultural challenge. To paraphrase Helmuth von Moltke, ideas rarely survive contact with customers. Objective evaluations of ideas in A/B tests show that most ideas are weaker than expected. Organizations must change incentives and be great at failing fast and pivoting in the face of data. We will look at real examples of cultural changes.
Setting a strategy with integrity implies tying it to an Overall Evaluation Criterion (OEC). We will discuss good characteristics of metrics, and in particular focus on the OEC.
A/B tests are commonly associated with evaluating new ideas over a week or two, but when implemented with near-real-time metrics, they provide a safety net for safe deployments, able to detect and abort deployments. We will discuss feature flags and why controlled experiments are so good at helping avoid severe outages.
In traditional software development, CI/CD automates many tasks, including testing, building and deploying software. But CI/CD for ML is a different beast. Testing and deployment of ML can be triggered by many event types, and observability and logging requirements are materially different for ML.Today, no single tool can facilitate end-to-end CI/CD for ML. The process of testing, building and deploying ML requires a symphony of tools and glue code to create an integrated CI/CD system. To offer an entry point that many data scientists and engineers are familiar with, we’ll teach you how to integrate GitHub with other ML tools to build custom CI/CD automations for ML that will increase your engineering efficiency and prevent errors from being released to production.
- Introduction to controlled experiments, or A/B testing
- Interesting experiments – you’re the decision maker
- Organizational tenets
- Advantages and limitations of controlled experiments
- End-to-end example: Running and Analyzing an Experiment
- Metrics and the OEC (large section with break)
- P-values and statistical power (short)
- Interesting examples
- Experiment example: speed matters
- Twyman’s Law and Trustworthy Experimentation
- Cultural challenge, Institutional memory, maturity model, prioritization/EVI, ethics
- AI/Machine Learning model, and triggering
- Observational causal studies
- Pitfalls in observational studies
- Leakage and interference
- Build vs. Buy - Guest speakers
- Experimentation Platform
- Challenges
- Requested Topics
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
.png)
Team?
Sphere offers a range of subscription packages that provide discounts on all courses in our library. We help upskill employees at some of the world’s best companies. Learn more about pricing options here or book a time to talk to one of our staff below.
Book a free consultation
Learn live from a world-class
Instructor
Learn live from a world-class
Instructor
Few people have accumulated as much experience as Ronny Kohavi when it comes to experimentation. His work at tech giants such as Amazon, Microsoft and Airbnb — just to name a few — has laid the foundation of modern online experimentation. He is a pioneer and leader of the data mining and machine learning community and wants to share his learnings with you.
Over the last 20 years, Ronny has served in numerous roles at Airbnb, Microsoft, Amazon, Blue Martini Software, and Silicon Graphics.
Ronny’s papers have over 53,000 citations. He co-authored the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu) which is in the top-10 data mining books on Amazon. He is the most viewed writer on Quora’s A/B testing and he received the Individual Lifetime Achievement Award for Experimentation Culture in September 2020.
Ronny also holds a PhD in Machine Learning from Stanford University where he led the MLC++ project, the Machine Learning library in C++ used at Silicon Graphics and at Blue Martini Software
Learn live from world-class
Instructors
Few people have accumulated as much experience as Ronny Kohavi when it comes to experimentation. His work at tech giants such as Amazon, Microsoft and Airbnb — just to name a few — has laid the foundation of modern online experimentation. He is a pioneer and leader of the data mining and machine learning community and wants to share his learnings with you.
Over the last 20 years, Ronny has served in numerous roles at Airbnb, Microsoft, Amazon, Blue Martini Software, and Silicon Graphics.
Ronny’s papers have over 53,000 citations. He co-authored the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (with Diane Tang and Ya Xu) which is in the top-10 data mining books on Amazon. He is the most viewed writer on Quora’s A/B testing and he received the Individual Lifetime Achievement Award for Experimentation Culture in September 2020.
Ronny also holds a PhD in Machine Learning from Stanford University where he led the MLC++ project, the Machine Learning library in C++ used at Silicon Graphics and at Blue Martini Software
Guest Lectures by
Industry Experts
Join a diverse and experienced
Community
This cohort gives you access to a rich community of like-minded professionals from some of the best businesses in the world. Even after the course ends, you will continue to learn and build with each other.
Exclusive Content
to advance your business
Get access to exclusive content through live sessions, meetups and our Student Portal (even after you finish the cohort). Ask questions and get personal feedback directly from your instructors and others taking the course.
Still have questions?
We’re here to help!
Do I have to attend all of the sessions live in real-time?
You don’t! We record every live session in the cohort and make each recording and the session slides available on our portal for you to access anytime.
Will I receive a certificate upon completion?
Each learner receives a certificate of completion, which is sent to you upon completion of the cohort (along with access to our Alumni portal!). Additionally, Sphere is listed as a school on LinkedIn so you can display your certificate in the Education section of your profile.
Is there homework?
Throughout the cohort, there may be take-home questions that pertain to subsequent sessions. These are optional, but allow you to engage more with the instructor and other cohort members!
Can I get the course fee reimbursed by my company?
While we cannot guarantee that your company will cover the cost of the cohort, we are accredited by the Continuing Professional Development (CPD) Standards Office, meaning many of our learners are able to expense the course via their company or team’s L&D budget. We even provide an email template you can use to request approval.
I have more questions, how can I get in touch?
Please reach out to us via our Contact Form with any questions. We’re here to help!
Join us for a next generation
Learning Experience
Ronny made the content amazingly easy to consume, the other attendees were accomplished experimenters, and being able to ask questions both in real time and via the Sphere portal helped with better understanding of the material.
The entire course and the way it was delivered was amazing - had tons of learning, especially on where we could go wrong. Examples followed by key concepts was great. Really appreciate Ronny taking time to answer each question even after the session.
Want to take on the challenge of evolving the scientific culture in your organization with experimentation? I'd highly recommend considering Ronny Kohavi's Sphere cohort… lessons from Ronny's career driving experimentation culture and impact, and invaluable networking opportunities with experimentation leaders. Thankful to have joined the first cohort, met with inspiring colleagues, and to have the opportunity to share the learning with my team.
The culture of Q&A during the session. The depth and width of the experimentation topic. Really eye-opening learnings from Ronny. Thank you, thank you, thank you!
Ronny covered similar topics that are in his book, but in a way that internalizes the points much deeper.
I love Ronny's stories. I think that apart from the learning, two of the most valuable things from this class are: 1) Stories because you will remember them and they will come in handy and 2) Links so that you can dive deeper.
I love that the session goes beyond the tech & stats of experimentation and also address the cultural aspects.
The curation of relevant examples/stories provides the type of evidence often needed to make a case internally. So helpful to have it assembled in a clear strategy/direction.
50% of what I've learned myself about experimentation is wrong. The other half I learned from Ronny.
Ronny's tons of experience is valuable and hearing it firsthand is amazing.
Favorite part: Practical info and learnings that weren't in the book but were probably not easily publishable.
.svg)
.svg)