🚨 Black Friday Sale 🚨 Purchase a course through an individual or team subscription and get $400 | Redeem by registering for a course | Expires Monday 5pm PT

Designed to help make organizations

Data Driven

Product Leaders

Program managers and product managers that are focused on metrics like growth and revenue, and prioritization decisions

Data Scientists

Data Scientists and Data Science managers who help map strategic decisions to actionable experimental designs and then interpret the results in a trustworthy manner

The motivation and basics of A/B testing (e.g., causality, surprising examples, metrics, interpreting results, trust and pitfalls, Twyman’s law, A/A tests)
Cultural challenges, humbling results (e.g., failing often, pivoting, iterating), experimentation platform, institutional memory and meta-analysis, ethics
Hierarchy of evidence, Expected Value of Information (EVI), complementary techniques, risks in observational causal studies

Engineering Leaders

Engineering managers, directors, VPs, and CTOs who want to make their organizations data-driven with metrics and A/B tests

Safe deployments
Triggering, especially in evaluating machine learning models
The benefits of agile product development

Designed for information retrieval practitioners

hoping to improve NDCG and search-to-conversion through more relevant search results

You will:
  • Learn the inner workings of machine learning based search relevance ranking
  • Create reliable search relevance training data that confidently labels what search results are relevant based on user clicks and other signals
  • Explore new ranking features in production to overcome the echo chambers present in existing search interaction data
You should:
  • Have experience tuning relevance for a ranking system (search, recos, etc) using a metric like NDCG in the past
  • Have experience tweaking search engine relevance settings (Elasticsearch, Solr, etc) to achieve some business goal
  • Have trained a model in Python with sklearn or similar tool in the past (however simple)
  • You’re about to embark on or are already implementing a machine-learning implementation for search

Learn live from a world-class

Instructor

Learn live from a world-class

Instructor

Since 2012, Doug Turnbull has worked with companies such as Reddit, Shopify, Careerbuilder, Wikimedia Foundation, Shipt, and LexisNexis on improving search experiences. Doug coaches and develops search teams from startups to Fortune 500 companies, setting up organizational practices to build relevant search. Doug wrote “Relevant Search” (2016) and co-authored “AI Powered Search” (2022). He also co-created the Elasticsearch Learning to Rank, bringing machine learning to the most popular search engine, which revamped Wikipedia and Yelp, driving intelligent search behind dozens of companies' search experiences.

Currently, Doug is a principal engineer at Reddit, which he described on LinkedIn as "a career opportunity to rethink how search can have a positive influence on the lives of real people."

Learn live from world-class

Instructors

Since 2012, Doug Turnbull has worked with companies such as Reddit, Shopify, Careerbuilder, Wikimedia Foundation, Shipt, and LexisNexis on improving search experiences. Doug coaches and develops search teams from startups to Fortune 500 companies, setting up organizational practices to build relevant search. Doug wrote “Relevant Search” (2016) and co-authored “AI Powered Search” (2022). He also co-created the Elasticsearch Learning to Rank, bringing machine learning to the most popular search engine, which revamped Wikipedia and Yelp, driving intelligent search behind dozens of companies' search experiences.

Currently, Doug is a principal engineer at Reddit, which he described on LinkedIn as "a career opportunity to rethink how search can have a positive influence on the lives of real people."

Recommended by

Industry Experts

Audrey Lorberfeld
Relevance Engineer @ Reddit

”Taking a course with Doug is one of the best career moves you can make. He is a veritable powerhouse of Search Relevance knowledge, with deep expertise in click models, ranking algorithms, and information retrieval fundamentals. Not only does Doug bring a wealth of experience to his teaching, he also brings levity and creativity. Learning from him is a pleasure not to be missed.”

Trey Grainger
CTO @ Presearch

“I've attended many of Doug's talks, referenced his articles, been a student in his classes (on learning to rank and click models), and even written a book with him. He is able to distill complex search relevance topics into intuitive, fun, and practical examples that make learning from him an invaluable and unforgettable experience.”

Bertrand Rigaldies
Principal Search Engineer @ Shipt

“At OpenSource Connections (2017-2022), Doug was to me and the other consultants our mentor as well as go-to resident expert of all Search matters. He possesses this rare combination of intellectual brilliance, humility, and friendliness. He helps others with a keen ability to explain how things work. I very much look forward to being in the same room as Doug again.”

Expense
the cost

90% of our learners expense the full cost of our courses to their employer. This includes leading startups and enterprises alike.

Check Expense Approval At Your Company

Exclusive Content

to advance your business

Get access to exclusive content through live sessions, meetups and our Student Portal (even after you finish the cohort). Ask questions and get personal feedback directly from your instructors and others taking the course.

Join a diverse and experienced

Community

This cohort gives you access to a rich community of like-minded professionals from some of the best businesses in the world. Even after the course ends, you will continue to learn and build with each other.

About Doug's

Live Cohort

With limited patience — and, often, small screens — many users avoid navigation altogether. Instead, they jump to a search bar to find what they need. Yet, with limited screen real-estate, many teams know search optimization can go bad. You can show only a handful of results out of millions to click on.

How can you deliver the magic your users seek with a conversational, relevant search experience? How do you give users the results that will achieve their goals? This course helps you analyze search behaviors to target relevance treatments, solving them using modern data science and machine learning.

Learn how your business can meet modern users where and how they expect to use applications.

  • How do I analyze search behaviors to arrive at reliable search relevance training data?
  • How do I train a ranking model to optimize for relevance?
  • What search measures correspond to my business needs?
  • What does a good search experience look like?
  • How do I explore novel ranking features?

Session 1: The Search Relevance Problem

Tuesday, January 24, 2023
1-3 pm PST

Tune a relevance solution using traditional, non-ML methods. Upon completing Session 1, learners will be able to:

  • Differentiate relevant from irrelevant results
  • Develop measures for ranking quality across many searches
  • Experiment with manually tweaking a relevance solution

Session 2: Now We Train!

Thursday, January 26, 2023
1-3 pm PST

Train a simple ranking model using hand-labeled results from Session 1. At the end of this session, learners will be able to:

  • Build models that categorize relevant from irrelevant results
  • Test ranking models with a test-train split

Session 3: Better Training Data

Tuesday, January 31, 2023
1-3 pm PST

Overcome core biases in search behavioral data to build a successful ranking model. After this session, learners will:

  • Overcome core biases in search behavioral data to build a successful ranking model
  • Be able to contrast approaches to overcoming biases

Session 4: Feature Discovery

Thursday, February 2, 2023
1-3 pm PST

Navigate echo-chambers by experimenting with novel ranking features. Learners who complete this session will be able to:

  • Understand how and why echo chambers occur in search
  • Obliterate echo chambers underlying search experiences to make positive step-changes in search relevance
  • Evaluate new query, user, or document ranking features that go beyond your current search solution using Active Learning

Session 5: Traction

Tuesday, February 7, 2023
1-3 pm PST

The playbook on how to get unstuck from the search relevance problem. Upon completion of this session, learners will be able to:

  • Manage rollout of search relevance experiments in search
  • Balance near-term search wins, with longer-term investments needed to take search to the next level
  • Convince team and stakeholders that your ML-Powered search team is moving in the right direction

Still have questions?

We’re here to help!

Do I have to attend all of the sessions live in real-time?

You don’t! We record every live session in the cohort and make each recording and the session slides available on our portal for you to access anytime.

Will I receive a certificate upon completion?

Each learner receives a certificate of completion, which is sent to you upon completion of the cohort (along with access to our Alumni portal!). Additionally, Sphere is listed as a school on LinkedIn so you can display your certificate in the Education section of your profile.

Is there homework?

Throughout the cohort, there may be take-home questions that pertain to subsequent sessions. These are optional, but allow you to engage more with the instructor and other cohort members!

Can I get the course fee reimbursed by my company?

While we cannot guarantee that your company will cover the cost of the cohort, we are accredited by the Continuing Professional Development (CPD) Standards Office, meaning many of our learners are able to expense the course via their company or team’s L&D budget. We even provide an email template you can use to request approval.

I have more questions, how can I get in touch?

Please reach out to us via our Contact Form with any questions. We’re here to help!

Book a time to talk with the Sphere team

Join us for a next generation

Learning Experience