Experiments at Airbnb

Jan Overgoor and Will Moss @airbnb

Wed, April 02, 2014

Experiments at Airbnb

This is the first talk in the upcoming series on Search and Matching at Airbnb, where we’ll be introducing our experimentation infrastructure for running A/B tests with the context of conversion optimization and learning about user behavior on the site and our mobile apps. We will also be covering the unique challenges of running A/B tests at Airbnb and discussing best practices for setting up proper experiments.

Controlled experiments are the way to go if you want to learn something about the world. They are also very useful in informing product development and design decisions. However, the complexity of our ecosystem has led us to consider a range of technical and conceptual issues that go further than the vanilla A/B-testing paradigm.

On the analytical side, we’ll talk about the importance of choosing the right unit of analysis and segmenting users into meaningful cohorts. Employing a good stopping criterion is also essential. Furthermore, we’ll discuss why understanding potential biases is crucial for running experiments effectively and soundly.

Translating the sound mathematical underpinnings of good experimentation into practice is crucially important. We’ll discuss a framework we’ve built for running experiments consisting of three main components. A mechanism for defining experiments and deploying them in both Ruby and Javascript. A method for defining reusable analysis metrics (i.e. bookings, revenue, etc.) and running analysis for each experiment given those metrics. Finally, a simple UI for visualizing the results of the experiments.

Will Moss is an engineer on the Data Infrastructure team at Airbnb, where he has been for the past 8 months. Before that, he was an engineer at Bump Technologies where he worked on all parts of the server stack. When not computering, Will likes to spend his time outdoors, hiking, biking, surfing and playing ultimate frisbee.

Jan Overgoor is a Data Scientist at Airbnb. He works on the search algorithm and the experiment framework used throughout the company. Previously he completed an Msc. of Symbolic Systems at Stanford University and wrote a thesis on trust in online communities.

Dinner, prepared by Airbnb's Kitchen Team, will be served at 6pm.
The talk begins at 7pm.

Check out our upcoming Tech Talks!

Follow Airbnb Engineering on Twitter! @AirbnbNerds

By signing up for the Event, you agree that Airbnb may record or photograph the Event, (including You, your likeness and voice), and use it publicly or privately in any form of media, including for commercial purposes. By attending, You consent to the recording and photographing and grant Airbnb an unrestricted, perpetual, non-exclusive and sublicenseable license, to take, use, display, perform, distribute or exploit for any lawful and commercial purpose any recordings or photographs. If You do not agree, please do not click "Sign Me Up" and do not attend the meetup.

Rats! There aren't any spots left in this talk.


Where?
Airbnb HQ
888 Brannan St
San Francisco,
94107, California

When?
Wed, April 02, 2014
(meetup over)

6:00 PM — 8:00 PM


Who's going?
Jason Robby Ju Jina Dave Milton Charlie Meghan Bryan David Jennifer Joy Becky Kat Vikram Kaila Luke & Emily Arthur Jeff Cenci Florian Erika Christine Louise Tamara Ho-Hsiang Jason Magee Hugues Peter Ingo Clara Joseph Ryan Ian Fang Bora Sathvik Brant Jenna Marianne Giacomo Kimberly Randy Kan Tess Lauren Amy Michael Rhea Anas Nicole Jonathan Tiffany May Giselle Steve Wendy Becky David Tom Livien Maya Peter Jeff Ainsley Annamarie Gurinder Jeremia Drew Kavi Nathan Vanida Song Matteo Emily Yueyang Jerre Davide Nicola Joody Andi