1

How does Facebook do A/B testing?

Profile picture
Engineering Manager at Sinch2 years ago

What kind of tools that Facebook uses for A/B testing?

Are they doing trunk-based development?

59
2

Discussion

(2 comments)
  • 1
    Profile picture
    Tech Lead/Manager at Meta, Pinterest, Kosei
    2 years ago

    There are generally 2 ways to do A/B testing at Facebook (note: Meta is so large now that I'm sure various orgs may have built specialized tools at this point):

    • Gatekeeper (GK) framework: If you have a simple binary experiment ("check the impact of this button being red or blue") then you can use a gatekeeper. You set the condition of which user gets which treatment, and then the framework will compute various stat sig metric differences
    • Quick Experiment (QE) framework: This is for more complicated, multi-bucket experiments. You still use GKs to decide who gets into the experiment, but this system lets you compare an arbitrary number of test cases and a control.

    There's a bit more nuance around things like experiment setup, universe setup, and how long your experiment needs to run.

    What do you mean by "trunk-based'? Almost everything at Meta is going to be based off mainline. There are very few long-lasting feature branches.

  • 1
    Profile picture
    Tech Lead @ Robinhood, Meta, Course Hero
    2 years ago

    Meta-question: Why are you curious as to how Meta specifically does A/B testing? Is it because engineering leadership at Sinch wants to emulate Meta specifically?

    I'm not sure if trying to copy what Meta will make sense as it's operating at a much larger scale than anything you're working with, which leads to far more complexity in the tooling. I recommend working backwards from your needs, building the simple thing first, and iterating from there.

    If you want to learn more about experimentation at Meta though, I left a lot of thoughts in this discussion from a Meta engineer.