Alex Deng
I’m a technical director of platform data science and machine learning at Airbnb, chairing the science track in the company-wide AI and ML council. I worked on a wide range of initiatives such as modernizing Airbnb’s applied research development, collaboration and prototyping ecosystem, building data foundation and consumption interface for event sequences analytics and user behavior modeling, and agile experimentation paradigm for offline and online evaluation of ML models. I also serve in the leadership teams for both the relevance and personalization team and the core data science team.
Prior to Airbnb I worked on the Microsoft Analysis and Experimentation Team and Azure Machine Learning, both part of the Cloud and AI platform. The Analysis and Experimentation team is responsible for one of the largest and best cutting-edge online experimentation system in the industry. I’m led a team focusing on methodological improvements of the experimentation platform as well as tackling the related engineering challenges. My team’s and my works span the area of causal inference and online experimentation, Bayesian optimization, reinforcement learning, auto Machine Learning, recommender systems, high throughput distributed computing, compiler and data analysis pipelines. In Azure Machine Learning, my team represented 1st party data and ML scientist experience of the product, and worked on large scale distributed training, auto machine learning, ML pipeline and product analytics.
I finished my Ph.D. study in statistics at Stanford in 2010 focusing on sequential Monte Carlo with Prof. Tze Lai.
Selected Publications (newest first)
- Metric Decomposition in A/B Tests.
Alex Deng, Luke Hagar, Nathaniel Stevens, Tatiana Xifara, Amit Gandhi. KDD 2024.
- The Price is Right: Removing A/B Test Bias in a Marketplace of Expirable Goods.
Thu Le, Alex Deng. CIKM 2023.
- Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed Outcomes.
Alex Deng, Michelle Du, Anna Matlin and Qing Zhang. KDD 2023.
- Zero to Hero: Exploiting null effects to achieve variance reduction in experiments with one-sided triggering
Alex Deng, Lo-Hua Yuan, Naoya Kanai and Alexandre Salama-Manteau. WSDM 2023.
- A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments
Ron Kohavi, Alex Deng, Lukas Vermeer. KDD 2022.
- On Post-selection Inference in A/B Testing
Alex Deng, Yicheng Li, Jiannan Lu and Vivek Ramamurthy. KDD 2021.
- Empirical Bayes Estimation of Treatment Effects with Many A/B Tests: An Overview
Eduardo Azevedo, Alex Deng, Jose Montiel and Glen Weyl. AEA Papers and Proceedings 2019
- Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas
Alex Deng, Ulf Knoblich, Jiannan Lu. KDD 2018.
- The A/B Testing Problem
Eduardo M Azevedo, Alex Deng, Jose Luis Montiel Olea, Justin Rao, E Glen Weyl. EC 2018
- A note on type S/M errors in hypothesis testing(psyarxiv)
Jiannan Lu, Yixuan Qiu, Alex Deng. British Journal of Mathematical and Statistical Psychology 2018.
- On randomization-based causal inference for matched-pair
factorial designs
Jiannan Lu, Alex Deng. Statistics and Probability Letters, 2017.
- Trustworthy analysis of online A/B tests: Pitfalls, challenges and solutions
Alex Deng, Jiannan Lu, Jonathan Litz. WSDM 2017.
- Continuous monitoring of A/B tests without pain: Optional stopping in Bayesian testing (ArXiv ver.)
Alex Deng, Jiannan Lu, Shouyuan Chen. DSAA 2016.
- Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned
Alex Deng, Xiaolin Shi. KDD 2016.
- Demystifying the Bias from Selective Inference: a Revisit to Dawid’s Treatment Selection Problem
Jiannan Lu, Alex Deng. Statistics and Probability Letters, 2016.
- Objective Bayesian Two Sample Hypothesis Testing for Online Controlled Experiments (Slides)
Alex Deng. WWW 2015, The 1st Workshop on Offline and Online Evaluation of Web-based Services
- Diluted Treatment Effect Estimation for Trigger Analysis in Online Controlled Experiments (Slides)
Alex Deng and Victor Hu. WSDM 2015.
- Seven Rules of Thumb for Web Site Experimenters
Ron Kohavi, Alex Deng, Roger Longbotham, and Ya Xu. KDD 2014.
- Statistical Inference in Two-stage Online Controlled Experiments with Treatment Selection and Validation
Alex Deng, Tianxi Li and Yu Guo. WWW 2014.
- Online Controlled Experiments at Large Scale
Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, Nils Pohlmann. KDD 2013.
- Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data
Alex Deng, Ya Xu, Ron Kohavi and Toby Walker. WSDM 2013.
- Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained
Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu. KDD 2012.
- Sequential importance sampling and resampling for dynamic portfolio credit risk
Shaojie Deng, Kay Giesecke and Tze Lai. Operations Research Feb, 2012.
- Rare-event simulation of heavy-tailed random walks by sequential importance sampling and resampling
Hock Peng Chan, Shaojie Deng and Tze-Leung Lai. Advances in Applied Probability 2012.
Preprints
- From Augmentation to Decomposition: A New Look at CUPED in 2023
Alex Deng, Lke Hagar, Nathaniel Stevens, Tatiana Xifara, Lo-Hua Yuan, Amit Gandhi. Extended Abstract. CODE@MIT 2023 poster
- Concise Summarization of Heterogeneous Treatment Effect Using Total Variation Regularized Regression
Alex Deng, Pengchuan Zhang, Shouyuan Chen, Dong Woo Kim and Jiannan Lu. ArXiv
- Flexible Online Repeated Measures Experiment
Yu Guo and Alex Deng. ArXiv
Other Talks/Slides
Other (non-peer-reviewed)