Alex Deng
I am heading the engineering and science for Responsible AI in Microsoft’s Core AI division. Our platform safeguards hundreds of billions of generative AI interactions annually, providing foundational safety systems for nearly all Azure OpenAI API traffic. This includes multimodal content moderation, jailbreak prevention, cross-domain prompt injection protection (XPIA), hallucination detection, and protected material identification. We are also at the forefront of safety evaluation, red teaming, model steering, and establishing the trust and security foundations for agentic AI systems.
From 2020 to July 2024, I served as Technical Director of Platform Data Science and Machine Learning at Airbnb, where I chaired the science track of the company-wide AI & ML Council. My work spanned modernizing applied research workflows, developing a collaborative ML prototyping ecosystem, building foundational data infrastructure for event stream analytics and behavioral modeling, and advancing agile experimentation frameworks for offline and online model evaluation. I also held leadership roles in both the Relevance & Personalization team and the Core Data Science organization.
Earlier in my career, I was part of Microsoft’s Cloud & AI Platform, contributing to both the Microsoft Analysis and Experimentation Team and Azure Machine Learning. At the former, I led methodological and engineering improvements to one of the industry’s largest online experimentation platforms, with work spanning causal inference, Bayesian optimization, reinforcement learning, AutoML, recommender systems, high-throughput distributed systems, and analytics infrastructure. In Azure ML, I focused on the end-to-end experience for internal ML practitioners, including distributed training, automated ML pipelines, and product analytics.
I earned my Ph.D. in Statistics from Stanford University in 2010, where I studied sequential Monte Carlo methods under Prof. Tze Lai
Selected Publications (newest first)
- Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking.
Qing Zhang, Alex Deng, Guqian Du, Huiji Gao, Liwei He and Sanjeev Katariya. KDD 2025.
- Transforming Location Retrieval at Airbnb: A Journey from Heuristics to Reinforcement Learning.
Dillon Davis, Huiji Gao, Thomas Legrand, Malay Haldar, Alex Deng, Han Zhao, Liwei He and Sanjeev Katariya. CIKM 2024.
- Metric Decomposition in A/B Tests.
Alex Deng, Luke Hagar, Nathaniel Stevens, Tatiana Xifara, Amit Gandhi. KDD 2024.
- Statistical challenges in online controlled experiments: A review of a/b testing methodology.
Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi and Nathaniel T. Stevens. The American Statistician 2027 Vol. 78.
- The Price is Right: Removing A/B Test Bias in a Marketplace of Expirable Goods.
Thu Le, Alex Deng. CIKM 2023.
- Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed Outcomes.
Alex Deng, Michelle Du, Anna Matlin and Qing Zhang. KDD 2023.
- Zero to Hero: Exploiting null effects to achieve variance reduction in experiments with one-sided triggering
Alex Deng, Lo-Hua Yuan, Naoya Kanai and Alexandre Salama-Manteau. WSDM 2023.
- A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments
Ron Kohavi, Alex Deng, Lukas Vermeer. KDD 2022.
- On Post-selection Inference in A/B Testing
Alex Deng, Yicheng Li, Jiannan Lu and Vivek Ramamurthy. KDD 2021.
- Empirical Bayes Estimation of Treatment Effects with Many A/B Tests: An Overview
Eduardo Azevedo, Alex Deng, Jose Montiel and Glen Weyl. AEA Papers and Proceedings 2019
- Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas
Alex Deng, Ulf Knoblich, Jiannan Lu. KDD 2018.
- The A/B Testing Problem
Eduardo M Azevedo, Alex Deng, Jose Luis Montiel Olea, Justin Rao, E Glen Weyl. EC 2018
- A note on type S/M errors in hypothesis testing(psyarxiv)
Jiannan Lu, Yixuan Qiu, Alex Deng. British Journal of Mathematical and Statistical Psychology 2018.
- On randomization-based causal inference for matched-pair
factorial designs
Jiannan Lu, Alex Deng. Statistics and Probability Letters, 2017.
- Trustworthy analysis of online A/B tests: Pitfalls, challenges and solutions
Alex Deng, Jiannan Lu, Jonathan Litz. WSDM 2017.
- Continuous monitoring of A/B tests without pain: Optional stopping in Bayesian testing (ArXiv ver.)
Alex Deng, Jiannan Lu, Shouyuan Chen. DSAA 2016.
- Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned
Alex Deng, Xiaolin Shi. KDD 2016.
- Demystifying the Bias from Selective Inference: a Revisit to Dawid’s Treatment Selection Problem
Jiannan Lu, Alex Deng. Statistics and Probability Letters, 2016.
- Objective Bayesian Two Sample Hypothesis Testing for Online Controlled Experiments (Slides)
Alex Deng. WWW 2015, The 1st Workshop on Offline and Online Evaluation of Web-based Services
- Diluted Treatment Effect Estimation for Trigger Analysis in Online Controlled Experiments (Slides)
Alex Deng and Victor Hu. WSDM 2015.
- Seven Rules of Thumb for Web Site Experimenters
Ron Kohavi, Alex Deng, Roger Longbotham, and Ya Xu. KDD 2014.
- Statistical Inference in Two-stage Online Controlled Experiments with Treatment Selection and Validation
Alex Deng, Tianxi Li and Yu Guo. WWW 2014.
- Online Controlled Experiments at Large Scale
Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, Nils Pohlmann. KDD 2013.
- Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data
Alex Deng, Ya Xu, Ron Kohavi and Toby Walker. WSDM 2013.
- Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained
Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu. KDD 2012.
- Sequential importance sampling and resampling for dynamic portfolio credit risk
Shaojie Deng, Kay Giesecke and Tze Lai. Operations Research Feb, 2012.
- Rare-event simulation of heavy-tailed random walks by sequential importance sampling and resampling
Hock Peng Chan, Shaojie Deng and Tze-Leung Lai. Advances in Applied Probability 2012.
Preprints
- From Augmentation to Decomposition: A New Look at CUPED in 2023
Alex Deng, Lke Hagar, Nathaniel Stevens, Tatiana Xifara, Lo-Hua Yuan, Amit Gandhi. Extended Abstract. CODE@MIT 2023 poster
- Concise Summarization of Heterogeneous Treatment Effect Using Total Variation Regularized Regression
Alex Deng, Pengchuan Zhang, Shouyuan Chen, Dong Woo Kim and Jiannan Lu. ArXiv
- Flexible Online Repeated Measures Experiment
Yu Guo and Alex Deng. ArXiv
Other Talks/Slides
Other (non-peer-reviewed)