Chapter 9 System Diagnosis and Quality Checks for A/B Tests

9.1 System Validation using A/A Test

9.2 Sample Ratio Mismatch

9.3 Trigger and Filter Condition

9.4 Interaction Detection

9.5 Metric Denominator Mismatch

  • Diagnosis
    • AA test and retroAA test,
      • p-value should be uniform!
      • Check performance balance!
    • SRM, common reason
      • Linkedin and Yahoo! both published paper, also our own papers
      • trigger condition issue
    • Check metric distribution, outlier
    • MDM
    • Trigger and filter compliment
    • Interaction detection