按年被提到
20152016 · 2 次2026
机器学习与深度学习›贝叶斯方法与 AutoML·arXiv:1512.04922 · math.ST

Always Valid Inference: Bringing Sequential Analysis to A/B Testing

Ramesh Johari, Leo Pekelis, David J. Walsh · 2015
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2016-04-08 · 最近 2016-08-17
2不同的人
2活跃月份
2讨论串
42被引(OpenAlex)
arXiv 摘要(候选层不写导读)

A/B tests are typically analyzed via frequentist p-values and confidence intervals; but these inferences are wholly unreliable if users endogenously choose samples sizes by *continuously monitoring* their tests. We define *always valid* p-values and confidence intervals that let users try to take advantage of data as fast as it becomes available, providing valid statistical inference whenever they make their decision. Always valid inference can be interpreted as a natural interface for a sequential hypothesis test, which empowers users to implement a modified test tailored to them. In particular, we show in an appropriate sense that the measures we develop tradeoff sample size and power efficiently, despite a lack of prior knowledge of the user's relative preference between these two goals. We also use always valid p-values to obtain multiple hypothesis testing control in the sequential context. Our methodology has been implemented in a large scale commercial A/B testing platform to analyze hundreds of thousands of experiments to date.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论leo_pekelis · 2016-04

“Hi, this is Leo, Optimizely's statistician. If you're looking for a more scientific paper, maybe take a look at this one we wrote recently: http://arxiv.org/abs/1512.04922 Should have everything you would ever want to know about the method. I agree with you that the problem of inference and interpretation between A/B data, algorithms, and the people who…”

20 lines of code that beat A/B testing (2012)原评论 ↗
被提交为 story · 1 次