按年被提到
20192024 · 2 次2026
机器学习与深度学习›贝叶斯方法与 AutoML·arXiv:1906.06852 · cs.LG

Learning Interpretable Models Using Uncertainty Oracles

Abhishek Ghose, Balaraman Ravindran · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-04-07 · 最近 2024-11-30
2不同的人
2活跃月份
2讨论串
0被引(OpenAlex)
arXiv 摘要(候选层不写导读)

A desirable property of interpretable models is small size, so that they are easily understandable by humans. This leads to the following challenges: (a) small sizes typically imply diminished accuracy, and (b) bespoke levers provided by model families to restrict size, e.g., L1 regularization, might be insufficient to reach the desired size-accuracy trade-off. We address these challenges here. Earlier work has shown that learning the training distribution creates accurate small models. Our contribution is a new technique that exploits this idea. The training distribution is encoded as a Dirichlet Process to allow for a flexible number of modes that is learnable from the data. Its parameters are learned using Bayesian Optimization; a design choice that makes the technique applicable to non-differentiable loss functions. To avoid the challenges with high dimensionality, the data is first projected down to one-dimension using uncertainty scores of a separate probabilistic model, that we refer to as the uncertainty oracle. We show that this technique addresses the above challenges: (a) it arrests the reduction in accuracy that comes from shrinking a model (in some cases we observe $\sim 100\%$ improvement over baselines), and also, (b) that this maybe applied with no change across model families with different notions of size; results are shown for Decision Trees, Linear Probability models and Gradient Boosted Models. Additionally, we show that (1) it is more accurate than its predecessor, (2) requires only one hyperparameter to be set in practice, (3) accommodates a multi-variate notion of model size, e.g., both maximum depth of a tree and number of trees in Gradient Boosted Models, and (4) works across different feature spaces between the uncertainty oracle and the interpretable model, e.g., a GRU might act as an oracle for a decision tree that ingests n-grams.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论abhgh · 2024-04

“Not for DL models, but I was exploring doing this in a specific setting (I've posted about this earlier). For small-sized models (for some reasonable definition of size, e.g., depth of a decision tree, or # trees in a gradient boosting forest, or # non-zero coefficients in a linear model), I realized that you could make them even smaller while retaining t…”

Language models are Super Mario: Absorbing abilities from homologous models原评论 ↗
评论aspenmayer · 2024-11

“Is a preprint of your paper available? I looked at your blog a bit and was able to find this, which may be it? > Learning Interpretable Models Using Uncertainty Oracles https://arxiv.org/abs/1906.06852 https://doi.org/10.48550/arXiv.1906.06852”

A statistical approach to model evaluations原评论 ↗