按年被提到
20152023 · 3 次2026
视觉、语音与图像生成›扩散模型·arXiv:1511.01844 · stat.ML

A note on the evaluation of generative models

Lucas Theis, Aäron van den Oord, Matthias Bethge · 2015
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2017-02-23 · 最近 2023-07-11
2不同的人
5活跃月份
5讨论串
699被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论317070 · 2017-02

“Actually, there is a formal method to evaluate generative models. All you need is a validation set. Then you need to find what the odds are that the model would have generated your validation set. That probability is the one you want to maximize [0]. In order to be able to calculate or estimate this probability, you need to make some amendments to your mo…”

AI learns to write its own code by stealing from other programs原评论 ↗
评论317070 · 2017-12

“GANs are for graphics, not machine learning. They have no test scores as they cannot run on the test set. Therefore, its hard to tell how much they overfit. But they have uses, like for lossy compression of images or texture generation. Stuff which focuses on the graphical side of things where it outperforms machine learning methods with its crisper samp…”

My Favorite Deep Learning Papers of 2017原评论 ↗
评论godelski · 2023-04

“It is also important to remember that there are different inductive biases for different networks. Sutton's "Bitter Lesson"[0] argues for scale + more arbitrary complexity. ViTs scale really well because some of this arbitrary complexity. But I agree with your point that what to use is not obvious and this gets really complicated real quickly. For exampl…”

Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows原评论 ↗
评论godelski · 2023-05

“Oh yeah, those are great papers, which I wish more people read. (I'll never not laugh at the journal name) I'll add this too[0] since it seems to be highly missed. But as far as validation sets, I have a hard time convincing anyone I talk to that tuning on test data is information leakage. I've also had issues with discussing people (even outside my lab, …”

Production AI systems are hard原评论 ↗
评论godelski · 2023-07

“> I have used LLMs on actually novel tasks for which there aren’t any datasets out there. They “get it”. Can you give an example so that we may better discuss or that I can adequately update my understandings? But I will say that simplifying this down to "just trained to predict the next token" is not accurate as it does not account for the differences i…”

What we learned from using GPT for 500k+ classifications原评论 ↗