按年被提到
20192019 · 1 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:1903.11257 · cs.LG

How Can We Be So Dense? The Benefits of Using Highly Sparse Representations

Subutai Ahmad, Luiz Scheinkman · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2019-08-19 · 最近 2020-10-07
2不同的人
2活跃月份
2讨论串
72被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as long as the underlying dimensionality is sufficiently high. A key intuition that we develop is that the ratio of the operable volume around a sparse vector divided by the volume of the representational space decreases exponentially with dimensionality. We then analyze computationally efficient sparse networks containing both sparse weights and activations. Simulations on MNIST and the Google Speech Command Dataset show that such networks demonstrate significantly improved robustness and stability compared to dense networks, while maintaining competitive accuracy. We discuss the potential benefits of sparsity on accuracy, noise robustness, hyperparameter tuning, learning speed, computational efficiency, and power requirements.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论p1esk · 2019-08

“Good point. I think it's a safe bet to focus on dense dot product in hardware for the foreseeable future. However, to their defense: 1. It's not clear that supporting sparse operands in hw would result in significant overhead. 2. DL models are still pretty sparse (I bet even those models with GELU still have lots of very small values that could be safel…”

Cerebras Systems unveils a 1.2T transistor chip for AI原评论 ↗
被提交为 story · 1 次