按年被提到
20142014 · 2 次2026
大模型与语言模型›词向量与预训练模型·arXiv:1411.2539 · cs.LG

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Ryan Kiros, Ruslan Salakhutdinov, Richard S. Zemel · 2014
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2014-11-18 · 最近 2016-10-10
3不同的人
2活跃月份
2讨论串
1,223被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduce the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder. The encoder allows one to rank images and sentences while the decoder can generate novel descriptions from scratch. Using LSTM to encode sentences, we match the state-of-the-art performance on Flickr8K and Flickr30K without using object detections. We also set new best results when using the 19-layer Oxford convolutional network. Furthermore we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* - "blue" + "red" is near images of red cars. Sample captions generated for 800 images are made available for comparison.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论benanne · 2014-11

“Looks like the Toronto group have been working on something very similar as well: http://lanl.arxiv.org/abs/1411.2539 Has anybody been able to find the Google paper? The article says it's on arxiv, but I can't seem to find it there. All that seems to be published so far is this blog post: http://googleresearch.blogspot.be/2014/11/a-picture-is-worth-thous…”

Researchers Announce Advance in Image-Recognition Software原评论 ↗
评论karpathy · 2014-11

“Hi all, I'm the author on the Stanford article, so I can answer any more detailed/technical questions. The article doesn't mention this, but closely related work I've only very recently become aware of (if anyone is interested) are additionally: From UofT: http://arxiv.org/pdf/1411.2539v1.pdf From Baidu/UCLA: http://arxiv.org/pdf/1410.1090v1.pdf From G…”

Researchers Announce Advance in Image-Recognition Software原评论 ↗
评论kastnerkyle · 2016-10

“There are a number of papers tackling a similar task [0][1][2] for anyone who is interested. There isn't enough information to tell exactly what is going on with Deepgram, but one way to approach this would be to construct a shared embedding space for words/phrases and speech. These types of embedding spaces are powerful [3][4][5], but not magic. Cool de…”

Deepgram – Find Damning Soundbites原评论 ↗