按年被提到
20232023 · 5 次2026
视觉、语音与图像生成›视觉语言与自监督表征·arXiv:2305.11675 · cs.CV

Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity

Zijiao Chen, Jiaxin Qing, Juan Helen Zhou · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 6 人首次 2023-05-24 · 最近 2024-01-16
6不同的人
2活跃月份
6讨论串
39被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Reconstructing human vision from brain activities has been an appealing task that helps to understand our cognitive process. Even though recent research has seen great success in reconstructing static images from non-invasive brain recordings, work on recovering continuous visual experiences in the form of videos is limited. In this work, we propose Mind-Video that learns spatiotemporal information from continuous fMRI data of the cerebral cortex progressively through masked brain modeling, multimodal contrastive learning with spatiotemporal attention, and co-training with an augmented Stable Diffusion model that incorporates network temporal inflation. We show that high-quality videos of arbitrary frame rates can be reconstructed with Mind-Video using adversarial guidance. The recovered videos were evaluated with various semantic and pixel-level metrics. We achieved an average accuracy of 85% in semantic classification tasks and 0.19 in structural similarity index (SSIM), outperforming the previous state-of-the-art by 45%. We also show that our model is biologically plausible and interpretable, reflecting established physiological processes.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论tlyleung · 2024-01

“This was presented at NeurIPS 2023: Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity https://arxiv.org/abs/2305.11675”

Ask HN: What's a promising area to work on in 2024 -ideally not crowded already?原评论 ↗
被提交为 story · 5 次