按年被提到
20242024 · 2 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2402.08268 · cs.LG

World Model on Million-Length Video And Language With Blockwise RingAttention

Hao Liu, Wilson Yan, Matei Zaharia, Pieter Abbeel · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-02-16 · 最近 2024-04-08
2不同的人
2活跃月份
2讨论串
45被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Enabling long-context understanding remains a key challenge in scaling existing sequence models -- a crucial component in developing generally intelligent models that can process and operate over long temporal horizons that potentially consist of millions of tokens. In this paper, we aim to address these challenges by providing a comprehensive exploration of the full development process for producing 1M context language models and video-language models, setting new benchmarks in language retrieval and new capabilities in long video understanding. We detail our long context data curation process, progressive context extension from 4K to 1M tokens, and present an efficient open-source implementation for scalable training on long sequences. Additionally, we open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论woadwarrior01 · 2024-02

“From the paper[1]: > We additionally scale our inference code to support million-length sequences by implementing RingAttention for decoding. Inference for such long sequences requires a minimum of v4-128 with a TPU mesh sharding of 32 tensor parallelism, and 4 sequence parallelism (ring dimension). Each TPU-v4 has 32 GiB of HBM memory, so about 4 TiB (…”

LWM – Open LLM with 1M Tokens Context Window原评论 ↗
被提交为 story · 1 次