按年被提到
20212023 · 1 次2026
大模型与语言模型›序列模型与注意力机制·arXiv:2103.06874 · cs.CL

CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting · 2021
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-03-21 · 最近 2024-12-14
2不同的人
2活跃月份
2讨论串
161被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE, a neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by 2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论korec123 · 2023-03

“There is another line of work for efficient Transformers that wasn't mentioned in the paper, i.e., Adaptive Computation on the sequence level, which pools similar tokens which are easy to predict, therefore reducing the complexity of a Transformer layer. Examples [#1](https://arxiv.org/pdf/2211.09761.pdf) [#2](https://arxiv.org/pdf/2103.06874.pdf) [#3](ht…”

CoLT5: Faster Long-Range Transformers With Conditional Computation原评论 ↗
评论binarymax · 2024-12

“I was really excited for CANINE [1] but it never really went anywhere. Tokens are a hack. They work for the most part, but it’s clear when they don’t. [1] https://arxiv.org/abs/2103.06874”

Byte Latent Transformer: Patches Scale Better Than Tokens原评论 ↗