“They don't perform poorly (depends on how much corpus training the language received). https://arxiv.org/abs/2108.13349. The important thing is you don't need an equivalent number of tokens for an equivalent performance. So even though GPT-3 only had 1.8 % french by word count, it can speak fluent french. There's positive transfer between languages. Also…”
On the Multilingual Capabilities of Very Large-Scale English Language Models
Generative Pre-trained Transformers (GPTs) have recently been scaled to unprecedented sizes in the history of machine learning. These models, solely trained on the language modeling objective, have been shown to exhibit outstanding few-shot learning capabilities in a number of different tasks. Nevertheless, aside from anecdotal experiences, little is known regarding their multilingual capabilities, given the fact that the pre-training corpus is almost entirely composed of English text. In this work, we investigate the multilingual skills of GPT-3, focusing on one language that barely appears in the pre-training corpus, Catalan, which makes the results especially meaningful; we assume that our results may be relevant for other languages as well. We find that the model shows an outstanding performance, particularly in generative tasks, with predictable limitations mostly in language understanding tasks but still with remarkable results given the zero-shot scenario. We investigate its potential and limits in extractive question-answering and natural language generation, as well as the effect of scale in terms of model size.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Doesn't really matter. There's lots of positive transfer in individual language learning. Competence in one language bleeds into competence in others. https://arxiv.org/abs/2108.13349 GPT-3 is fluent in many languages despite English taking up 93% of the corpus by word count. French is next with 1.8% https://github.com/openai/gpt-3/blob/master/dataset_s…”
“1. LLMs are general pattern machines. They can generalize and complete in context to a wide variety of complex non linguistic patterns.https://general-pattern-machines.github.io/ 2. LLMs see positive transfer in multi lingual capability. For example, an LLM trained on 500B tokens of English and 10B tokens of French will not speak in french like a model t…”
“>on what basis? What makes a predictor LLM better at translating Latin than a system trained specifically for translation? They just are. Sure it sounds a bit strange if you've never thought about it but they are. >I'm sure they can do a decent job but it's weird to me that someone would leap to GPT-style tech despite its known tendency to hallucinate/m…”
“Language Models trained at the size and data they tested are pretty terrible to begin with, translation or not. Like I said, the crucial thing is that it looks to be a diminishing gap. I'm not saying parallel corpora is doing nothing or is unimportant but that's a clue that monolingual competency of the languages in question is the biggest factor here. T…”
“>I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili. A thought experiment from other people comments on another language. So...No. Fabricating failure modes from their personally constructed ideas …”