{"id":122,"date":"2024-01-06T13:23:58","date_gmt":"2024-01-06T13:23:58","guid":{"rendered":"https:\/\/www.tensorloops.it\/?p=122"},"modified":"2024-01-06T18:32:00","modified_gmt":"2024-01-06T18:32:00","slug":"seminal-papers-about-large-language-models","status":"publish","type":"post","link":"https:\/\/www.tensorloops.it\/it\/2024\/01\/06\/seminal-papers-about-large-language-models\/","title":{"rendered":"Seminal Papers about Large Language Models"},"content":{"rendered":"
\"\"<\/figure>\n\n\n\n
    \n
  1. “<\/strong>Improving Language Understanding by Generative Pre-Training” by Radford et al. (2018)<\/strong><\/a>: This is the paper that introduced the first version of the GPT model. It laid the foundation for the use of transformer-based models in natural language processing.<\/li>\n\n\n\n
  2. “<\/strong>Language Models are Unsupervised Multitask Learners” by Radford et al. (2019)<\/strong><\/a>: This paper presents GPT-2, an extension of the original GPT model, with significantly more parameters and trained on a larger dataset.<\/li>\n\n\n\n
  3. “<\/strong>Language Models are Few-Shot Learners” by Brown et al. (2020)<\/strong><\/a>: This paper introduces GPT-3, the third iteration in the GPT series. It highlights the model’s few-shot learning capabilities, where it performs tasks with minimal task-specific data.<\/li>\n\n\n\n
  4. BERT: “Pre-training of Deep Bidirectional Transformers for Language Understanding<\/strong><\/a>” by Devlin et al. (2018)<\/strong>: While not a GPT paper, this work by researchers at Google is a seminal paper in the field of LLMs. BERT introduced a new method of pre-training language representations that was revolutionary in the field.<\/li>\n\n\n\n
  5. “<\/strong>Attention Is All You Need” by Vaswani et al. (2017)<\/strong><\/a>: This paper, although not directly related to GPT, is crucial as it introduced the transformer architecture, which is the backbone of models like GPT-2 and GPT-3.<\/li>\n\n\n\n
  6. “<\/strong>Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer<\/strong><\/a>” by Raffel et al. (2019)<\/strong>: This paper from Google researchers presents the T5 model, which treats every language problem as a text-to-text problem, providing a unified framework for various NLP tasks.<\/li>\n\n\n\n
  7. “<\/strong>XLNet: Generalized Autoregressive Pretraining for Language Understanding<\/strong><\/a>” by Yang et al. (2019)<\/strong>: XLNet is another important model in the LLM domain, which outperformed BERT on several benchmarks by using a generalized autoregressive pretraining method. <\/li>\n\n\n\n
  8. “<\/strong>ERNIE: Enhanced Representation through Knowledge Integration<\/strong><\/a>” by Sun et al. (2019)<\/strong>: Developed by Baidu, ERNIE is an LLM that integrates lexical, syntactic, and semantic information effectively, showing significant improvements over BERT in various NLP tasks<\/li>\n<\/ol>\n\n\n\n

    <\/p>","protected":false},"excerpt":{"rendered":"","protected":false},"author":2,"featured_media":125,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[6,23,9,24,25,8,22,20,21],"class_list":["post-122","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-generative-ai","tag-artificial-intelligence","tag-chatgpt","tag-generative-ai","tag-gpt-2","tag-gpt-4","tag-large-language-models","tag-open-ai","tag-papers","tag-transformes"],"_links":{"self":[{"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/posts\/122","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/comments?post=122"}],"version-history":[{"count":2,"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/posts\/122\/revisions"}],"predecessor-version":[{"id":126,"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/posts\/122\/revisions\/126"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/media\/125"}],"wp:attachment":[{"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/media?parent=122"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/categories?post=122"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tensorloops.it\/it\/wp-json\/wp\/v2\/tags?post=122"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}