Google releases BERT, transforming NLP with bidirectional pre-training

事件摘要

Google AI published 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,' introducing a method that pre-trained a Transformer model bidirectionally on a large text corpus using masked language modeling. BERT achieved state-of-the-art results on 11 NLP benchmarks within months of release, and its open-source model and pre-trained weights made transfer learning in NLP accessible to anyone, launching the 'BERT era' of NLP.

影响评估

• Capability Leap +2 · Short-term

  Introduced masked language modeling for bidirectional pre-training, achieving state-of-the-art on 11 NLP benchmarks. Pre-trained BERT embeddings became the universal starting point for NLP systems until being superseded by larger-scale autoregressive models.

  Affected Groups: NLP researchers, AI engineers, Google

• Access Democratization +2 · Medium-term

  Open-sourced pre-trained models and weights made state-of-the-art NLP accessible to anyone with a GPU, significantly lowering the barrier to entry. Thousands of companies and research groups built on BERT, creating a rich ecosystem of fine-tuned models.

  Affected Groups: students, startups, independent researchers, small businesses

共识度与来源

• 1
  Devlin, J. et al. (2018) 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.' arXiv:1810.04805.

  URL: https://arxiv.org/abs/1810.04805

  BERT obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% absolute improvement).

  Reference Evidence Citation logged Live source
• 2
  "Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing" — Google AI Blog (2018)

  URL: https://research.google/blog/open-sourcing-bert-state-of-the-art-pre-training-for-natural-language-processing/

  We are releasing the pre-trained BERT model and code so that anyone can use it to build a question answering system or other language understanding system.

  Reference Evidence Citation logged Live source