Google releases BERT and transforms NLP with bidirectional pre-training

Event Summary

BERT: Google’s 2018 breakthrough that revolutionized NLP with bidirectional pre-training. How BERT achieved SOTA on 11 benchmarks and became the foundation of modern language AI.

Impact Assessment

• Capability Leap +2 · Short-term

  Introduced masked language modeling for bidirectional pre-training, achieving state-of-the-art on 11 NLP benchmarks. Pre-trained BERT embeddings became the universal starting point for NLP systems until being superseded by larger-scale autoregressive models.

  Affected Groups: NLP researchers, AI engineers, Google

• Access Democratization +2 · Medium-term

  Open-sourced pre-trained models and weights made state-of-the-art NLP accessible to anyone with a GPU, significantly lowering the barrier to entry. Thousands of companies and research groups built on BERT, creating a rich ecosystem of fine-tuned models.

  Affected Groups: students, startups, independent researchers, small businesses

Consensus & Sources

• 1
  Devlin, J. et al. (2018) 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.' arXiv:1810.04805.

  URL: https://arxiv.org/abs/1810.04805

  BERT obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% absolute improvement).

  Reference Evidence Citation logged Live source
• 2
  "Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing" — Google AI Blog (2018)

  URL: https://research.google/blog/open-sourcing-bert-state-of-the-art-pre-training-for-natural-language-processing/

  We are releasing the pre-trained BERT model and code so that anyone can use it to build a question answering system or other language understanding system.

  Reference Evidence Citation logged Live source