Baking General Understanding into Language Models
In this article we’ll discuss “Bidirectional Encoder Representations from Transformers” (BERT), a model designed to understand language. While BERT is similar to models like GPT, the focus of BERT is to understand text rather than generate it. This is useful in a variety of tasks like ranking how positive a review of a product is, or predicting if an answer to a question is correct.
Before we get into BERT we’ll briefly discuss the transformer architecture, which is the direct inspiration of BERT. Using that understanding we’ll dive into BERT and discuss how it’s built and trained to solve problems by leveraging a general understanding of language. Finally, we’ll create a BERT model ourselves from scratch and use it to predict if product reviews are positive or negative.
Who is this useful for? Anyone who wants to form a complete understanding of the state of the art of AI.
How advanced is this post? Early parts of this article are accessible to readers of all levels, while later sections concerning the from-scratch implementation are fairly advanced. Supplemental resources are provided as necessary.
Pre-requisites: I would highly recommend understanding fundamental ideas about…