BERT Model - NLP

40 min read Generative AI

This is a free course. Log in to run the code exercises and track your progress.

Log in Enroll free

BERT Model - NLP

BERT (Bidirectional Encoder Representations from Transformers) is a Transformer-based language model from Google designed to capture the meaning and context of words in text. It builds contextual representations by using information from both the left and right sides of each token.

Working

1. Input Processing

BERT first splits the input into subword tokens with WordPiece tokenization and inserts special tokens that mark input structure.

  • [CLS] is placed at the start of the input. Its final representation can support tasks such as text classification.
  • [SEP] marks the end of a sentence or separates two sentences.

For example, a single sentence can be represented as:

[CLS] The cat is sleeping. [SEP]

For a pair of sentences:

[CLS] The cat is sleeping. [SEP] It is on the mat. [SEP]

Those tokens are converted to numerical IDs and passed through BERT’s Transformer encoder.

2. Masked Language Modeling (MLM)

  • In pre-training, BERT chooses some tokens as prediction targets and learns to recover their original values from surrounding context.

For example:

Input: The cat is [MASK] on the mat. Target: The cat is sleeping on the mat.

  • Most selected tokens are replaced with [MASK]; some are swapped for a random token or left unchanged.
  • Because BERT reads both directions, it can use words before and after the target when making its prediction.

3. Next Sentence Prediction (NSP)

  • BERT is also trained to model relationships between sentence pairs.
  • It receives two sentences and predicts whether the second follows the first in the training text.

For example:

Sentence A: The student went to the library. Sentence B: She borrowed a book.

  • Consecutive sentences in the training text form a positive pair.
  • For a negative pair, the second sentence is sampled from elsewhere in the corpus.
  • The model predicts whether the pair is IsNext or NotNext.

4. Combined Pre-training

  • BERT is pretrained with both MLM and NSP. It learns to recover selected tokens from context while also learning whether two sentences form a consecutive pair.
  • Those objectives give BERT general language representations before it is adapted to specific NLP tasks.

5. Fine-tuning

  • After pre-training, BERT can be adapted to a downstream NLP task such as text classification, Named Entity Recognition (NER), or question answering.
  • A task-specific output layer is attached, and the model is fine-tuned on labeled data for that task.

For example:

For sentiment analysis, BERT can be fine-tuned to classify a sentence as positive or negative.

  • The same pretrained BERT checkpoint can therefore be reused across many downstream tasks without training a language model from scratch.

Architecture

Transformer Encoder

BERT stacks multiple Transformer encoder layers. Each layer includes:

  • Multi-Head Self-Attention: Lets each token attend to other tokens in the sequence and capture contextual relationships.
  • Feed-Forward Network: Further transforms the representations coming from attention.
  • Residual Connections: Add a sub-layer’s input to its output to ease information flow through the network.
  • Layer Normalization: Normalizes representations to stabilize training.

Stacking several encoder layers yields increasingly contextual representations.

Input Representation

For every input token, BERT combines three embedding types:

  • Token Embeddings: Represent the individual tokens.
  • Segment Embeddings: Indicate which sentence a token belongs to when a pair of sentences is provided.
  • Position Embeddings: Encode each token’s position in the sequence.

These embeddings are summed and passed into the Transformer encoder.

BERTBASE and BERTLARGE

BERT was released in two sizes:

  • BERTBASE contains 12 Transformer encoder layers, 12 attention heads and a hidden size of 768, with about 110 million parameters.
  • BERTLARGE contains 24 encoder layers, 16 attention heads and a hidden size of 1024, with about 340 million parameters.

Both configurations support input sequences of up to 512 tokens.

Output Representations

BERT emits a contextual representation for each input token. Unlike static word embeddings, a word’s representation can change with the surrounding words.

For example, the representation of bank can differ in:

I deposited money in the bank.

and

We sat near the river bank.

The [CLS] output can support sequence-level classification, while token-level outputs suit tasks such as Named Entity Recognition.

BERT Tokenization

BERT uses WordPiece tokenization to split text into subword units. When a full word is missing from the vocabulary, the model can still represent it with smaller pieces.

For example:

A word can be divided into multiple subword tokens, where ## indicates a continuation subword in BERT's WordPiece vocabulary.

Python

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
text = "BERT understands language context."
tokens = tokenizer.tokenize(text)

print("Tokens:", tokens)