What the model is looking at

• Stage · i

  Query, key, value

  Each token is transformed into three vectors, what to look for, what can be offered, and the content itself.

• Stage · ii

  Attention scores

  Dot products between query and key vectors, then softmax, producing the attention weights.

• Stage · iii

  Weighted sum

  Multiply attention weights with value vectors. That yields the final output for each token.

Multi-head attention transformers run several of these in parallel. Each head is free to specialise on a different relationship: syntax, semantics, a hunch about coreference. Together, they make the model rich.

• load →The cat sat on the mat
• load →Artificial intelligence is transforming the world
• load →Attention is all you need for transformers
• load →Machine learning models learn patterns from data