What is 'Self-Attention' in Transformers?

What is 'Self-Attention' in Transformers?

Approach: Describe how the model relates different positions of a single sequence.

A mechanism that relates different positions of a single sequence in order to compute a representation of the sequence. It allows each word to 'attend' to every other word in the sentence to understand context.

Follow-up: What is the time complexity of self-attention relative to sequence length L?

Key concepts: query key value, weights, parallelism.