What is 'Self-Attention' vs 'Cross-Attention'?

What is 'Self-Attention' vs 'Cross-Attention'?

Approach: Compare attention within a single sequence and between two different sequences.

Self-attention relates elements of a single sequence to each other (e.g., in an encoder). Cross-attention relates elements from two different sequences (e.g., decoder attending to encoder output in translation).

Follow-up: Which one is used in the original Transformer decoder for translation?

Key concepts: transformers, encoder-decoder, context.