Show that self-attention without positional information is permutation equivariant, and state what that means for a model reading a price series. Give two ways of supplying position and what each implies about extrapolating to longer sequences.

Show that self-attention without positional information is permutation equivariant, and state what that means for a model reading a price series. Give two ways of supplying position and what each implies about extrapolating to longer sequences.

Approach: Permute the input rows and track what happens to the query, key and value matrices and to the resulting output rows.

Permuting the input rows permutes the queries, keys and values identically, so the score matrix is permuted in both index sets and the output rows are the same vectors in the permuted order, which means attention alone cannot tell a rising price series from its reversal and position must be supplied explicitly. Formally, replacing X by PX gives Q = PXW_q and similarly for K and V, so QK^T becomes P(QK^T)P^T, the row-wise softmax is unchanged, and the output is P times the original output. Absolute encodings add a fixed or learned vector per index to the input; sinusoidal versions of different frequencies are defined for any index so they extend past the training length, while learned absolute embeddings have no value defined beyond the longest sequence seen and fail outright. Relative encodings instead bias the attention score by a function of i - j, which makes the model depend on distance rather than on absolute index, so it extrapolates to longer sequences and matches the structure of a price series where what matters is how many steps ago an event occurred. For financial sequences sampled irregularly the correct quantity is elapsed time rather than index distance, so the relative term should be a function of the time gap.

Follow-up: Rotary position embeddings apply a rotation to the queries and keys. Why does the resulting score depend only on i - j?

Key concepts: permutation equivariance, positional encoding, relative position, extrapolation.