What is the 'Shuffle' process in Spark?

What is the 'Shuffle' process in Spark?

Approach: Describe data redistribution across a cluster.

The mechanism for redistributing data across partitions. It occurs when a transformation requires data from other partitions (e.g., reduceByKey or join), involving high disk and network I/O.

Follow-up: Why should you try to avoid shuffles?

Key concepts: network io, partitioning, wide transformation.