thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

Data & Analytics · Advanced

Shuffle

Redistributing data across Spark executors by key, involving disk and network I/O, to perform joins, aggregations or repartitioning.

Definition

Shuffles are often the main performance bottleneck. spark.sql.shuffle.partitions (default 200) sets the output partition count, and Adaptive Query Execution can coalesce them automatically. Broadcast joins avoid shuffling the large table.

Exam tip

Reduce shuffles with broadcast joins, filtering early and avoiding unnecessary repartitioning.

Related terms