thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

Data & Analytics · Advanced

Repartition and Coalesce

Spark methods that change partition counts: repartition() fully shuffles to increase or rebalance; coalesce() merges partitions without a full shuffle.

Definition

Use coalesce to reduce the number of output files cheaply and repartition to increase parallelism or partition by a column.

Example

df.coalesce(8).write.parquet(path)

Exam tip

Reduce partitions = coalesce (no full shuffle); increase or rebalance = repartition (full shuffle).

Related terms