thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

Data & Analytics · Intermediate

Spark Transformation

A lazy Spark operation that defines a new DataFrame from an existing one, such as select, filter, withColumn, join or groupBy.

Definition

Narrow transformations (filter, select) work within a partition. Wide transformations (groupBy, join, distinct, orderBy) need a shuffle across partitions.

Exam tip

filter and select are narrow; groupBy and join are wide and trigger a shuffle.

Related terms