thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

Data & Analytics · Intermediate

Spark DataFrame

A distributed collection of rows organized into named, typed columns, optimized by Spark's Catalyst optimizer.

Definition

DataFrames are immutable: transformations return new DataFrames. They are preferred over raw RDDs because Catalyst and Tungsten optimize them. PySpark, Scala and Spark SQL DataFrame operations compile to the same plans.

Example

df.filter(df.amount > 100).groupBy('region').agg(F.sum('amount'))

Exam tip

Prefer DataFrames and built-in functions over RDDs and Python UDFs for performance.

Related terms