thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

Data & Analytics · Advanced

Catalyst Optimizer

Spark SQL's extensible optimizer, which turns logical plans into optimized physical plans using rule-based and cost-based optimization.

Definition

Catalyst applies predicate pushdown, column pruning and constant folding, and chooses join strategies. Python UDFs are opaque to it, which limits optimization.

Exam tip

Built-in Spark functions are optimized by Catalyst; Python UDFs are black boxes.

Related terms