thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

Data & Analytics · Foundational

Apache Spark

An open-source distributed engine for large-scale data processing with in-memory computation and APIs for SQL, Python, Scala, Java and R.

Definition

Spark provides Spark SQL and DataFrames, Structured Streaming, MLlib and GraphX. A driver program plans work as a DAG of stages and tasks executed in parallel by executors across a cluster. It runs on Databricks, EMR, Dataproc, Fabric, Synapse and Kubernetes, and NVIDIA's RAPIDS Accelerator can run Spark SQL on GPUs.

Exam tip

Spark is faster than MapReduce because it keeps intermediate data in memory and optimizes whole DAGs.

Related terms