Databricks Certified Data Engineer Associate: key terms

Flashcards for this exam Databricks Certified Data Engineer Associate guide

• ACIDThe four properties of reliable transactions: Atomicity, Consistency, Isolation and Durability.Foundational
• Adaptive Query Execution (AQE)A Spark 3 feature that re-optimizes query plans at run time using actual shuffle statistics.Advanced
• Anti JoinA join pattern that returns rows from one table that have no match in another.Intermediate
• Apache AirflowAn open-source platform for authoring, scheduling and monitoring workflows as Python-defined DAGs.Intermediate
• Apache AvroA row-based binary file format with an embedded JSON schema, well suited to streaming, write-heavy workloads and schema evolution.Intermediate
• Apache HiveA data warehouse system on Hadoop that lets users query files with a SQL-like language (HiveQL) using schema on read.Intermediate
• Apache KafkaA distributed event streaming platform that stores ordered, durable, replayable logs of records in partitioned topics.Intermediate
• Apache ParquetAn open-source columnar file format with efficient compression, encoding and predicate pushdown, widely used for analytics.Intermediate
• Apache SparkAn open-source distributed engine for large-scale data processing with in-memory computation and APIs for SQL, Python, Scala, Java and R.Foundational
• Auto LoaderA Databricks feature (cloudFiles source) that incrementally and efficiently ingests new files from cloud storage, with schema inference and evolution.Intermediate
• BackfillRunning a pipeline over historical periods to populate or correct past data.Intermediate
• Batch ProcessingProcessing accumulated data in discrete scheduled jobs, such as hourly or nightly loads.Foundational
• Broadcast JoinA join strategy that sends a copy of a small table to every executor so the large table is joined without being shuffled.Advanced
• Bronze LayerThe raw landing layer of a medallion architecture, holding source data as ingested, often append-only with load metadata.Intermediate
• Catalyst OptimizerSpark SQL's extensible optimizer, which turns logical plans into optimized physical plans using rule-based and cost-based optimization.Advanced
• Change Data Capture (Data Engineering) (CDC)Identifying and capturing row-level inserts, updates and deletes in a source so only changes are propagated downstream.Intermediate
• Change Data Feed (CDF)A Delta Lake feature that records row-level inserts, updates and deletes between table versions so downstream jobs can process only changes.Advanced
• CheckpointingPersisting a streaming query's progress (offsets) and state to durable storage so it can recover after failure.Advanced
• Column-Level SecurityRestricting access to specific columns, for example through column GRANTs, masking policies or object-level security.Intermediate
• COPY INTOThe command that bulk-loads staged files into a table (COPY INTO <table>) or unloads query results to a stage (COPY INTO <location>).Intermediate
• CREATE TABLE AS SELECT (CTAS)A statement that creates a new table and fills it with the results of a query in one step.Intermediate
• Data CatalogAn inventory of an organization's data assets with metadata, descriptions, ownership, classifications, lineage and search for discovery.Intermediate
• Data ContractA formal agreement between data producers and consumers that specifies schema, semantics, quality expectations, SLAs and change management for a dataset.Advanced
• Data FederationRunning a single query across multiple independent databases and returning combined results.Intermediate
• Data FreshnessHow recently data was updated relative to expectations, usually monitored as the time since the last successful load.Intermediate
• Data IngestionThe process of importing data from sources into a storage or processing platform, in batches or as a stream.Foundational
• Data LakeA central repository that stores large volumes of raw data in any format (structured, semi-structured or unstructured) cheaply, usually in object storage.Foundational
• Data LakehouseAn architecture that adds warehouse features such as ACID transactions, schema enforcement and fast SQL to low-cost data lake storage using open table formats.Intermediate
• Data ObservabilityContinuous monitoring of data health across pipelines (freshness, volume, schema, distribution and lineage) to detect and resolve issues quickly.Intermediate
• Data OrchestrationCoordinating the scheduling, dependencies, retries and monitoring of data pipeline tasks.Intermediate
• Data PipelineAn automated series of steps that moves data from sources through transformations to destinations.Foundational
• Data SkewAn uneven distribution of data across partitions, so a few tasks process far more data and slow the whole job.Advanced
• Data SkippingUsing per-file min/max statistics to skip files that cannot contain rows matching a query filter.Advanced
• Databricks Asset Bundles (DABs)A Databricks infrastructure-as-code format that defines jobs, pipelines and other resources in YAML for versioned CI/CD deployment across environments.Advanced
• Databricks SQL WarehouseDatabricks compute optimized for SQL analytics and BI, available as serverless, pro or classic, with Photon and intelligent workload management.Intermediate
• dbtA transformation framework (data build tool) where analysts write SQL SELECT models that run in the warehouse, with tests, docs and lineage.Intermediate
• Delta LakeAn open-source table format that adds ACID transactions, schema enforcement and evolution, time travel and MERGE to Parquet files.Intermediate
• Delta Lake Time TravelQuerying or restoring earlier versions of a Delta table by version number or timestamp.Intermediate
• Delta SharingAn open protocol for securely sharing live Delta Lake and other data with any platform, without copying it.Advanced
• Delta Transaction LogThe _delta_log folder of ordered JSON commit files and Parquet checkpoints that records every change to a Delta table and provides ACID guarantees.Advanced
• Directed Acyclic Graph (DAG)A graph of tasks with directed dependencies and no cycles, used to define pipeline workflows and execution plans.Intermediate
• Event TimeThe time an event actually occurred at its source, as opposed to processing time, when the system handles it.Advanced
• Exactly-Once Semantics (EOS)A processing guarantee that each record affects the output exactly one time despite failures and retries.Advanced
• External TableA table whose data files reside in external storage and are queried in place, with only metadata kept in the database.Intermediate
• Gold LayerThe curated medallion layer that holds business-level aggregates and dimensional models ready for BI and reporting.Intermediate
• Hive Metastore (HMS)A central repository of table definitions, schemas, partitions and file locations used by Hive, Spark and other engines.Advanced
• Idempotent PipelineA pipeline that produces the same result no matter how many times it runs on the same input.Advanced
• Identity ColumnA column whose values the database generates automatically, typically incrementing integers, often used as a surrogate key.Intermediate
• Incremental LoadLoading only data that is new or changed since the last run, instead of reloading everything.Intermediate
• LAG and LEAD FunctionsWindow functions that return a value from a previous row (LAG) or a following row (LEAD) within a partition.Intermediate
• Lakeflow Declarative PipelinesDatabricks' declarative ETL framework (formerly Delta Live Tables) where you define tables and the platform manages orchestration and data quality.Intermediate
• Lakeflow JobsDatabricks' orchestration service, formerly Workflows/Jobs, for scheduling multi-task DAGs of notebooks, SQL, pipelines, dbt and Python tasks.Intermediate
• Lakehouse FederationA Databricks feature that queries external databases (PostgreSQL, Snowflake, SQL Server) through Unity Catalog foreign catalogs without ingesting data.Advanced
• Late-Arriving DataRecords that arrive after the window or batch for their event time has already been processed.Advanced
• Lazy EvaluationDeferring execution of transformations until an action requires a result, which lets the engine optimize the whole plan.Intermediate
• Liquid ClusteringA Delta Lake data layout feature that incrementally clusters data by chosen keys, which can be changed without rewriting the table.Advanced
• Managed TableA table whose data files and metadata are both managed by the platform; dropping it deletes the data.Intermediate
• Materialized ViewA database object that stores the precomputed result of a query and refreshes it as base data changes.Intermediate
• Medallion ArchitectureA lakehouse design pattern that refines data progressively through Bronze (raw), Silver (cleansed) and Gold (business-ready) layers.Intermediate
• MERGE StatementA single DML statement that inserts, updates or deletes target rows based on whether they match rows in a source.Intermediate
• Micro-Batch ProcessingA streaming approach that processes incoming data in very small, frequent batches.Intermediate
• Open Table FormatA metadata layer over data files (usually Parquet) that adds ACID transactions, schema evolution, time travel and efficient pruning to data lakes.Advanced
• OPTIMIZE CommandA Delta Lake command that compacts small files into larger ones and optionally applies Z-ordering or liquid clustering.Advanced
• Partition PruningSkipping partitions whose data cannot satisfy a query's filter, which reduces the data scanned.Intermediate
• PhotonDatabricks' native vectorized query engine, written in C++, that accelerates Spark SQL and DataFrame workloads.Advanced
• Pipeline ExpectationsData quality constraints in Lakeflow Declarative Pipelines (DLT) that warn, drop invalid rows or fail the update when records violate them.Advanced
• QUALIFY ClauseA clause in Snowflake, Databricks SQL, BigQuery and Teradata that filters rows on the result of window functions.Intermediate
• Query Execution PlanThe sequence of physical operations (scans, seeks, joins, sorts) that the database chooses to run a query.Intermediate
• Repartition and CoalesceSpark methods that change partition counts: repartition() fully shuffles to increase or rebalance; coalesce() merges partitions without a full shuffle.Advanced
• Resilient Distributed Dataset (RDD)Spark's low-level, immutable, partitioned collection of objects that can be rebuilt from lineage after a failure.Intermediate
• ROW_NUMBER FunctionA window function that assigns a unique sequential integer to rows within each partition according to the ORDER BY.Intermediate
• SCD Type 1A slowly changing dimension method that overwrites the old attribute value, keeping no history.Intermediate
• SCD Type 2A slowly changing dimension method that inserts a new row for each change, with effective dates and a current-row flag, preserving full history.Intermediate
• Schema DriftUnplanned changes in a source's structure, such as new, removed or renamed columns or changed types, that can break pipelines.Intermediate
• Schema EvolutionChanging a table's or stream's schema over time, for example by adding columns, while keeping existing data and consumers working.Intermediate
• Semi JoinA join pattern that returns rows from one table that have at least one match in another, without duplicating them.Advanced
• Separation of Storage and ComputeAn architecture in which data lives in independent storage and compute resources scale, start and stop separately.Intermediate
• Session WindowA dynamic window per key that groups events separated by less than a gap of inactivity.Advanced
• ShuffleRedistributing data across Spark executors by key, involving disk and network I/O, to perform joins, aggregations or repartitioning.Advanced
• Silver LayerThe medallion layer that holds cleansed, deduplicated, validated and conformed data at roughly entity grain.Intermediate
• Sliding Window (Stream Processing)Fixed-size windows that overlap because they advance by a smaller interval, so an event can fall into several windows.Advanced
• Slowly Changing Dimension (SCD)Techniques for handling changes to dimension attributes over time, such as a customer moving city.Intermediate
• Small File ProblemPerformance degradation caused by large numbers of tiny files, which increase metadata overhead and per-file open costs.Advanced
• Spark ActionA Spark operation that triggers execution and returns a result to the driver or writes output, such as count, collect, show, take or write.Intermediate
• Spark CachingPersisting a DataFrame in memory and/or disk with cache() or persist() so that repeated actions reuse it instead of recomputing.Intermediate
• Spark DataFrameA distributed collection of rows organized into named, typed columns, optimized by Spark's Catalyst optimizer.Intermediate
• Spark DriverThe Spark process that runs the main program, builds the DAG, schedules tasks and collects results.Intermediate
• Spark ExecutorA JVM process on a worker node that runs Spark tasks and stores cached data for an application.Intermediate
• Spark StageA set of Spark tasks that can run without a shuffle; a job is split into stages at shuffle boundaries.Advanced
• Spark TransformationA lazy Spark operation that defines a new DataFrame from an existing one, such as select, filter, withColumn, join or groupBy.Intermediate
• SpillingWriting intermediate query data to local disk, or remote cloud storage, when warehouse or executor memory is insufficient.Advanced
• Stream ProcessingContinuously processing data records as they arrive to produce near-real-time results.Intermediate
• Structured StreamingSpark's stream processing engine, which treats a stream as an unbounded table and processes it incrementally with the DataFrame API.Intermediate
• Table PartitioningDividing a large table into smaller physical pieces by a key such as date, while queries still see a single table.Intermediate
• Three-Level NamespaceAddressing tables as catalog.schema.table in Unity Catalog, or database.schema.object in Snowflake.Intermediate
• Tumbling WindowFixed-size, non-overlapping, contiguous time windows; each event belongs to exactly one window.Advanced
• Unity Catalog (UC)Databricks' unified governance layer for data and AI assets, providing centralized access control, auditing, lineage and discovery across workspaces.Intermediate
• User-Defined Function (UDF)A custom function created in the database that returns a scalar value or a table and can be called inside SQL statements.Intermediate
• VACUUM CommandA Delta Lake command that permanently deletes data files no longer referenced by the table and older than the retention threshold (7 days by default).Advanced
• Watermark (Streaming)A threshold in stream processing that estimates how far event time has progressed, determining when windows can close and how late data is handled.Advanced
• Wide TransformationA Spark transformation in which output partitions depend on data from many input partitions, requiring a shuffle.Advanced
• Window FunctionA function that computes a value across a set of rows related to the current row, defined with OVER(), without collapsing them.Intermediate
• WindowingGrouping unbounded streaming data into finite time-based buckets for aggregation.Advanced
• Z-OrderingA Delta Lake technique that co-locates related values across files using a space-filling curve, improving data skipping on frequently filtered columns.Advanced