Top Databricks Interview Questions & Answers (2026)

Auto Loader is the cloudFiles source for Structured Streaming. It discovers new files in cloud storage incrementally, by directory listing or file notifications, and records processed files in the stream's checkpoint, so each file is processed once even after restarts. I usually run it with an availableNow trigger from a scheduled job, which processes everything new and then stops. For schema, I set a schemaLocation so it can infer and track the schema. In the default addNewColumns mode, when a new column appears the stream fails, updates the schema, and continues on restart, which is why I run it as a job with retries. Data that does not fit lands in the _rescued_data column instead of being silently lost.

I use MERGE INTO target USING source ON the business key, with WHEN MATCHED THEN UPDATE and WHEN NOT MATCHED THEN INSERT, and optionally WHEN NOT MATCHED BY SOURCE for deletes. The trap is duplicates: if two source rows match the same target row, MERGE fails because it cannot decide which update wins. So I deduplicate the batch first, typically with a window function, ROW_NUMBER over the key ordered by event timestamp descending, keeping row one. For streaming, I run the MERGE inside foreachBatch so each micro-batch is deduplicated and merged. I also make sure the merge condition can use data skipping, for example including a date range, otherwise MERGE scans the whole target table.

They are the declarative pipeline framework formerly called Delta Live Tables, now built on Apache Spark Declarative Pipelines. I define streaming tables and materialized views in SQL or Python, and the framework works out dependencies, orchestration, retries, incremental processing and infrastructure. Data quality is built in through expectations: warn and keep the record, drop the row, or fail the update. I would use them for standard ingestion and transformation flows, especially bronze to silver with quality rules and change data capture, because there is much less code to maintain. I would still use notebook or Python tasks in Lakeflow Jobs for unusual logic, API calls, or steps that do not fit the declarative model.

I define a job with tasks forming a DAG: for example an ingestion pipeline task, then notebook or SQL tasks for silver and gold, then a dashboard refresh. Dependencies decide order, and tasks can run in parallel where independent. Each task gets retries with a sensible count and timeout, so transient failures recover but a stuck task does not run forever. For triggers I choose between a schedule, a file arrival trigger when data lands unpredictably, or a table update trigger when the job should run after an upstream table changes. I use job parameters and task values to pass dates or flags between tasks, if/else condition tasks for branching, and for-each tasks for looping. Failure notifications go to a channel the team watches.

For the column, I create a SQL function that returns the real value when is_account_group_member of an authorised group is true and a masked value otherwise, then attach it to the column with ALTER TABLE SET MASK. For rows, I create a filter function that returns true only for rows the user's group may see, for example by joining region to a mapping, and apply it with SET ROW FILTER. Everyone queries the same table and sees what they are entitled to. At scale, attribute-based access control policies let me tag columns, like PII, and apply masking centrally rather than table by table. Underneath, grants follow least privilege: groups not individuals, and USE CATALOG and USE SCHEMA before SELECT.