Create your first workflow with Lakeflow Jobs

Orchestrate tasks with Lakeflow Jobs to read and process a sample dataset. In this quickstart, you:

1. Create a new notebook and add code to read a sample dataset of property bookings.
2. Save the dataset to Unity Catalog.
3. Create a new notebook and add code to read the dataset from Unity Catalog, filter it by year, and display the results.
4. Create a new job and configure two tasks using the notebooks.
5. Run the job and view the results.

Requirements

If your workspace is Unity Catalog-enabled and Serverless Jobs is enabled, by default, the job runs on Serverless compute. You do not need cluster creation permission to run your job with Serverless compute.

Otherwise, you must have cluster creation permission to create job compute or permissions to all-purpose compute resources.

This quickstart reads from samples.wanderbricks.bookings, which is available in every Unity Catalog-enabled workspace, so there is no source data to set up. To write the table that the second notebook reads, you need permission to create a schema in a catalog (the USE CATALOG and CREATE SCHEMA privileges).

To set these permissions, see your Databricks administrator or Unity Catalog privileges reference.

Create notebooks

The following steps create two notebooks to run in this workflow.

Retrieve and save data

To create a notebook that reads the sample dataset and saves it to Unity Catalog:

1. Click New in the sidebar, then click Notebook. Databricks creates and opens a new, blank notebook in your default folder. The default language is the language you most recently used, and the notebook is automatically attached to the compute resource that you most recently used.

2. (Optional) Rename the notebook Retrieve booking data.

3. If necessary, change the default language to Python.

4. Copy the following Python code and paste it into the first cell of the notebook. Before you run it, check that catalog and schema point to a location you can write to. The code creates the schema if it doesn't exist.

   catalog = "main"
   schema = "example_output"
   
   spark.sql(f"CREATE SCHEMA IF NOT EXISTS {catalog}.{schema}")
   
   bookings = spark.read.table("samples.wanderbricks.bookings")
   bookings.write.mode("overwrite").saveAsTable(f"{catalog}.{schema}.bookings")
   

Read and display filtered data

To create a notebook that filters and displays your data:

1. Click New in the sidebar, then click Notebook.

2. (Optional) Rename the notebook Filter booking data.

3. The following Python code reads the table you saved in the previous step and creates a temporary view. It also creates a widget that you can use to filter the data in the view by check-in year. Use the same catalog and schema values that you used in the first notebook.

   from pyspark.sql.functions import year
   
   catalog = "main"
   schema = "example_output"
   
   bookings = spark.read.table(f"{catalog}.{schema}.bookings")
   bookings.createOrReplaceTempView("bookings_table")
   years = spark.sql("SELECT DISTINCT year(check_in) AS year FROM bookings_table").toPandas()["year"].tolist()
   years.sort()
   dbutils.widgets.dropdown("year", "2025", [str(x) for x in years])
   display(bookings.filter(year(bookings.check_in) == dbutils.widgets.get("year")))
   

Create a job

The job you are creating consists of two tasks.

To create the first task:

1. In your workspace, click Jobs & Pipelines in the sidebar.
2. Click Create, then Job.
3. Click the Notebook tile to configure the first task. If the Notebook tile is not available, click Add another task type and search for Notebook.
4. (Optional) Replace the name of the job, which defaults to New Job <date-time>, with your job name.
5. In the Task name field, enter a name for the task; for example, retrieve-bookings.
6. If necessary, select Notebook from the Type drop-down menu.
7. In the Source drop-down menu, select Workspace, which allows you to use a notebook you saved previously.
8. For Path, use the file browser to find the first notebook you created, click the notebook name, and click Confirm.
9. Click Save task. A notification appears in the upper-right corner of the screen.

To create the second task:

1. Click Add task > Notebook.
2. In the Task name field, enter a name for the task; for example, filter-bookings.
3. For Path, use the file browser to find the second notebook you created, click the notebook name, and click Confirm.
4. Click Add under Parameters. In the Key field, enter year. In the Value field, enter 2025.
5. Click Save task.

Run the job

To run the job immediately, click in the upper right corner.

View run details

1. Click the Runs tab and click the link in the Start time column to open the run that you want to view.

2. Click either task to see the output and details. For example, click the filter-bookings task to view the output and run details for the filter task:

   

Run with different parameters

To re-run the job and filter bookings for a different year:

1. Click next to Run now and select Run now with different settings.
2. In the Value field, enter 2024.
3. Click Run.

Additional resources

• Configure job settings such as schedules, notifications, and parameters. See Configure and edit Lakeflow Jobs.
• Explore the available task types. See Types of tasks.
• Automatically run jobs on a schedule or when new data arrives. See Automate jobs with schedules and triggers.
• Monitor job runs and set up alerts. See Monitor Lakeflow Jobs.