Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Databricks Lakeflow Connect has fully managed streaming connectors for continuously ingesting data from message buses and event streaming sources into streaming tables. Each managed connector handles authentication for each source, message decoding, and pipeline lifecycle management through a declarative configuration, so you don't need to write Structured Streaming code directly. For sources without a managed connector, you can stream directly using Structured Streaming or Lakeflow pipelines.
Managed streaming connectors
| Connector | Description |
|---|---|
| Kafka | Stream data from Apache Kafka topics into streaming tables. |
| RabbitMQ | Stream messages from RabbitMQ classic queues into streaming tables. |
Stream from other sources
For streaming sources without a managed connector, including Amazon Kinesis, Google Pub/Sub, and Apache Pulsar, you can stream directly with Structured Streaming or Lakeflow pipelines. You can also read from Apache Kafka directly with Structured Streaming when you need finer control over how the stream is read and processed than the managed Kafka connector offers, such as custom offset handling or per-batch transformations. To choose a connector by source and level of pipeline customization, see Choose a standard connector.
Connector components
A streaming connector has the following components:
| Component | Description |
|---|---|
| Connection | A Unity Catalog securable object that stores the source endpoint and authentication credentials for your streaming source. The managed connector uses this connection to authenticate without requiring credentials in your pipeline configuration. |
| Ingestion pipeline | A pipeline that continuously reads messages from the streaming source and writes the results to streaming tables. The pipeline runs on serverless compute. |
| Destination tables | The streaming tables where the ingestion pipeline writes the data. |
