Skip to content
Talk to us
Login

Integrating Patchworks to Apache Kafka

In today's data-driven world, connecting disparate systems efficiently is paramount. Patchworks excels at iPaaS (Integration Platform as a Service), seamlessly linking e-commerce platforms, ERPs, and other business-critical applications. Apache Kafka, on the other hand, is the gold standard for real-time, high-throughput event streaming.

Combining them creates a powerful, scalable, and resilient data pipeline and it’s a common request from large enterprises and the global system integrators. When integrated, you can use Patchworks' intuitive interface to manage complex transformations and business logic, while leveraging Kafka as the central nervous system for your data streams. This blog explores several methods for integrating these two powerhouses using Apache Kafka Connect, a framework for connecting Kafka with external systems.

Method 1: The Universal Approach with HTTP

The most obvious way to connect Patchworks and Kafka is via HTTP and RESTful web services, using a generic Kafka HTTP Connector such as those available from Confluent. This method is universal and can be adapted for many scenarios.

HTTP Source: Patchworks to Kafka

As demonstrated in this video

you can easily configure a Patchworks process flow to post data to Kafka via a secure endpoint.

  1. Build a connector and Endpoint in Patchworks: Use the Patchworks Connector Builder to create a new Kafka connector. Configure a flow that gathers the required data (e.g., new orders, customer updates) and exposes it via the REST API endpoint.

  2. Configure Kafka's HTTP Source Connector: Deploy an HTTP Source Connector (like the Confluent HTTP Source Connector) to your Kafka Connect cluster listen for POSTED messages.

  3. Data Flow: Patchworks runs the flow based on the Trigger shape (periodically, based on being called externally or from another flow or from an event queue or webhook) and posts the data to the Kafka topic.

HTTP Sink: Kafka to Patchworks

To send data from a Kafka topic to Patchworks, you simply reverse the flow.

  1. Build a Sink Endpoint in Patchworks: Create a process in Patchworks that can be triggered by a webhook or API call. This process will receive a payload and execute a business workflow (e.g., create a customer in an ERP).

  2. Configure Kafka's HTTP Sink Connector: Deploy an HTTP Sink Connector. Configure it to subscribe to a Kafka topic. For each message on the topic, the connector will make a POST request to the Patchworks webhook or API endpoint, passing the message payload in the request body.

Method 2: File-Based Integration with SFTP

For systems that rely on batch file processing, integrating via SFTP is a robust solution.

  • Source: Use the Patchworks SFTP Connector to schedule a process that generates a data file (e.g., a CSV of daily sales) and places it on an SFTP server. Configure the Kafka Connect FilePulse or a similar SFTP Source Connector to monitor that directory, read new files, and stream their contents into a Kafka topic.

  • Sink: Configure a Kafka Connect SFTP Sink Connector to write messages from a topic into a file on the SFTP server. A scheduled Patchworks process can then use its SFTP Connector to pick up this file and process its contents.

Method 3: Database Integration with the JDBC Connector

Direct database integration is a powerful way to capture data changes.

  • Source: Use the Patchworks Database Connector to write data into a specific table in a staging database. Configure the Kafka Connect JDBC Source Connector to monitor this table for new or updated rows (using a timestamp or an incrementing ID column). The connector will publish each change as an event to a Kafka topic.

  • Sink: The JDBC Sink Connector can consume messages from a Kafka topic and insert/update rows in a database table. A Patchworks process can then periodically query this table to fetch and process the new data.

Method 4: Cloud Storage with Amazon S3

For cloud-native architectures, Amazon S3 provides a scalable intermediate storage layer.

  • Source: A Patchworks flow can use the Patchworks S3 Connector to save data as objects in an S3 bucket. The Kafka Connect S3 Source Connector monitors the bucket and streams any new object data into a Kafka topic.

  • Sink: The S3 Sink Connector subscribes to Kafka topics and writes the messages as objects into an S3 bucket. Patchworks can then be configured to poll the database for new messages.

Method 5: Message Queues with RabbitMQ

If you're already using RabbitMQ, you can bridge it with Kafka.

  • Source: Configure a Patchworks process to use the Patchworks Event Connector to publish messages to a RabbitMQ exchange. You can then use a RabbitMQ Source Connector for Kafka to consume messages from the RabbitMQ queue and pipe them into a Kafka topic.

  • Sink: Similarly, a RabbitMQ Sink Connector can take messages from Kafka and publish them to a RabbitMQ exchange, where a waiting Patchworks consumer can process them.

Method 6: The Ultimate Control with a Custom Patchworks Connector

For the tightest integration, you can build a custom Java connector specifically for Patchworks within the Kafka Connect framework. This gives you complete control over authentication, data mapping, and error handling.

Custom Source Connector: Polling a Patchworks Flow

A custom source connector can be built to periodically call a Patchworks flow and pull data directly.

  • How it Works - Method 1: The connector's poll() method would be implemented to make an authenticated API call to a specific Patchworks endpoint. This endpoint would trigger a flow that gathers and returns a batch of data. The connector then transforms this data into SourceRecord objects and sends them to Kafka. This is ideal for fetching data from systems that don't support webhooks.

  • How it Works - Method 2: Expose an external API call (as per the HTTP Connector method) for a Patchworks flow to call via an API call using Connector Builder

Custom Sink Connector: Triggering a Patchworks Flow

A custom sink connector subscribes to a topic and triggers a Patchworks flow for each message.

  • How it Works: The connector's put() method would receive a collection of SinkRecord objects from Kafka. For each record, it would extract the payload, connect to the Patchworks API, and call a specific flow, passing the payload. This provides a highly efficient, real-time mechanism for pushing events from Kafka directly into your Patchworks business workflows.

Patchworks has an example open source Patchworks Kafka Connector available for use by customers under the Apache Software License as demonstrated here:

 

Want to learn more? Contact Us