In this tutorial we will learn how access an Apache Kafka cluster running on OpenShift / Kubernetes using the Kafka Bridge.
The Apache Kafka Bridge includes a RESTful interface that let HTTP-based clients to interact with a Kafka cluster. This can simplify accessing the Kafka cluster which might be running on a Cloud Environment such as Kubernates or OpenShift.
In order to complete this tutorial, install the Strimzi Operator, the Kafka cluster and a Topic as discussed in this tutorial: Apache Kafka on OpenShift quickstart guide
Now that the Strimzi Operator has been installed, let’s create a Kafka Bridge. Using your editor, create this yaml file (available at: https://github.com/fmarchioni/masteringintegration/blob/master/kafka/openshift/kafka-bridge.yaml)
apiVersion: kafka.strimzi.io/v1alpha1 kind: KafkaBridge metadata: name: my-bridge namespace: kafka-demo spec: replicas: 1 bootstrapServers: 'my-cluster-kafka-bootstrap:9092' http: port: 8080
This file will create a Kafka Bridge which points to the Kafka cluster via the bootstrap servers available at: ‘my-cluster-kafka-bootstrap:9092’
oc create -f kafka-bridge.yaml Verify that the service has been created:
oc get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-bridge-bridge-service ClusterIP 172.30.43.5 <none> 8080/TCP 12s my-kafka-kafka-bootstrap ClusterIP 172.30.4.52 <none> 9091/TCP,9092/TCP,9093/TCP 3h2m my-kafka-kafka-brokers ClusterIP None <none> 9091/TCP,9092/TCP,9093/TCP 3h2m my-kafka-zookeeper-client ClusterIP 172.30.39.12 <none> 2181/TCP 3h3m my-kafka-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 3h3m Next, expose the service through a Route:
oc expose service my-bridge-bridge-service Check that the Route is available:
oc get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD my-bridge-bridge-service my-bridge-bridge-service-kafka-demo.apps-crc.testing my-bridge-bridge-service rest-api Create a Kafka Producer
In order to create a Kafka producer using the REST API, we will send a JSON formatted POST to the Router address which is available outside of OpenShift:
curl -X POST http://my-bridge-bridge-service-kafka-demo.apps-crc.testing/topics/demo-queue -H 'content-type: application/vnd.kafka.json.v2+json' -d '{ "records": [ { "key": "my-key1", "value": "value-0001" }, { "key": "my-key2", "value": "value-0002" } ] }' Provided that you have created the Topic “demo-queue”, the above command will complete successfully.
Creating a Kafka Consumer
To create a Kafka Consumer using the REST API is a three step procedure. At first, you need to define a Consumer Group:
curl -X POST http://my-bridge-bridge-service-kafka-demo.apps-crc.testing/consumers/bridge-quickstart-consumer-group -H 'content-type: application/vnd.kafka.v2+json' -d '{ "name": "bridge-quickstart-consumer", "auto.offset.reset": "earliest", "format": "json", "enable.auto.commit": false, "fetch.min.bytes": 512, "consumer.request.timeout.ms": 30000 }' Next, attach the Consumer Group to a Topic, for example the ‘demo-queue’:
curl -X POST http://my-bridge-bridge-service-kafka-demo.apps-crc.testing/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer/subscription -H 'content-type: application/vnd.kafka.v2+json' -d '{ "topics": [ "demo-queue" ] }' Finally, you can retrieve messages by sending a GET to the Consumer Group which we have defined:
curl -X GET http://my-bridge-bridge-service-kafka-demo.apps-crc.testing/consumers/bridge-quickstart-consumer-group/instances/bridge-quickstart-consumer/records -H 'accept: application/vnd.kafka.json.v2+json' After creating and subscribing to a Kafka Bridge consumer, a first GET request will return an empty response because the poll operation starts a rebalancing process to assign partitions.
Repeat step two to retrieve messages from the Kafka Bridge consumer.
[{"topic":"demo-queue","key":"my-key1","value":"value-0001","partition":0,"offset":8},{"topic":"demo-queue","key":"my-key2","value":"value-0002","partition":0,"offset":9}]
Congratulations! You have managed to connect to a Kafka cluster on OpenShift using the Bridge API. For more details on the Bridge REST API check https://strimzi.io/docs/bridge/latest/
Found the article helpful? if so please follow us on Socials
