Home > Open Source Projects > Apache Kafka HTTP Producer and Consumer

Apache Kafka HTTP Producer and Consumer

Big Data Open Source Security LLC has released an Apache Kafka HTTP endpoint for producing and consuming data with Apache Kafka brokers https://github.com/stealthly/dropwizard-kafka-http.

This project creates an Apache Kafka HTTP endpoint for producing and consuming messages.

Running Project

1) Install Vagrant (http://www.vagrantup.com/)
2) Install Virtual Box (https://www.virtualbox.org/)
3) git clone https://github.com/stealthly/dropwizard-kafka-http
4) cd dropwizard-kafka-http
5) vagrant up

Then produce and consume messages over HTTP e.g.

curl -d "topic=http&message=hello&key=0" "http://192.168.22.10:8080/message"

and

curl "http://192.168.22.10:8080/message?topic=http"

Your original message produced from the consumer:

[{"topic":"http","key":"0","message":"hello","partition":0,"offset":0}]

* Zookeeper will be running on 192.168.22.5
* dropwizard-kafka-http is built cleanly before Zookeeper installs in `vagrant/zk.sh`
* Broker One is running on 192.168.22.10
* dropwizard-kafka-http launches on 192.168.22.10 after the Kafka Broker starts in `vagrant/broker.sh`

If you want you can login to the machines using `vagrant ssh zookeeper` and `vagrant ssh brokerOne`.

API Methods

Produce messages

POST /message?topic=$topic&async={true|false}&message=$message0&key=$key0,
message=$message1,key=$key1, …

Produces messages on topic.

Parameters:
topic – required topic name
async – optional true|false value indicating should the producer be async
key – required key text. Multiple values are possible
message – required message text. Multiple values are possible

Note: passed keys count should match messages count.

Errors:
400 – wrong parameters passed

Consume messages

GET /message?topic=$topic&timeout=$timeout
Returns consumed messages available on topic.

Parameters:
topic – required topic name
timeout – optional timeout in ms

Errors:
400 – wrong parameters passed

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 Twitter: @allthingshadoop
********************************************/
Advertisements
Categories: Open Source Projects
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: