All Things Hadoop

Originally posted on https://www.linkedin.com/pulse/distributed-trace-systems-joe-stein?trk=pulse_spock-articles

For a while now folks have known about and have been using distributed trace within their infrastructure systems. If you aren’t familiar with the concept take a look at Google’s paper about it. Basically, it gives you macro level insight into how a distributed system behaves and allows you to optimize your “Data Center Apps” like Chrome Dev Tools lets you optimize your web apps.

Ok, so what do you need?

1) A Mesos Cluster. Sorry if you don’t have one of these yet, you should. Apache Mesos and Mesosphere’s DCOS have revolutionized how businesses utilize their compute resources. Instead of managing each resource individually, the technology unifies the compute in your data center into one centrally managed resource. Don’t worry if you are new to Mesos, hang tight and we will come back to it after discussing the other required elements.

2) Stack Deploy. This software  is one of the open source components of Elodina’s Sawfly Platform as a service product and is available athttps://github.com/elodina/stack-deploy. Stack deploy was developed to run and manage schedulers (Kafka, Cassandra, HDFS, etc) in orchestrated ways with other applications. These can be state-full or state-less (yes with Docker too but not required) and are intended to be deployed in a multi tenant and micro segmented software defined data center model.

3) Run This Stack!

https://github.com/elodina/stack-deploy/blob/master/stacks/zipkin-full.stack

The stack will deploy:

• 1x Exhibitor-Mesos
• 1x Exhibitor
• 1x DSE-Mesos
• 1x Cassandra node
• 1x Kafka-Mesos
• 1x Kafka 0.8 broker
• 1x Zipkin-Mesos
• 1x Zipkin Collector
• 1x Zipkin Query
• 1x Zipkin Web

Should you want to customize the number of Kafka brokers, Cassandra nodes, or Zipkin topic name, do so by modifying the corresponding fields (e.g. node count, cpu, mem, etc) in the stack file.

In order to deploy this stack the only thing that needs to be specified is the zookeeper url:

export SD_API=stackdeploy_url>
# stackdeploy_user would most be probably “admin”
export SD_USER=stackdeploy_user>
export SD_KEY=stackdeploy_key>

./stack-deploy add –file zipkin-full.stack
# for “Mesos Consul”-enabled clusters zk_url would probably be zookeeper.service:2181
./stack-deploy run zipkin-full –var “zk_url=zk_url>”

Note: if you want to restart your stack for some reason, make sure to clean up theExhibitor Mesos framework znode storage in Zookeeper if the Exhibitor phase was already completed. Failure to do this step may cause subsequent deployments of the stack to fail.

Launching from existing infrastructure

If you want to include Zipkin in already running infrastructure, use the following file: zipkin-standalone.stack

In order to zipkin stand alone, you will need a Cassandra cluster and 0.8 Kafka brokers accesible from your clusterfor Zipkin to use.

# Presuming you have SD_API, SD_USER and SD_KEY already set

./stack-deploy add –file stacks/zipkin-standalone.stack
./stack-deploy run zipkin-standalone –var “kafka_zk_url=kafka_zk_url>” –var “cassandra_url=cassandra_contact_points>”

Launching from existing infrastructure

If you want to include Zipkin in already running infrastructure, use the following file: zipkin-standalone.stack

In order to run, you will need a Cassandra node and 0.8 Kafka broker on your cluster which you get in the full stack or use the setup you have is ok too.

Is it working?

The Easiest way to verify if the Zipkin stack has launched successfully is to use the trace generator provided by Zipkin Mesos framework. Please refer to the corresponding documentation section on details:https://github.com/elodina/zipkin-mesos-framework#verifying-all-components-running

After the traces aresent, you can check them out by opening Zipkin web UI in your browser. If you plan on using  stack file on Consul-enabled clusters, the link will look like this:

http://zipkin-web-0.service.{cluster_name}:31983

And you are good to go!

If you need a client example how to generate these trace events, take a look at our Go Zipkin sample for that utility here https://github.com/elodina/go-zipkin.

To learn more about Elodina please contact us or come meet us in person at theApache Mesos NYC or Apache Kafka NYC Meetups about getting started.

~ Joestein