Nifty Tool To Export Files From HDFS Into MySQL
This is an interesting open source project I have recently heard about http://code.google.com/p/hiho/.
What is very interesting to me about this project is the export utility which takes data from HDFS and loads it into MySQL.
It also has a nice way for querying and importing data from a JDBC database directly into HDFS. It looks much more robust than the out of the box DBInputFormat that Hadoop provides. You can import the data as delimited records, with choice of delimiter. You can also import the data and save them as Avro records. It supports queries – you can say join two tables. It splits on user specified column ranges, instead of using LIMIT and OFFSET. It does no code generation or ORM mapping.
There are other ETL tools out there (e.g. Sqoop http://www.cloudera.com/developers/downloads/sqoop/). In Cloudera’s Distrobution for Hadoop Version 3 (CDH3) Sqoop supports HDFS back into MySQL also now.
I am definitely going to have to give this utility a try. I here from their (HIHO) project folks that the next to-do is support for more databases for export.
[tweetmeme http://wp.me/pTu1i-q%5D
/*
Joe Stein
http://www.linkedin.com/in/charmalloc/
*/
Leave a Reply
All Things Hadoop Podcast
- Tips And Tricks To Your First Cluster
- Hadoop, BigData and Cassandra with Jonathan Ellis
- Ruby Streaming with Wukong, Guest Flip Kromer
- Hadoop and Pig with Alan Gates from Yahoo
- Hadoop Development Tools with Shevek from Karmasphere
- NoSQL HBase and Hadoop with Todd Lipcon from Cloudera
- Unified analytics and large scale machine learning with Milind Bhandarkar
- Hortonworks HDP1, Apache Hadoop 2.0, NextGen MapReduce (YARN), HDFS Federation and the future of Hadoop with Arun C. Murthy
- Hadoop, Mesos, Cascading, Scalding, Cascalog and Data Science with Paco Nathan
- Hadoop, The Cloudera Development Kit, Parquet, Apache BigTop and more with Tom White
All Things Hadoop 
As a brief update, Sqoop now includes export support (HDFS-to-RDBMS). The version of Sqoop shipped with Cloudera’s Distribution for Hadoop 3 (CDH3) includes this functionality.
– Aaron
Thanks Aaron, I updated the post appropriately.
HIHO for Hadoop sucks. Don’t bother trying it out. There are better tools available for HDFS-TO-RDBMS.
HIHO sucks. Don’t bother trying it. There are much better tools available
HIHO is a piece of Junk.Sqoop is much better.