Home > ETL, Open Source Projects > Nifty Tool To Export Files From HDFS Into MySQL

Nifty Tool To Export Files From HDFS Into MySQL

This is an interesting open source project I have recently heard about http://code.google.com/p/hiho/.

What is very interesting to me about this project is the export utility which takes data from HDFS and loads it into MySQL.

It also has a nice way for querying and importing data from a JDBC database directly into HDFS.  It looks much more robust than the out of the box DBInputFormat that Hadoop provides.  You can import the data as delimited records, with choice of delimiter. You can also import the data and save them as Avro records. It supports queries – you can say join two tables. It splits on user specified column ranges, instead of using LIMIT and OFFSET. It does no code generation or ORM mapping.

There are other ETL tools out there (e.g. Sqoop http://www.cloudera.com/developers/downloads/sqoop/).  In Cloudera’s Distrobution for Hadoop Version 3 (CDH3) Sqoop supports HDFS back into MySQL also now.

I am definitely going to have to give this utility a try. I here from their (HIHO) project folks that the next to-do is support for more databases for export.

[tweetmeme http://wp.me/pTu1i-q%5D

Joe Stein

  1. April 26, 2010 at 1:08 am

    As a brief update, Sqoop now includes export support (HDFS-to-RDBMS). The version of Sqoop shipped with Cloudera’s Distribution for Hadoop 3 (CDH3) includes this functionality.

    – Aaron

  2. April 26, 2010 at 7:01 am

    Thanks Aaron, I updated the post appropriately.

  3. Hadoop Fan
    May 6, 2010 at 12:46 am

    HIHO for Hadoop sucks. Don’t bother trying it out. There are better tools available for HDFS-TO-RDBMS.

  4. Hadoop Fan
    May 6, 2010 at 12:25 pm

    HIHO sucks. Don’t bother trying it. There are much better tools available

  5. Hadoop
    December 2, 2010 at 4:39 pm

    HIHO is a piece of Junk.Sqoop is much better.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: