Comments on: Hadoop Analysis of Apache Logs Using Flume-NG, Hive and Pig

By: lilufmj

lilufmj — Wed, 16 Jan 2013 16:12:34 +0000

|
|
|

By: lilulzm

lilulzm — Wed, 16 Jan 2013 14:57:52 +0000

|
|
|

By: Kristoffer

Kristoffer — Thu, 10 Jan 2013 16:28:52 +0000

I would recommend taking a look at ELSA aswell,:
http://enterprise-log-search-and-archive.googlecode.com/
Example for using ELSA to parse bro ids log files: http://blog.bro-ids.org/2012/01/monster-logs.html

By: benr

benr — Sat, 29 Dec 2012 05:13:08 +0000

There is a lot of good info out there, such as this Cloudera post (http://blog.cloudera.com/blog/2009/02/the-small-files-problem/). I’m not making a judgement call here as to whether lots of files are good or bad, but by default Flume creates a lot and therefore something to think about.

As for the realistic nature of Logstash/etc versus Hadoop… in the real world, for most shops, you likely should be sending data into an ElasticSearch like solution (Splunk, Graylog2, Logstash, etc) for a period of 2-4 weeks and then archiving the logs out to Hadoop for data warehousing.

By: benr

benr — Sat, 29 Dec 2012 05:05:28 +0000

I’d add to that SplunkStorm and SumoLogic, which I think are the leading options in the space. I really should do a blog post on SaaS Logging solutions.

Remember that Splunk is free for less than 500MB per day, which is enough for many small shops. And Splunk is an incredibly kool company and very supportive of SA’s, so despite the big bucks they ask (and get) for their software, I have nothin’ but love for Splunk.

By: bryan w. berry

bryan w. berry — Fri, 28 Dec 2012 06:42:04 +0000

thanks for the great article benr

” you end up with a LOT of HDFS files, which may or may not be what you want. Setting any value to 0 disables that type of rolling.”

If I understand correctly the default block size for HDFS is 64 MB. Lots of small files would cause incredible waste. When would you want lots of small files?

re: logstash vs. flume/hadoop/hive/pig, I could see the hadoop solution being useful over logstash if you have non-sysadmins who are familliar w/ pig/hive and want to use those skills to interact w/ the webserver logs.

By: gazarsgo

gazarsgo — Fri, 28 Dec 2012 04:27:30 +0000

both papertrailapp.com and loggly.com are agent-less alternatives to Splunk — I assume I can’t afford Splunk since they don’t advertise their pricing.