dimanche 3 mai 2015

S3 as input for Fluentd

I'm evaluating the use of Fluentd versus Logstash to process a wide variety of logs (syslog messages, Apache access logs, application logs, ELB, CloudFront, etc.). Something that works rather easily in Logstash is the import of logs from S3 - e.g I can parse ELB and CloudFront logs stored in my S3 bucket.

However, I haven't found a straight forward way of doing the same with Fluentd. I have a couple of log files that are stored on S3: ELB and CloudFront, as well as custom CSV files generated by the system. How can I periodically pull these logs from S3, process them with Fluentd and forward them to any other Fluentd output plugin?

So far I've found the elb-log plugin (http://ift.tt/1lyGnbL) which solves my problem specifically for ELB logs. There's also a CloudFront plugin which seems to pull its data from CloudWatch metrics. But how about other data stored on S3 such as my CSV files? Is this possible?




Aucun commentaire:

Enregistrer un commentaire