mercredi 5 août 2015

logstash cloudfront codec plugin: Error: Object: #Version: 1.0 is not a legal argument to this wrapper, cause it doesn't respond to "read"

Logstash version 1.5.0.1

I am trying to use the logstash s3 input plugin to download cloudfront logs and the cloudfront codec plugin to filter the stream.

I installed the cloudfront codec with bin/plugin install logstash-codec-cloudfront.

I am getting the following: Error: Object: #Version: 1.0 is not a legal argument to this wrapper, cause it doesn't respond to "read".

Here is the full error message from /var/logs/logstash/logstash.log

 {:timestamp=>"2015-08-05T13:35:20.809000-0400", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n  Plugin: <LogStash::Inputs::S3 bucket=>\"[BUCKETNAME]\", prefix=>\"cloudfront/\", region=>\"us-east-1\", type=>\"cloudfront\", secret_access_key=>\"[SECRETKEY]/1\", access_key_id=>\"[KEYID]\", sincedb_path=>\"/opt/logstash_input/s3/cloudfront/sincedb\", backup_to_dir=>\"/opt/logstash_input/s3/cloudfront/backup\", temporary_directory=>\"/var/lib/logstash/logstash\">\n  Error: Object: #Version: 1.0\n is not a legal argument to this wrapper, cause it doesn't respond to \"read\".", :level=>:error}

My logstash config file: /etc/logstash/conf.d/cloudfront.conf

input {
  s3 {
    bucket => "[BUCKETNAME]"
    delete => false
    interval => 60 # seconds
    prefix => "cloudfront/"
    region => "us-east-1"
    type => "cloudfront"
    codec => "cloudfront"
    secret_access_key => "[SECRETKEY]"
    access_key_id => "[KEYID]"
    sincedb_path => "/opt/logstash_input/s3/cloudfront/sincedb"
    backup_to_dir => "/opt/logstash_input/s3/cloudfront/backup"
    use_ssl => true
  }
}

I'm using a similar s3 input stream successfully to get my cloudtrail logs into logstash that is based on the Answer from a stackoverflow post.

CloudFront logfile from s3 (I only included the header from the file):

 #Version: 1.0
 #Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type

The header looks like it is basically the correct format based on lines 26-29 from the cloudfront plugin github repo cloudfront_spec.rb and the official AWS CloudFront Access Logs docs.

Any ideas? Thanks!




Aucun commentaire:

Enregistrer un commentaire