Logstash multiple input output

When you have multiple input and want to create multiple output based on index, you cannot using default config in Logstash.

Logstash only process single pipeline processing, so if you have 2 configuration file like this (this example use filebeat and NetFlow):

input {
  beats {
    port => 5044
    type => filebeat
  }
}

output {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      manage_template => false
      index => "%{[beat][hostname]}_%{+YYYY.MM}"
    }
}

and other file like this:

input {
  udp {
    port  => 2055
    codec => netflow
  }
}

output {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      manage_template => false
      index => "netflow_%{+YYYY.MM}"
    }
}

Logstash will send all input data into output and the data that should be separated, combine into the output. To separate the data, we can use tags or type mechanism.

input {
  beats {
    port => 5044
    type => filebeat
    tags => "filebeat-input"
  }

  udp {
    port  => 2055
    codec => netflow
    tags => "netflow-input"
  }
}

output {

  if "filebeat-input" in [tags] {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      manage_template => false
      index => "%{[beat][hostname]}_%{+YYYY.MM}"
    }
  }

  else if "netflow-input" in [tags] {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      manage_template => false
      index => "netflow_%{+YYYY.MM}"
    }
  }
}

This configuration have tags the input, and doing selection based on tags in the output.

Leave a Comment

Your email address will not be published.