Using GeoIP with Logstash and ElasticSearch

GeoIP is a filter in Logstash that can collect information about the geographical location of the IP Address.

This tutorial is using Elasticsearch 6.8.0. The filter configuration of GeoIP:

filter {
  if "apache" in [tags] {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    geoip {
      source => "clientip"
    }
  }
}

For example, I get apache logs from the filebeat. I can parse the logs via grok and get a variable name clientip which is the source IP from packet accessing the apache. GeoIP will take the clientip variable and create another variable related to geographical information.

But before you change the configuration in logstash (by restarting it), you need to do some work on Elasticsearch by creating a template.

Why we need a template? If you want to create a map visualization in kibana, the variable needs to convert to geo_point. I am using dev tools in Kibana for simplicity

PUT _template/YOUR-TEMPLATE-NAME
{   
    "index_patterns": ["apache_access_node_A"],
    "mappings": {
        "_default_": {
            "properties": {
                "geoip": {
                    "dynamic": true,
                    "properties": {
                        "ip": {
                            "type": "ip"
                        },
                        "latitude": {
                            "type": "half_float"
                        },
                        "location": {
                            "type": "geo_point"
                        },
                        "longitude": {
                            "type": "half_float"
                        }
                    }
                }
            }
        }
    }
}

For example, my filebeat index created in elasticsearch will be apache_access_node_A_2019_11 and my index pattern will be apache_access_node_A_* (you can create a template with whatever name you want.

Leave a Comment

Your email address will not be published.