How to check socket connection ?

How to check socket connection between filebeat, logstash and elasticseearch ?

netstat -anp | grep 9200
netstat -anp | grep 5044

a – Show all listening and non-listening sockets
n – numberical address
p – process id and name that socket belongs to

9200 – Elasticsearch port
5044 – Filebeat port

ESTABLISHED” status for the sockets that established connection between logstash and elasticseearch / filebeat.

LISTEN” status for the sockets that listening for incoming connections.

To view the count of socket, use

 wc -l

command

logstash – field conditional expression

Check if the field myToken exists

filter { 
if [myToken] {  
##my program goes here 
}
}

Check if the field [myCategory][myToken] exists

filter {  
if [myCategory] { 
   if [myCategory][myToken] {
      ##my program goes here
    }
  }
}

Check if the field myToken NOT exists

filter { 
if ![myToken] {
  ##my program goes here
 }
}

To check if the field myToken is empty

if [myToken] !~ /.+/ {
  ## my code goes here
  ##drop { } 
}

 

Reference From Elastic:

To check if field foo exists:
1) For numeric type fields use:

 if ([foo]) {    ... }

2) For types other than numeric like boolean, string use:

if ("" in [foo]) {    ...}
filter {  
if [foo] in [foobar] {
    mutate {
 add_tag => "field in field"
 }  }

if [foo] in "foo" {
    mutate { add_tag => "field in string" }
  }  

if "hello" in [greeting] {
    mutate { add_tag => "string in field" }  }

if [foo] in ["hello", "world", "foo"] {
    mutate { add_tag => "field in list" }  }

if [missing] in [alsomissing] {
    mutate { add_tag => "shouldnotexist" }  }

if !("foo" in ["hello", "world"]) {
    mutate { add_tag => "shouldexist" } 
 }}

 

logstash test config

Logstash Test Config File

##/opt/mohan/mytest.conf
input{
 stdin {}
}
filter {
## You can drive here
# My Test 
 json {
  source => "message"
 }
}
output {
 stdout {
  codec => rubydebug
 }
}

How to execute?

/opt/logstash241/bin/logstash -f /opt/mohan/mytest.conf < mydata.json

 

logstash expression

if EXPRESSION {

} else if EXPRESSION {

} else {

}

EXPRESSION are Comparison tests, boolean logic, etc.

Comparison operators:
equality: ==, !=, <, >, <=, >=
regexp: =~, !~ (checks a pattern on the right against a string value on the left)
inclusion: in, not in

boolean operators:
and, or, nand, xor

unary operator:
!

group with parentheses (…)

Any “value” contains in “tags”:

if “value” in [tags] {
}

Check String Field:

if [myFieldName] =~ /.+/ {
# exists
}

if [myFieldName] !~ /.+/ {
# doesn’t exist
}

Check Number Field:

if [myFieldName] {
# exists
}

 

how to test logstash config

How do test logstash config?

$ cat test.config
input {
stdin { }
}
output {
stdout {
codec => rubydebug }
}
filter {
date {
match => [“message”, “dd/MM/YYYY HH.mm.ss”]
}
}

$ echo ’12/28/2016 10.50.11′ | /opt/logstash/bin/logstash -f test.config
{
“message” => “12/28/2016 10.50.11”,
“@version” => “1”,
“@timestamp” => “2016-12-28T10:50:11.000Z”,
“host” => “mydearhost”
}

Logstash – Filter

Contains:

if "foo" in [tags] {
 ...
}

Mapping Parsing Exception:
this has been mapped as some data type and how you're suddenly sending something else, hence mapping_parsing_exception.