Category: ELK

Logstash Pipeline Architecture

Logstash Pipeline Architecture


How to check socket connection ?

How to check socket connection between filebeat, logstash and elasticseearch ?

netstat -anp | grep 9200
netstat -anp | grep 5044

a – Show all listening and non-listening sockets
n – numberical address
p – process id and name that socket belongs to

9200 – Elasticsearch port
5044 – Filebeat port

ESTABLISHED” status for the sockets that established connection between logstash and elasticseearch / filebeat.

LISTEN” status for the sockets that listening for incoming connections.

To view the count of socket, use

 wc -l


logstash – field conditional expression

Check if the field myToken exists

filter { 
if [myToken] {  
##my program goes here 

Check if the field [myCategory][myToken] exists

filter {  
if [myCategory] { 
   if [myCategory][myToken] {
      ##my program goes here

Check if the field myToken NOT exists

filter { 
if ![myToken] {
  ##my program goes here

To check if the field myToken is empty

if [myToken] !~ /.+/ {
  ## my code goes here
  ##drop { } 


Reference From Elastic:

To check if field foo exists:
1) For numeric type fields use:

 if ([foo]) {    ... }

2) For types other than numeric like boolean, string use:

if ("" in [foo]) {    ...}
filter {  
if [foo] in [foobar] {
    mutate {
 add_tag => "field in field"
 }  }

if [foo] in "foo" {
    mutate { add_tag => "field in string" }

if "hello" in [greeting] {
    mutate { add_tag => "string in field" }  }

if [foo] in ["hello", "world", "foo"] {
    mutate { add_tag => "field in list" }  }

if [missing] in [alsomissing] {
    mutate { add_tag => "shouldnotexist" }  }

if !("foo" in ["hello", "world"]) {
    mutate { add_tag => "shouldexist" } 


How to install npm and nodejs in offline on linux ?

How to install npm and nodejs in offline on linux ?

Download the latest Linux binaries from the site

Extract it TAR file to the desired location.

cd /opt/mohan/
tar -xvf node-v6.11.0-linux-x64.tar.gz

set the soft link to npm and node

ln -s /opt/mohan/node-v6.10.3-linux-x64/bin/node /usr/bin/node
ln -s /opt/mohan/node-v6.10.3-linux-x64/bin/npm /usr/bin/npm

Check the npm  and node version

npm -v
node -v



Elasticdump is the import and export tool for Elasticsearch indexes.

How to install elasticdump and how to copy elasticsearch index?

Install npm and node if not installed already.

Set the proxy if required as below.

npm config set proxy
npm config set http_proxy=

Install the elasticdump as below.

npm install elasticdump -g

-g is the global mode. It means it installs the current working directory as a global package directory.

Set the soft link to elasticdump

ln -s /opt/mohan/node-v6.10.3-linux-x64/bin/elasticdump /usr/bin/elasticdump

Go to the help to check all the commands in the elasticdump.

elasticdump --help


To export to a file:

elasticdump --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json --type=data

elasticdump --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json --type=mapping

if elasticsearch is secured with certs, then use NODE_TLS_REJECT_UNAUTHORIZED as 0 in the beginning of the command.

# NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json

if elasticsearch authorized with shield then, use –httpAuthFile option. Create a file with below user and password.



# NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --httpAuthFile=/opt/mohan/myAuth.file --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json

If need to zip the output file, use as below.

# NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --httpAuthFile=/opt/mohan/myAuth.file --input=http://localhost:9200/mohan-index-2017.05.* --output=$ | gzip > /opt/mohan/mydata.json.gz



elasticsearch shield esusers

Elasticsearch Shield esusers management commands

esusers list
esusers list username
esusers useradd username
esusers useradd username -p secret
esusers useradd username -r comma,separated,list,of,role,names
esusers passwd username
esusers passwd username -p password
esusers roles username -a comma,separated,list,of,roles -r comma,separated,list,of,roles
esusers userdel username

ELK – Watcher Commands

GET _watcher/watch/<watch_id>
DELETE _watcher/watch/my-watch
PUT _watcher/watch/my-watch?active=false
PUT _watcher/watch/<watch_id>/_activate
PUT _watcher/watch/<watch_id>/_deactivate
GET _watcher/stats
GET _watcher
PUT _watcher/_stop
PUT _watcher/_start
GET _watcher/stats/queued_watches
GET _watcher/stats/current_watches
GET _watcher/stats?metric=executing_watches
PUT _watcher/watch/my-watch/_ack?master_timeout=30s
DELETE _watcher/watch/my-watch?master_timeout=30s
PUT _watcher/watch/my-watch/my-action/_ack
PUT _watcher/watch/my-watch/_ack
PUT _watcher/watch/my-watch/action1,action2/_ack