Category: Techie

Logstash Pipeline Architecture

Logstash Pipeline Architecture


RabbitMQ Cluster Setup


RabbitMQ Queue

rabbitmq-plugins enable rabbitmq_management
service rabbitmq_server restart
rabbitmqadmin declare queue name=”mohantest-queue”
rabbitmqadmin declare exchange name=”mohantest-direct-exchange” type=”direct”
rabbitmqadmin declare binding source=”mohantest-direct-exchange” destination_type=”queue” destination=”mohantest-queue” routing_key=”mohantest-dxqb-key”
rabbitmq-plugins disable rabbitmq_management
service rabbitmq_server restart

RabbitMQ configuration

[ {rabbit, [{tcp_listeners, [{“”, 5672}]}]}]

rabbitmqctl stop_app
rabbitmqctl start_app
RabbitMQ Cluster Setup

Step A: The cookie .erlang.cookie is to be the same for all nodes which is to be part of cluster.
1. Stop RabbitMQ and Erlang instance
service rabbitmq-server stop
epmd -kill
2. Make same copy of .erlang.cookie to all nodes
3. Start the RabbitMQ
service rabbitmq-server start
Step B: Join rabbit@rabbit2 with rabbit@rabbit1
NodeName: rabbit2
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@rabbit1
rabbitmqctl start_app

Step C: Join rabbit@rabbit3 with rabbit@rabbit2
NodeName: rabbit3
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@rabbit2
rabbitmqctl start_app

Step D: Check the cluster status and verify rabbit1, rabbit2 and rabbit3 are clustered together.
rabbitmqctl cluster_status

if need to detach the node from the cluster:

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app

RabbitMQ HA Mode

rabbitmqctl set_policy ha-mohantest “^(mohantest-queue$)” ‘{“ha-mode”:”all”, “ha-sync-mode”:”automatic”}’ -p /
rabbitmqctl list_policies
rabbitmqctl list_connections

To remove policy:
rabbitmqctl clear_policy ha-mohantest

How to check socket connection ?

How to check socket connection between filebeat, logstash and elasticseearch ?

netstat -anp | grep 9200
netstat -anp | grep 5044

a – Show all listening and non-listening sockets
n – numberical address
p – process id and name that socket belongs to

9200 – Elasticsearch port
5044 – Filebeat port

ESTABLISHED” status for the sockets that established connection between logstash and elasticseearch / filebeat.

LISTEN” status for the sockets that listening for incoming connections.

To view the count of socket, use

 wc -l


logstash – field conditional expression

Check if the field myToken exists

filter { 
if [myToken] {  
##my program goes here 

Check if the field [myCategory][myToken] exists

filter {  
if [myCategory] { 
   if [myCategory][myToken] {
      ##my program goes here

Check if the field myToken NOT exists

filter { 
if ![myToken] {
  ##my program goes here

To check if the field myToken is empty

if [myToken] !~ /.+/ {
  ## my code goes here
  ##drop { } 


Reference From Elastic:

To check if field foo exists:
1) For numeric type fields use:

 if ([foo]) {    ... }

2) For types other than numeric like boolean, string use:

if ("" in [foo]) {    ...}
filter {  
if [foo] in [foobar] {
    mutate {
 add_tag => "field in field"
 }  }

if [foo] in "foo" {
    mutate { add_tag => "field in string" }

if "hello" in [greeting] {
    mutate { add_tag => "string in field" }  }

if [foo] in ["hello", "world", "foo"] {
    mutate { add_tag => "field in list" }  }

if [missing] in [alsomissing] {
    mutate { add_tag => "shouldnotexist" }  }

if !("foo" in ["hello", "world"]) {
    mutate { add_tag => "shouldexist" } 


How to install npm and nodejs in offline on linux ?

How to install npm and nodejs in offline on linux ?

Download the latest Linux binaries from the site

Extract it TAR file to the desired location.

cd /opt/mohan/
tar -xvf node-v6.11.0-linux-x64.tar.gz

set the soft link to npm and node

ln -s /opt/mohan/node-v6.10.3-linux-x64/bin/node /usr/bin/node
ln -s /opt/mohan/node-v6.10.3-linux-x64/bin/npm /usr/bin/npm

Check the npm  and node version

npm -v
node -v



Elasticdump is the import and export tool for Elasticsearch indexes.

How to install elasticdump and how to copy elasticsearch index?

Install npm and node if not installed already.

Set the proxy if required as below.

npm config set proxy
npm config set http_proxy=

Install the elasticdump as below.

npm install elasticdump -g

-g is the global mode. It means it installs the current working directory as a global package directory.

Set the soft link to elasticdump

ln -s /opt/mohan/node-v6.10.3-linux-x64/bin/elasticdump /usr/bin/elasticdump

Go to the help to check all the commands in the elasticdump.

elasticdump --help


To export to a file:

elasticdump --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json --type=data

elasticdump --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json --type=mapping

if elasticsearch is secured with certs, then use NODE_TLS_REJECT_UNAUTHORIZED as 0 in the beginning of the command.

# NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json

if elasticsearch authorized with shield then, use –httpAuthFile option. Create a file with below user and password.



# NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --httpAuthFile=/opt/mohan/myAuth.file --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json

If need to zip the output file, use as below.

# NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --httpAuthFile=/opt/mohan/myAuth.file --input=http://localhost:9200/mohan-index-2017.05.* --output=$ | gzip > /opt/mohan/mydata.json.gz