Tag: elastic

elasticdump

Elasticdump is the import and export tool for Elasticsearch indexes.

How to install elasticdump and how to copy elasticsearch index?

Install npm and node if not installed already.

Set the proxy if required as below.

npm config set proxy http://myproxy.com:8080
npm config set http_proxy=http://myproxy.com:8080

Install the elasticdump as below.

npm install elasticdump -g

-g is the global mode. It means it installs the current working directory as a global package directory.

Set the soft link to elasticdump

ln -s /opt/mohan/node-v6.10.3-linux-x64/bin/elasticdump /usr/bin/elasticdump

Go to the help to check all the commands in the elasticdump.

elasticdump --help

 

To export to a file:

elasticdump --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json --type=data

elasticdump --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json --type=mapping

if elasticsearch is secured with certs, then use NODE_TLS_REJECT_UNAUTHORIZED as 0 in the beginning of the command.

# NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json

if elasticsearch authorized with shield then, use –httpAuthFile option. Create a file with below user and password.

user=myuser
password=mypassword

Example:

# NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --httpAuthFile=/opt/mohan/myAuth.file --input=http://localhost:9200/mohan-index-2017.05.* --output=/opt/mohan/mydata.json

If need to zip the output file, use as below.

# NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --httpAuthFile=/opt/mohan/myAuth.file --input=http://localhost:9200/mohan-index-2017.05.* --output=$ | gzip > /opt/mohan/mydata.json.gz

 

 

Filebeat

Logstash
Grok
Elasticsearch
Kibana

Filebeat

./bin/plugin install logstash-input-beats
Update the beats plugin if it is 92 then it should be to 96
If [fields][appid] == appid
No tab allowed filebeat.yml file
curl -L -O https://download.elastic.co/beats/filebeat/filebeat-1.0.1-x86_64.rpm
/etc/filebeat.yml
Output to Logstash
curl -XPUT ‘http://localhost:9200/_template/filebeat?pretty’ -d@/etc/filebeat/filebeat.template.json

/usr/bin/filebeat -c /etc/filebeat/filebeat.yml –e

Install Filebeat
In the remote host, download filebeat-1.0.1 from elastic link below and install the RPM with root access.
curl -L -O https://download.elastic.co/beats/filebeat/filebeat-1.0.1-x86_64.rpm

rpm -ivh filebeat-1.0.1-x86_64.rpm

Contents of filebeat
ilamuruhan@ubuntu:/opt$ rpm -qlp filebeat-1.0.1-x86_64.rpm
/etc/filebeat/filebeat.template.json
/etc/filebeat/filebeat.yml
/etc/init.d/filebeat
/usr/bin/filebeat
/usr/bin/filebeat-god

Configure filebeat
File Name : /etc/filebeat/filebeat.yml
Prospector – where from file to read
prospectors:

paths:
– /var/log/*.log
input_type: log
document_type: beat
registry: /var/lib/filebeat/registry

Output – where to send file
output:
logstash:
hosts: [“10.0.0.1:5044”]

Logging – filebeat own logs
logging:
to_files: true
files:
path: /var/log/filebeat
name: filebeat
rotateeverybytes: 10485760
level: error

Configure the path

– Get everything sub-directories
– Track the progress of the file and recognizes the log rotation
– Send logs from remote to Logstash or Elasticsearch