備忘録/にわかエンジニアが好きなように書く

個人的にとりあえず仕組みを知るために、触りたように好きにとりあえず動くような構築してみる 個人用の備忘録となるので内容の保証はないのでその点はご了承ください。 ※変な内容や間違いを書いているなどありましたらコメントやご指摘いただけると幸いです。

インストール:Elasticsearch, Logstash, and Kibana (ELK Stack) + Nginx(ReverseProxy) on CentOS7.4

構成/接続イメージ

f:id:pocket01:20180222221737p:plain

 適当なイメージ

インストール環境

  • OS:CentOS7.4
  • ElasticSearch 6.2
  • Kibana 6.2
  • Nginx 1.12.2
  • Logstash 6.2
  • Java: openjdk1.8.0 161

※ubuntu16.04では公式インストール手順はインストールまではできるが、そのあとのパッケージが動しなかったのでCentOSで代わりに実施とした

事前準備

Javaインストール

[user@localhost ~]$ sudo yum install -y java-1.8.0-openjdk
[user@localhost ~]$ java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
[user@localhost ~]$

パッケージ更新

[user@localhost ~]$ sudo yum update
[user@localhost ~]$ sudo yum upgrade

Elasticsearch

www.elastic.co

リポジトリ追加

パブリックキー取得
[user@localhost ~]$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[user@localhost ~]$
リポジトリ追加
[user@localhost ~]$ sudo vi /etc/yum.repos.d/elasticsearch.repo

■■■■ 追加 ■■■■
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
■■■■■■■■■■■■■■■■■

Elasticsearchインストール

[user@localhost ~]$ sudo yum install -y elasticsearch

設定修正

[user@localhost ~]$ sudo vi /etc/elasticsearch/elasticsearch.yml
■修正前
#network.host: 192.168.0.1

■修正後(追加)
network.host: 0.0.0.0

Elasticsearch起動

[user@localhost ~]$ sudo systemctl daemon-reload
[user@localhost ~]$ sudo systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[user@localhost ~]$ sudo systemctl start elasticsearch.service

Elasticsearch起動確認

URL: http://127.0.0.1:9200/ で接続確認

[user@localhost ~]$ curl http://127.0.0.1:9200/
{
"name" : "5KgF1O0",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "Su8pjFvdRBKvASBmVJjfwg",
"version" : {
"number" : "6.2.1",
"build_hash" : "7299dc3",
"build_date" : "2018-02-07T19:34:26.990113Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
[user@localhost ~]$

 

kibana

www.elastic.co

リポジトリ追加

パブリックキー取得(取得済みなら不要)
[user@localhost ~]$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[user@localhost ~]$
リポジトリ追加
[user@localhost ~]$ sudo vi /etc/yum.repos.d/kibana.repo

■■■■ 追加 ■■■■
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
■■■■■■■■■■■■■■■■■

kibanaインストール

[user@localhost ~]$ sudo yum install -y kibana

kibana起動

[user@localhost ~]$ sudo systemctl daemon-reload
[user@localhost ~]$ sudo systemctl enable kibana.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
[user@localhost ~]$ sudo systemctl start kibana.service
[user@localhost ~]$

kibana起動確認

[user@localhost ~]$ curl http://127.0.0.1:5601/
<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';

var hash = window.location.hash;
if (hash.length) {
window.location = hashRoute + hash;
} else {
window.location = defaultRoute;
}</script>

 

NGINX

https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-centos-7

リポジトリ追加

[user@localhost ~]$ sudo yum install epel-release

アクセス制限(basic認証)用パッケージ追加

[user@localhost ~]$ sudo yum install -y httpd-tools

nginxインストール

[user@localhost ~]$ sudo yum install -y nginx
[user@localhost ~]$ nginx -v
nginx version: nginx/1.12.2
[user@localhost ~]$

nginx設定

ほぼプロキシとして間的に動かすのみの設定としている。

[user@localhost ~]$ sudo vi /etc/nginx/nginx.conf
■■■■■■以下をコメントアウト■■■■■■
# server {
# listen 80 default_server;
# listen [::]:80 default_server;
# server_name _;
# root /usr/share/nginx/html;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
# location / {
# }
#
# error_page 404 /404.html;
# location = /40x.html {
# }
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
# }
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
[user@localhost ~]$ sudo vi /etc/nginx/conf.d/kibana.conf
■■■■ 追加 ■■■■

server {
listen *:80;
server_name _;
location / {
proxy_pass http://localhost:5601;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;
}
}
■■■■■■■■■■■■■■■■■

ユーザ作成

ユーザ名:adminuser

[user@localhost ~]$ sudo htpasswd -c /etc/nginx/.htpasswd adminuser
New password:
Re-type new password:
Adding password for user adminuser
[user@localhost ~]$

nginx起動

[user@localhost ~]$ sudo systemctl enable nginx
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
[user@localhost ~]$ sudo systemctl start nginx

kibana起動確認

f:id:pocket01:20180220223820p:plain

 

f:id:pocket01:20180220225851p:plain

 

logstash

www.elastic.co

リポジトリ追加

パブリックキー取得
[user@localhost ~]$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[user@localhost ~]$
リポジトリ追加
[user@localhost ~]$ sudo vi /etc/yum.repos.d/logstash.repo

■■■■ 追加 ■■■■
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
■■■■■■■■■■■■■■■■■

logstashインストール

[user@localhost ~]$ sudo yum install -y logstash

logstash設定編集①

[user@localhost ~]$ sudo vi /etc/logstash/jvm.options
■■■■ 追加 ■■■■
## GC configuration
-XX:-AssumeMP
■■■■■■■■■■■■■■■■■

logstash起動

[user@localhost ~]$ sudo systemctl daemon-reload
[user@localhost ~]$ sudo systemctl start logstash.service
[user@localhost ~]$ sudo systemctl enable logstash.service
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.

logstash簡易テスト

Logstashはタイムスタンプとホスト名(IPアドレスから逆引きしている?)情報をメッセージに追加され出力される。

[user@localhost ~]$ sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -e 'input { stdin { } }
output { stdout {} }'
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
The stdin plugin is now waiting for input:
hello world
2018-02-22T12:30:57.535Z localhost.localdomain hello world
[user@localhost ~]$

#終了:CTRL + d コマンド

標準入力から入力を受け取り、stdin入力からstdout標準出力へ構造化形式で処理されてるらしいです。

(番外)logstash起動テスト(オプション無+設定変更しない場合)

エラーやwarningなどが発生する。

Logstashのコマンドを入力した後、2-3分程度待ってから "hello world" を入力する

[user@localhost ~]$ sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
[sudo] user のパスワード:
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-02-22 20:25:50.368 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-02-22 20:25:50.422 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[INFO ] 2018-02-22 20:25:50.517 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2018-02-22 20:25:50.574 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2018-02-22 20:25:51.761 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-02-22 20:25:51.837 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"d9bef938-f22e-4481-bf42-16209501bc66", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2018-02-22 20:25:52.250 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.1"}
[INFO ] 2018-02-22 20:25:52.832 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-02-22 20:25:53.290 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[INFO ] 2018-02-22 20:25:53.433 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x40971174@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run>"}
[INFO ] 2018-02-22 20:25:53.456 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}

hello world
2018-02-22T11:26:50.196Z localhost.localdomain hello world
[INFO ] 2018-02-22 20:26:57.619 [[main]-pipeline-manager] pipeline - Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x40971174@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run>"}
[user@localhost ~]$

終了:CTRL + d コマンド

一応この場合は、マニュアルどおりのタイムスタンプが付与された出力結果となっているが、いろいろwarningやファイルがなかったりしているが正しい動作なのかは不明。