構成/接続イメージ
適当なイメージ
インストール環境
- OS:CentOS7.4
- ElasticSearch 6.2
- Kibana 6.2
- Nginx 1.12.2
- Logstash 6.2
- Java: openjdk1.8.0 161
※ubuntu16.04では公式インストール手順はインストールまではできるが、そのあとのパッケージが動しなかったのでCentOSで代わりに実施とした
事前準備
Javaインストール
[user@localhost ~]$ sudo yum install -y java-1.8.0-openjdk
[user@localhost ~]$ java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
[user@localhost ~]$
パッケージ更新
[user@localhost ~]$ sudo yum update
[user@localhost ~]$ sudo yum upgrade
Elasticsearch
リポジトリ追加
パブリックキー取得
[user@localhost ~]$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[user@localhost ~]$
リポジトリ追加
[user@localhost ~]$ sudo vi /etc/yum.repos.d/elasticsearch.repo
■■■■ 追加 ■■■■
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
■■■■■■■■■■■■■■■■■
Elasticsearchインストール
[user@localhost ~]$ sudo yum install -y elasticsearch
設定修正
[user@localhost ~]$ sudo vi /etc/elasticsearch/elasticsearch.yml
■修正前
#network.host: 192.168.0.1
■修正後(追加)
network.host: 0.0.0.0
Elasticsearch起動
[user@localhost ~]$ sudo systemctl daemon-reload
[user@localhost ~]$ sudo systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[user@localhost ~]$ sudo systemctl start elasticsearch.service
Elasticsearch起動確認
URL: http://127.0.0.1:9200/ で接続確認
[user@localhost ~]$ curl http://127.0.0.1:9200/
{
"name" : "5KgF1O0",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "Su8pjFvdRBKvASBmVJjfwg",
"version" : {
"number" : "6.2.1",
"build_hash" : "7299dc3",
"build_date" : "2018-02-07T19:34:26.990113Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
[user@localhost ~]$
kibana
リポジトリ追加
パブリックキー取得(取得済みなら不要)
[user@localhost ~]$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[user@localhost ~]$
リポジトリ追加
[user@localhost ~]$ sudo vi /etc/yum.repos.d/kibana.repo
■■■■ 追加 ■■■■
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
■■■■■■■■■■■■■■■■■
kibanaインストール
[user@localhost ~]$ sudo yum install -y kibana
kibana起動
[user@localhost ~]$ sudo systemctl daemon-reload
[user@localhost ~]$ sudo systemctl enable kibana.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
[user@localhost ~]$ sudo systemctl start kibana.service
[user@localhost ~]$
kibana起動確認
[user@localhost ~]$ curl http://127.0.0.1:5601/
<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';
var hash = window.location.hash;
if (hash.length) {
window.location = hashRoute + hash;
} else {
window.location = defaultRoute;
}</script>
NGINX
https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-centos-7
リポジトリ追加
[user@localhost ~]$ sudo yum install epel-release
アクセス制限(basic認証)用パッケージ追加
[user@localhost ~]$ sudo yum install -y httpd-tools
nginxインストール
[user@localhost ~]$ sudo yum install -y nginx
[user@localhost ~]$ nginx -v
nginx version: nginx/1.12.2
[user@localhost ~]$
nginx設定
ほぼプロキシとして間的に動かすのみの設定としている。
[user@localhost ~]$ sudo vi /etc/nginx/nginx.conf
■■■■■■以下をコメントアウト■■■■■■
# server {
# listen 80 default_server;
# listen [::]:80 default_server;
# server_name _;
# root /usr/share/nginx/html;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
# location / {
# }
#
# error_page 404 /404.html;
# location = /40x.html {
# }
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
# }
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
[user@localhost ~]$ sudo vi /etc/nginx/conf.d/kibana.conf
■■■■ 追加 ■■■■
server {
listen *:80;
server_name _;
location / {
proxy_pass http://localhost:5601;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;
}
}
■■■■■■■■■■■■■■■■■
ユーザ作成
ユーザ名：adminuser
[user@localhost ~]$ sudo htpasswd -c /etc/nginx/.htpasswd adminuser
New password:
Re-type new password:
Adding password for user adminuser
[user@localhost ~]$
nginx起動
[user@localhost ~]$ sudo systemctl enable nginx
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
[user@localhost ~]$ sudo systemctl start nginx
kibana起動確認
logstash
リポジトリ追加
パブリックキー取得
[user@localhost ~]$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[user@localhost ~]$
リポジトリ追加
[user@localhost ~]$ sudo vi /etc/yum.repos.d/logstash.repo
■■■■ 追加 ■■■■
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
■■■■■■■■■■■■■■■■■
logstashインストール
[user@localhost ~]$ sudo yum install -y logstash
logstash設定編集①
[user@localhost ~]$ sudo vi /etc/logstash/jvm.options
■■■■ 追加 ■■■■
## GC configuration
-XX:-AssumeMP
■■■■■■■■■■■■■■■■■
logstash起動
[user@localhost ~]$ sudo systemctl daemon-reload
[user@localhost ~]$ sudo systemctl start logstash.service
[user@localhost ~]$ sudo systemctl enable logstash.service
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
logstash簡易テスト
Logstashはタイムスタンプとホスト名(IPアドレスから逆引きしている?)情報をメッセージに追加され出力される。
[user@localhost ~]$ sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -e 'input { stdin { } }
output { stdout {} }'
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
The stdin plugin is now waiting for input:
hello world
2018-02-22T12:30:57.535Z localhost.localdomain hello world
[user@localhost ~]$
#終了：CTRL + d コマンド
標準入力から入力を受け取り、stdin入力からstdout標準出力へ構造化形式で処理されてるらしいです。
(番外)logstash起動テスト(オプション無＋設定変更しない場合)
エラーやwarningなどが発生する。
Logstashのコマンドを入力した後、2-3分程度待ってから "hello world" を入力する
[user@localhost ~]$ sudo /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
[sudo] user のパスワード:
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-02-22 20:25:50.368 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-02-22 20:25:50.422 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[INFO ] 2018-02-22 20:25:50.517 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2018-02-22 20:25:50.574 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2018-02-22 20:25:51.761 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-02-22 20:25:51.837 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"d9bef938-f22e-4481-bf42-16209501bc66", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2018-02-22 20:25:52.250 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.1"}
[INFO ] 2018-02-22 20:25:52.832 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-02-22 20:25:53.290 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
The stdin plugin is now waiting for input:
[INFO ] 2018-02-22 20:25:53.433 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x40971174@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run>"}
[INFO ] 2018-02-22 20:25:53.456 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
hello world
2018-02-22T11:26:50.196Z localhost.localdomain hello world
[INFO ] 2018-02-22 20:26:57.619 [[main]-pipeline-manager] pipeline - Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x40971174@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run>"}
[user@localhost ~]$
終了：CTRL + d コマンド
一応この場合は、マニュアルどおりのタイムスタンプが付与された出力結果となっているが、いろいろwarningやファイルがなかったりしているが正しい動作なのかは不明。
