elasticsearch Getting Started

서비스 로그를 모아서 실시간으로 검색하고 분석할 수 있는 시스템이 필요해서 검색중

로그를 수집하고(fluentd), 검색(elasticsearch) 가능한 시스템이 있어서 우리 시스템이 도입하려고 한다.

elasticsearch

http://www.elasticsearch.org/

download

$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.tar.gz

$ tar zxvf elasticsearch-1.4.2.tar.gz

start

$ ./bin/elasticsearch -Xmx1g -Xms1g

[2015-01-30 16:30:55,456][WARN ][common.jna               ] Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK (ulimit).

[2015-01-30 16:30:55,519][INFO ][node                     ] [Apocalypse] version[1.4.2], pid[32010], build[927caff/2014-12-16T14:11:12Z]

[2015-01-30 16:30:55,519][INFO ][node                     ] [Apocalypse] initializing ...

[2015-01-30 16:30:55,524][INFO ][plugins                  ] [Apocalypse] loaded [], sites [head, bigdesk]

[2015-01-30 16:30:57,393][INFO ][node                     ] [Apocalypse] initialized

[2015-01-30 16:30:57,394][INFO ][node                     ] [Apocalypse] starting ...

[2015-01-30 16:30:57,466][INFO ][transport                ] [Apocalypse] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/server ip:9300]}

[2015-01-30 16:30:57,476][INFO ][discovery                ] [Apocalypse] elasticsearch/xN1-Ey89RaSGxvFB3TW4ww

[2015-01-30 16:31:01,246][INFO ][cluster.service          ] [Apocalypse] new_master [Apocalypse][xN1-Ey89RaSGxvFB3TW4ww][server host name][inet[/172.20.51.135:9300]], reason: zen-disco-join (elected_as_master)

[2015-01-30 16:31:01,270][INFO ][http                     ] [Apocalypse] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/server ip:9200]}

[2015-01-30 16:31:01,270][INFO ][node                     ] [Apocalypse] started

[2015-01-30 16:31:01,277][INFO ][gateway                  ] [Apocalypse] recovered [0] indices into cluster_state

Conenct localhost:9200

{
"status" : 200,
"name" : "Apocalypse",
"cluster_name" : "elasticsearch",
"version" : {
    "number" : "1.4.2",
    "build_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c",
    "build_timestamp" : "2014-12-16T14:11:12Z",
    "build_snapshot" : false,"lucene_version" : "4.10.2"
},
"tagline" : "You Know, for Search"}

설치가 끝났다.

서버 모니터링을 위해서 필수 플러그인 설치

site plugins

$ bin/plugin --install mobz/elasticsearch-head
$ bin/plugin --install lukas-vlcek/bigdesk

http://localhost:9200/_plugin/head/
http://localhost:9200/_plugin/bigdesk/

다른 플러그인은 여기서..

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html

setup

자세한 설명은 여기서

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html

변경한 설정

###### Cluster ######
cluster.name : 클러스터 이름

###### Node ######
node.name : "노드 이름"

node.master : 노드가 마스터 가 될 수 있는지
node.data : 데이터를 저장하는 노드인지 설정

# 규모가 큰 topologies에서는 각 노드 별로 역할을 분리하여 구성하는게 좋다.
# You can exploit these settings to design advanced cluster topologies.
#
# 1. You want this node to never become a master node, only to hold data.
#    This will be the "workhorse" of your cluster.
#
#node.master: false
#node.data: true
#
# 2. You want this node to only serve as a master: to not store any data and
#    to have free resources. This will be the "coordinator" of your cluster.
#
#node.master: true
#node.data: false
#
# 3. You want this node to be neither master nor data node, but
#    to act as a "search load balancer" (fetching data from nodes,
#    aggregating results, etc.)
#
#node.master: false
#node.data: false

# Use the Cluster Health API [http://localhost:9200/_cluster/health], the
# Node Info API [http://localhost:9200/_nodes]

###### Index ######
# 샤드와 리플리카 수 변경
index.number_of_shards: 5
index.number_of_replicas: 1

###### Paths ######
path.conf : 설정파일 위치
path.data : index 데이터 저장 위치
path.log : 로그위치

###### Memory ######
bootstrap.mlockall: JVM 스왑 방지 true 로 설정

###### Discovery ######
discovery.zen.ping.timeout: 각 노드간의 상태 체크 타임 아웃  기본 3s
discovery.zen.ping.multicast.enabled: false 멀티 캐스트하여 노드를 찾지 않도록
discovery.zen.ping.unicast.hosts: ["host:port", "host:port"", "host:port""] 유니 캐스트 목록설정

Cluster mode

클러스터 모드로 서비스를 올리면 각 노드가 시작되면 클러스터에 추가되고 아래 로그가 나온다.
[2015-01-30 17:36:10,549][INFO ][cluster.service          ] [노드명] added {[mars][oCu8yDXgRKympVU4h1MO9Q][클러스터에 추가된 노드명][inet[/host:port]]{master=true},}, reason: zen-disco-receive(join from node[[클러스터에 추가된 노드명][oCu8yDXgRKympVU4h1MO9Q][클러스터에 추가된 노드명][inet[/host:port]]{master=true}])

클러스터 정보 : http://localhost:9200/_cluster/health?pretty

{
  "cluster_name" : "클러스터명",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}

데이터 저장은... 다음에..