収集対象ログ

/etc/elasticsearch/logging.ymlにて設定された動作でログを次のファイルに出力します。

/var/log/elasticsearch/elasticsearch.log
/var/log/elasticsearch/elasticsearch_index_search_slowlog.log
/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log

slowlogに関しては/etc/elasticsearch/elasticsearch.ymlにて設定した閾値時間を超えて処理したクエリを記録できますが、デフォルトでは設定が行われていません。後々のパフォーマンス観察を容易にするためにも設定しておくと便利です。

設定例

Fluentdでログを収集する設定例は次の通りです。
この例ではelasticsearch.*に該当するタグでログをまとめて、fluentdのstdoutログに出力します。

<source>
  type     tail
  tag      elasticsearch.general_log
  format   /^\[(?<time>[^ ]* [^ ]*)\]\[(?<log_level>[^ ]*) *?\]\[(?<log_type>[^ ]*) *\] \[(?<node_name>[^ ]*) *\] (?<message>.*)$/
  path     /var/log/elasticsearch/elasticsearch.log
  pos_file /var/log/td-agent/elasticsearch.log.pos
</source>

<source>
  type     tail
  tag      elasticsearch.search_slowlog
  format   /^\[(?<time>[^ ]* [^ ]*)\]\[(?<log_level>[^ ]*) *?\]\[(?<log_type>[^ ]*) *\] \[(?<node_name>[^ ]*) *\] (?<message>.*)$/
  path     /var/log/elasticsearch/elasticsearch_index_search_slowlog.log
  pos_file /var/log/td-agent/elasticsearch_index_search_slowlog.log.pos
</source>

<source>
  type     tail
  tag      elasticsearch.indexing_slowlog
  format   /^\[(?<time>[^ ]* [^ ]*)\]\[(?<log_level>[^ ]*) *?\]\[(?<log_type>[^ ]*) *\] \[(?<node_name>[^ ]*) *\] (?<message>.*)$/
  path     /var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log
  pos_file /var/log/td-agent/elasticsearch_index_indexing_slowlog.log.pos
</source>

<match elasticsearch.*>
  type     stdout
</match>

ログデータサンプル

実際にどのように分解されるかは、Fluentularで確認すると便利です。

elasticsearch.log

[2014-03-18 18:27:34,897][INFO ][http                     ] [es01] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.0.0.185:9200]}
[2014-03-18 18:27:35,647][INFO ][gateway                  ] [es01] recovered [3] indices into cluster_state
[2014-03-18 18:27:35,648][INFO ][node                     ] [es01] started
[2014-03-18 18:27:37,426][INFO ][cluster.service          ] [es01] added {[es02][fC4QjocCREagqkxz3nQ8rQ][v157-7-203-186.z1d4.static.cnode.jp][inet[/10.0.0.186:9300]]{master=false},}, reason: zen-disco-receive(join from node[[es02][fC4QjocCREagqkxz3nQ8rQ][v157-7-203-186.z1d4.static.cnode.jp][inet[/10.0.0.186:9300]]{master=false}])

elasticsearch_index_indexing_slowlog.log

[2014-03-18 18:26:12,630][TRACE][index.indexing.slowlog.index] [es01] [b][0] took[70.8ms], took_millis[70], type[es02], id[1], routing[], source[{"user":"kimchy","post_date":"2009-11-15T14:12:12","message":"trying out Elasticsearch"}]

elasticsearch_index_search_slowlog.log
elasticsearch index.search.slowlog.query解释-IT技术精华网より引用

[2013-11-27 13:02:51,436][TRACE][index.search.slowlog.query] [h35] [cms][2] took[1.1s], took_millis[1136], types[news], stats[], search_type[QUERY_THEN_FETCH], total_shards[10], source[{"from":250,"size":10,"query":{"custom_score":{"query":{"filtered":{"query":{"query_string":{"query":"尼日利亚","fields":["title","content"],"analyzer":"ik"}},"filter":{"bool":{"must":{"range":{"updatetime":{"from":1353992570,"to":1385528570,"include_lower":true,"include_upper":true}}}}}}},"script":"long x=Long.parseLong(doc['updatetime'].value);x=x/86400l;long nows=yy/86400l;int nowDay=(int)nows;int day=(int)x;if (day < nowDay) {int temp=nowDay-day;int delta=(int)temp/5+1;if(delta<32){_score=_score+6*pow(0.9,delta);}else{_score=_score+3*pow(0.9,31);}} else if (day > nowDay) {_score=_score+3*pow(0.9,31);} else{_score=_score+3;}","params":{"yy":1385528570}}},"fields":["id","channelid","catid","model","title","content","thumb","url","updatetime"],"sort":[{"_score":{"order":"desc"}},{"updatetime":{"order":"desc"}}],"highlight":{"pre_tags":["<em>"],"post_tags":["</em>"],"encoder":"UTF-8","fields":{"title":{},"content":{"fragment_size":72,"number_of_fragments":3}}}}], extra_source[]