当前位置：首页>数据库>正文

elk做服务监控 elasticsearch监控工具

数据库2024-05-21 23:30:29

环境

MacBook Pro
Java：1.8
skywalking:7.0.0
elasticsearch:7.6.2

前言

公司有一套查看日志的工具，基于kibana+es搭建的日志系统，
但是，在微服务中，某次请求，会涉及多个微服务的调用，公司使用的是SDK去调服务的，
当请求异常时，只能根据相应的请求URL去不断搜索日志。

skywalking的作用就是可以已更直观的界面，已请求链的方式去查看请求-响应。
搭建这个工具需要：skywalking和elasticsearch。

elasticsearch下载和配置

官网的下载链接很慢，这里推荐一个国内下载地址：
https://www.newbe.pro/Mirrors/Mirrors-Elasticsearch/

修改配置

# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: yutao
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
discovery.seed_hosts: ["127.0.0.1"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

需要配置的地方：

cluster.name: yutao – 这个需要和skywalking中的elasticsearch配置保持一致
node.name: node-1
network.host: 0.0.0.0
http.port: 9200 （默认就是，可以不解开注释，我解开了）
discovery.seed_hosts: [“127.0.0.1”] 一定要配置，不然找不到主节点
cluster.initial_master_nodes: [“node-1”] 一定要配置，不然找不到主节点

启动elasticsearch

命令：

yutaodeMacBook-Pro:elasticsearch-7.6.2 yutao$ ./bin/elasticsearch

是否启动成功可以查看日志有没有报错：

yutaodeMacBook-Pro:elasticsearch-7.6.2 yutao$ cd logs/
yutaodeMacBook-Pro:logs yutao$ pwd
/Users/yutao/ProgramFiles/elasticsearch/elasticsearch-7.6.2/logs
yutaodeMacBook-Pro:logs yutao$ tail -f yutao.log

重点留意下，日志里面的警告[WARN]

浏览器访问：URL：http://localhost:9200/?pretty

{
  "name" : "node-1",
  "cluster_name" : "yutao",
  "cluster_uuid" : "cMzO-5LLQem5OHvFuvD0Ig",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

skywalking 下载和配置

官网下载地址：http://skywalking.apache.org/downloads/

本人下载的是Binary Distribution for ElasticSearch 7

elasticsearch的版本要和skywalking的版本对应

修改配置文件

命令：

yutaodeMacBook-Pro:skywalking-es7 yutao$ pwd
/Users/yutao/ProgramFiles/skywalking/skywalking-es7
yutaodeMacBook-Pro:skywalking-es7 yutao$ vim config/application.yml 


storage:
  selector: elasticsearch7
  elasticsearch:
    nameSpace: ${SW_NAMESPACE:"yutao"}
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
    protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}
    trustStorePath: ${SW_SW_STORAGE_ES_SSL_JKS_PATH:"../es_keystore.jks"}
    trustStorePass: ${SW_SW_STORAGE_ES_SSL_JKS_PASS:""}
    user: ${SW_ES_USER:""}
    password: ${SW_ES_PASSWORD:""}
    secretsManagementFile: ${SW_ES_SECRETS_MANAGEMENT_FILE:""} # Secrets management file in the properties format includes the username, password, which are managed by 3rd party tool.
    enablePackedDownsampling: ${SW_STORAGE_ENABLE_PACKED_DOWNSAMPLING:true} # Hour and Day metrics will be merged into minute index.
    dayStep: ${SW_STORAGE_DAY_STEP:1} # Represent the number of days in the one minute/hour/day index.
    indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:2}
    indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0}
    # Those data TTL settings will override the same settings in core module.
    recordDataTTL: ${SW_STORAGE_ES_RECORD_DATA_TTL:7} # Unit is day
    otherMetricsDataTTL: ${SW_STORAGE_ES_OTHER_METRIC_DATA_TTL:45} # Unit is day
    monthMetricsDataTTL: ${SW_STORAGE_ES_MONTH_METRIC_DATA_TTL:18} # Unit is month
    # Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
    bulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:1000} # Execute the bulk every 1000 requests
    flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:10} # flush the bulk every 10 seconds whatever the number of requests
    concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requests
    resultWindowMaxSize: ${SW_STORAGE_ES_QUERY_MAX_WINDOW_SIZE:10000}
    metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:5000}
    segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}
    profileTaskQueryMaxSize: ${SW_STORAGE_ES_QUERY_PROFILE_TASK_SIZE:200}
    advanced: ${SW_STORAGE_ES_ADVANCED:""}
  elasticsearch7:
    nameSpace: ${SW_NAMESPACE:"yutao"}
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
    protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}
    trustStorePath: ${SW_SW_STORAGE_ES_SSL_JKS_PATH:"../es_keystore.jks"}
    trustStorePass: ${SW_SW_STORAGE_ES_SSL_JKS_PASS:""}
    enablePackedDownsampling: ${SW_STORAGE_ENABLE_PACKED_DOWNSAMPLING:true} # Hour and Day metrics will be merged into minute index.
    dayStep: ${SW_STORAGE_DAY_STEP:1} # Represent the number of days in the one minute/hour/day index.
    user: ${SW_ES_USER:""}
    password: ${SW_ES_PASSWORD:""}
    secretsManagementFile: ${SW_ES_SECRETS_MANAGEMENT_FILE:""} # Secrets management file in the properties format includes the username, password, which are managed by 3rd party tool.
    indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:2}
    indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0}
    # Those data TTL settings will override the same settings in core module.
    recordDataTTL: ${SW_STORAGE_ES_RECORD_DATA_TTL:7} # Unit is day
    otherMetricsDataTTL: ${SW_STORAGE_ES_OTHER_METRIC_DATA_TTL:45} # Unit is day
    monthMetricsDataTTL: ${SW_STORAGE_ES_MONTH_METRIC_DATA_TTL:18} # Unit is month
    # Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
    bulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:1000} # Execute the bulk every 1000 requests
    flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:10} # flush the bulk every 10 seconds whatever the number of requests
    concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requests
    resultWindowMaxSize: ${SW_STORAGE_ES_QUERY_MAX_WINDOW_SIZE:10000}
    metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:5000}
    segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}
    profileTaskQueryMaxSize: ${SW_STORAGE_ES_QUERY_PROFILE_TASK_SIZE:200}
    advanced: ${SW_STORAGE_ES_ADVANCED:""}
  #h2:
  #  driver: ${SW_STORAGE_H2_DRIVER:org.h2.jdbcx.JdbcDataSource}
  #  url: ${SW_STORAGE_H2_URL:jdbc:h2:mem:skywalking-oap-db}
  #  user: ${SW_STORAGE_H2_USER:sa}
  #  metadataQueryMaxSize: ${SW_STORAGE_H2_QUERY_MAX_SIZE:5000}

需要注意的地方：

# 因为安装的是elasticsearch7所以这里选的是elasticsearch7
storage:
  selector: elasticsearch7


elasticsearch7:
    nameSpace: ${SW_NAMESPACE:"yutao"} # 这和上面elasticsearch配置中的cluster.name保存一直

网上很多资料，都说注释掉H2配置，所以我也跟着注释了，个人觉得不注释也是可以的，
只要在selector: 中不选择H2就行。

启动skywalking

yutaodeMacBook-Pro:skywalking-es7 yutao$ bin/oapService.sh 
SkyWalking OAP started successfully!

虽然启动后，会返回SkyWalking OAP started successfully!但是并不表示真的启动成功了。
具体的得查看日志：

yutaodeMacBook-Pro:skywalking-es7 yutao$ tail -f logs/skywalking-oap-server.log

日志打印：

2020-04-16 20:36:33,916 - org.eclipse.jetty.server.AbstractConnector - 280 [main] INFO  [] - Started ServerConnector@2e159116{HTTP/1.1,[http/1.1]}{0.0.0.0:12800}
2020-04-16 20:36:33,917 - org.eclipse.jetty.server.Server - 444 [main] INFO  [] - Started @66759ms
2020-04-16 20:36:33,920 - org.apache.skywalking.oap.server.core.storage.PersistenceTimer - 59 [main] INFO  [] - persistence timer start
2020-04-16 20:36:33,929 - org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer - 50 [main] INFO  [] - Cache updateServiceInventory timer start

看到Started @66759ms表示启动成功。

启动UI

yutaodeMacBook-Pro:skywalking-es7 yutao$ bin/webappService.sh
SkyWalking Web Application started successfully!

也是一样，SkyWalking Web Application started successfully!并不表示正在的成功；

查看日志：

2020-04-16 20:38:34.253  INFO 10062 --- [main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase 0
2020-04-16 20:38:34.297  INFO 10062 --- [main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase 2147483647
2020-04-16 20:38:34.305  INFO 10062 --- [main] ration$HystrixMetricsPollerConfiguration : Starting poller
2020-04-16 20:38:34.381  INFO 10062 --- [main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
2020-04-16 20:38:34.387  INFO 10062 --- [main] o.a.s.apm.webapp.ApplicationStartUp      : Started ApplicationStartUp in 28.361 seconds (JVM running for 29.066)

浏览器访问：http://127.0.0.1:8080/

elk做服务监控 elasticsearch监控工具,elk做服务监控 elasticsearch监控工具_java,第1张

遇到的问题

下面这个问题是在启动skywalking时报的错误：

2020-04-16 20:18:21,220 - org.apache.skywalking.oap.server.library.module.BootstrapFlow - 55 [main] INFO  [] - start the provider default in alarm module.
2020-04-16 20:18:21,220 - org.apache.skywalking.oap.server.library.module.BootstrapFlow - 55 [main] INFO  [] - start the provider elasticsearch7 in storage module.
2020-04-16 20:18:21,220 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient - 176 [main] INFO  [] - elasticsearch cluster nodes: localhost:9200
2020-04-16 20:18:51,967 - org.apache.skywalking.oap.server.starter.OAPServerBootstrap - 57 [main] ERROR [] - method [HEAD], host [http://localhost:9200], URI [/_template/yutao_alarm_record?master_timeout=30s], status line [HTTP/1.1 503 Service Unavailable]
org.elasticsearch.ElasticsearchStatusException: method [HEAD], host [http://localhost:9200], URI [/_template/yutao_alarm_record?master_timeout=30s], status line [HTTP/1.1 503 Service Unavailable]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1680) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1446) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1418) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
	at org.elasticsearch.client.IndicesClient.existsTemplate(IndicesClient.java:1209) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.isExistsTemplate(ElasticSearch7Client.java:152) ~[storage-elasticsearch7-plugin-7.0.0.jar:7.0.0]
	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.StorageEsInstaller.isExists(StorageEsInstaller.java:56) ~[storage-elasticsearch-plugin-7.0.0.jar:7.0.0]
	at org.apache.skywalking.oap.server.core.storage.model.ModelInstaller.install(ModelInstaller.java:65) ~[server-core-7.0.0.jar:7.0.0]
	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.StorageModuleElasticsearch7Provider.start(StorageModuleElasticsearch7Provider.java:219) ~[storage-elasticsearch7-plugin-7.0.0.jar:7.0.0]
	at org.apache.skywalking.oap.server.library.module.BootstrapFlow.start(BootstrapFlow.java:58) ~[library-module-7.0.0.jar:7.0.0]
	at org.apache.skywalking.oap.server.library.module.ModuleManager.init(ModuleManager.java:69) ~[library-module-7.0.0.jar:7.0.0]
	at org.apache.skywalking.oap.server.starter.OAPServerBootstrap.start(OAPServerBootstrap.java:43) [server-bootstrap-7.0.0.jar:7.0.0]
	at org.apache.skywalking.oap.server.starter.OAPServerStartUp.main(OAPServerStartUp.java:27) [server-starter-es7-7.0.0.jar:7.0.0]
Caused by: org.elasticsearch.client.ResponseException: method [HEAD], host [http://localhost:9200], URI [/_template/yutao_alarm_record?master_timeout=30s], status line [HTTP/1.1 503 Service Unavailable]
	at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260) ~[elasticsearch-rest-client-7.0.0.jar:7.0.0]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238) ~[elasticsearch-rest-client-7.0.0.jar:7.0.0]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212) ~[elasticsearch-rest-client-7.0.0.jar:7.0.0]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
	... 10 more

在参考https://github.com/apache/skywalking/issues/3317

It shows ElasticSearch 503, so you need to find out what is the error of your ElasticSearch

也就是说，elasticsearch应该也报错，具体得去看看elasticsearch报的什么错误，才能解决问题，
后面我又去看elasticsearch的日志：

[2020-04-16T20:18:51,834][WARN ][r.suppressed             ] [yutaodeMacBook-Pro.local] path: /_template/yutao_alarm_record, params: {master_timeout=30s, name=yutao_alarm_record}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.2.jar:7.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]

重点就是这个错误：org.elasticsearch.discovery.MasterNotDiscoveredException: null

错误原因：

elasticsearch没有发现master和集群初始化主节点的配置为空

而且我也发现elasticsearch一只再报一个警告

[2020-04-16T20:33:52,965][WARN ][o.e.c.c.ClusterFormationFailureHelper] [yutaodeMacBook-Pro.local] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1] to bootstrap a cluster: have discovered [{yutaodeMacBook-Pro.local}{yT65McXYQA2P7whwFHREbg}{s7hXcC29SQKtKKpcpqXu6g}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=8589934592, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [] from hosts providers and [{yutaodeMacBook-Pro.local}{yT65McXYQA2P7whwFHREbg}{s7hXcC29SQKtKKpcpqXu6g}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=8589934592, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
 ^C[2020-04-16T20:33:57,027][INFO ][o.e.x.m.p.NativeController] [yutaodeMacBook-Pro.local] Native controller process has stopped - no new native processes can be started

这个警告信息：没有发现master，导致启动的这个节点没有加入集群。
但是请求:http://localhost:9200/?pretty又是可以的。

我的原因是配置没有配置好：

# 这个注解接口
node.name: node-1
discovery.seed_hosts: ["127.0.0.1"]
cluster.initial_master_nodes: ["node-1"]

Intellij IDEA

方式一：

elk做服务监控 elasticsearch监控工具,elk做服务监控 elasticsearch监控工具_大数据_02,第2张

在VM options:

-javaagent:/Users/yutao/ProgramFiles/skywalking/skywalking-es7/agent/skywalking-agent.jar

上面是我的路径，skywalking-agent.jar这个应该是skywalking安装路径。

在Environment variables:

SW_AGENT_NAME=wms;SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800

SW_AGENT_NAME 填写应用名称
SW_AGENT_COLLECTOR_BACKEND_SERVICES 因为我是单机部署，所以填写本地地址

方式二

elk做服务监控 elasticsearch监控工具,elk做服务监控 elasticsearch监控工具_搜索引擎_03,第3张

在VM options:

-Dskywalking.agent.service_name=skywalking-test-local -Dskywalking.collector.backend_service=127.0.0.1:11800 -javaagent:/Users/yutao/ProgramFiles/skywalking/skywalking-es7/agent/skywalking-agent.jar

然后启动，利用浏览器或者postman请求接口，就可以到skywalking页面看到效果，记得点击右上角的刷新按钮；

elk做服务监控 elasticsearch监控工具,elk做服务监控 elasticsearch监控工具_大数据_04,第4张

追踪

elk做服务监控 elasticsearch监控工具,elk做服务监控 elasticsearch监控工具_elk做服务监控_05,第5张

参考地址：

https://github.com/apache/skywalking/issues/3317

查看全文

https://www.xamrdz.com/database/6r21961150.html

环境

前言