当前位置: 首页>编程语言>正文

Logstash实现mysql同步数据到ElasticSearch(安全认证)

  1. Logstash安装:
    1.下载安装包:wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.2.tar.gz
    2.下载mysql的连接库jar包(版本库):wget http://central.maven.org/maven2/mysql/mysql-connector-java/6.0.6/mysql-connector-java-6.0.6.jar
    2.解压安装包:tar -xvf logstash-6.6.2.tar.gz
    4.安装jdbc、elasticsearch插件:
    Logstash实现mysql同步数据到ElasticSearch(安全认证),第1张
logstash插件安装命令
cd logstash-6.6.2/bin
./logstash-plugin install logstash-input-jdbc
./logstash-plugin install logstash-output-elasticsearch
  1. 创建配置文件
    在config目录下,创建配置文件(logstash-mysql-es.conf):
input {
# stdin { }
    jdbc {
        jdbc_connection_string => "jdbc:mysql://localhost:30012/data_acquisition?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull&allowMultiQueries=true"

        jdbc_user => "root"

        jdbc_password => "123456"

        jdbc_driver_library => "/path/mysql-connector-java-6.0.6.jar" #全路径

        jdbc_driver_class => "com.mysql.jdbc.Driver"

        jdbc_paging_enabled => "true"

        jdbc_page_size => "50000"

        statement => "select * from result_comment where create_time >= '2019-03-20'"
        #schedule => "* * * * *"
    }
 }

 output {
     # 这里输出调试,正式运行时可以注释掉
     stdout {
        codec => json_lines
    }
    elasticsearch {
        hosts => "127.0.0.1:9200"
        index => "wl_travel_analyze"
        document_type => "travel"
        document_id => "%{id}"
        #ssl => true
        #ssl_certificate_verification => true
        #truststore => "/opt/elk/search-guard-tlstool/search-guard-certificates/truststore.jks"
        #truststore_password => "286a7b7a8970af4e8467"
        user => "admin"
        password => "admin"
    }
}
注意添加ElasticSearch用户名和密码
  1. 启动改脚本
    nohup ./bin/logstash -f config/logstash-mysql-es.conf > logs/logstash.out &
  2. 查看日志
    tail -f logs/logstash.out

借鉴文章如下:
Elasticsearch - Logstash实现mysql同步数据到elasticsearch


https://www.xamrdz.com/lan/5ef2016427.html

相关文章: