当前位置： 首页>后端>正文

vector 安装 与 日志采集

后端2024-05-04 22:35:04

一、安装

[root@VM-201-12-centos ~]# rpm -ivh vector-0.36.0-1.x86_64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
        package vector-0.36.0-1.x86_64 is already installed

[root@VM-201-12-centos ~]# rpm -ql vector
/etc/default/vector
/etc/vector/examples/docs_example.yaml
/etc/vector/examples/environment_variables.yaml
/etc/vector/examples/es_s3_hybrid.yaml
/etc/vector/examples/file_to_cloudwatch_metrics.yaml
/etc/vector/examples/file_to_prometheus.yaml
/etc/vector/examples/namespacing
/etc/vector/examples/namespacing/sinks
/etc/vector/examples/namespacing/sinks/es_cluster.yaml
/etc/vector/examples/namespacing/sinks/s3_archives.yaml
/etc/vector/examples/namespacing/sources
/etc/vector/examples/namespacing/sources/apache_logs.yaml
/etc/vector/examples/namespacing/transforms
/etc/vector/examples/namespacing/transforms/apache_parser.yaml
/etc/vector/examples/namespacing/transforms/apache_sample.yaml
/etc/vector/examples/namespacing/vector.yaml
/etc/vector/examples/prometheus_to_console.yaml
/etc/vector/examples/stdio.yaml
/etc/vector/examples/wrapped_json.yaml
/etc/vector/vector.yaml
/usr/bin/vector
/usr/lib/systemd/system/vector.service
/usr/share/doc/vector-0.36.0
/usr/share/doc/vector-0.36.0/README.md
/usr/share/licenses/vector-0.36.0
/usr/share/licenses/vector-0.36.0/LICENSE
/usr/share/vector/LICENSE-3rdparty.csv
/usr/share/vector/NOTICE
/usr/share/vector/licenses/0BSD
/usr/share/vector/licenses/Apache-2.0
/usr/share/vector/licenses/BSD-2-Clause
/usr/share/vector/licenses/BSD-3-Clause
/usr/share/vector/licenses/BSL-1.0
/usr/share/vector/licenses/CC0-1.0
/usr/share/vector/licenses/ISC
/usr/share/vector/licenses/MIT
/usr/share/vector/licenses/MPL-2.0
/usr/share/vector/licenses/OpenSSL
/usr/share/vector/licenses/Unicode-DFS-2016
/usr/share/vector/licenses/Zlib
/var/lib/vector

二、服务管理

1. 服务管理配置文件 /usr/lib/systemd/system/vector.service 

[Unit]
Description=Vector
Documentation=https://vector.dev
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStartPre=/usr/bin/vector validate
ExecStart=/usr/bin/vector
ExecReload=/usr/bin/vector validate
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
AmbientCapabilities=CAP_NET_BIND_SERVICE
EnvironmentFile=-/etc/default/vector
# Since systemd 229, should be in [Unit] but in order to support systemd <229,
# it is also supported to have it here.
StartLimitInterval=10
StartLimitBurst=5
[Install]
WantedBy=multi-user.target

2. 环境变量 EnvironmentFile=-/etc/default/vector

https://vector.dev/docs/reference/configuration/#environment-variables

三、配置文件

为什么使用 yaml 替换 toml 作为默认配置文件格式 - vector.dev

--config 设置配置文件 - vector.dev，默认 /etc/vector/vector.yaml

范例 vector.yaml：

sources:
  catalinaout_in:
    type: file
    include:
    - /lvmdata/tomcat/tomcat-running/logs/catalina.out.*.log
    read_from: beginning
    multiline:
      start_pattern: ^[^\s]
      condition_pattern: ^\[DEBUG]|^\[INFO\]|^\[ERROR\]|^\[WARN\]|^DEBUG|^INFO|^WARN|^ERROR|^[0-2][0-9]-\w+-2[0-1][0-9][0-9]|^2[0-1][0-9][0-9]-[0-1][0-9]-[0-2][0-9]
      mode: halt_before
      timeout_ms: 1000

transforms:
  catalinaout_tran:
    inputs:
    - catalinaout_in
    type: remap
    source: |
      msg = string!(.message)
      if starts_with(msg, "[") {
      . |= parse_regex!(.message,r'^(\[\w+\]) (?P<timestamp>\d+-\d+-\d+ \d+:\d+:\d+,\d+)')
      .timestamp, err = parse_timestamp(.timestamp, "%Y-%m-%d %H:%M:%S,%3f")
      } else if match(msg, r'^DEBUG|^INFO|^ERROR|^WARN') {
      . |= parse_regex!(.message,r'^((DEBUG|INFO|WARN|ERROR)) (?P<timestamp>\d+-\d+-\d+ \d+:\d+:\d+.\d+)')
      .timestamp, err = parse_timestamp(.timestamp, "%Y-%m-%d %H:%M:%S%.3f")
      } else if match(msg, r'^[0-9]{2}-\w{3}-[0-9]{4}') {
      . |= parse_regex!(.message,r'^(?P<timestamp>\d+-\w+-\d+ \d+:\d+:\d+.\d+)')
      .timestamp, err = parse_timestamp(.timestamp, "%d-%b-%Y %H:%M:%S%.3f")
      } else {
      . |= parse_regex!(.message,r'^(?P<timestamp>\d+-\d+-\d+ \d+:\d+:\d+.\d+)')
      .timestamp, err = parse_timestamp(.timestamp, "%Y-%m-%d %H:%M:%S%.3f")
      }

sinks:
  victorialogs:
    inputs:
    - catalinaout_tran
    healthcheck:
      enabled: false
    type: elasticsearch
    endpoints:
    - http://10.41.201.5:9428/insert/elasticsearch/
    api_version: v8
    mode: bulk
    compression: none
    request:
      headers:
        AccountID: '0'
        ProjectID: '0'
    query:
      _stream_fields: host,file
      _msg_field: message
      _time_field: timestamp
  openobserve:
    inputs:
    - catalinaout_tran
    healthcheck:
      enabled: false
    type: http
    uri: http://10.41.201.7:5080/api/default/core_rh/_json
    method: post
    auth:
      strategy: basic
      user: admin@my.com
      password: admin
    encoding:
      codec: json
      timestamp_format: rfc3339

复杂日志在转换处理的时候，可能会报错，日志写不进去，丢了不少。transforms 还是用在特定场合吧。

四、子目录/多配置文件

https://vector.dev/docs/reference/configuration/#multiple-files

在 /etc/vector 目录下，创建子目录：sources、tranforms、sinks，范例如下：

[root@VM-1-11-centos ~]# tree /etc/vector/
/etc/vector/
|-- sinks
|   |-- catalina-openobserve.yaml
|   |-- catalina-vlog.yaml
|   `-- test-console.yaml
|-- sources
|   |-- src_catalina.yaml
|   `-- src_test.yaml.bak
|-- transforms
|   `-- trans_add_filed.yaml
`-- vector.yaml

3 directories, 7 files
  • /etc/vector.yaml，只需要包含一些全局配置：
data_dir: /var/lib/vector/
  • sources/src_catalina.yaml
    type: file
    include:
    - /lvmdata/tomcat/tomcat-running/logs/catalina.out.*.log
    read_from: beginning
    multiline:
      start_pattern: ^[^\s]
      condition_pattern: ^\[DEBUG]|^\[INFO\]|^\[ERROR\]|^\[WARN\]|^DEBUG|^INFO|^WARN|^ERROR|^[0-2][0-9]-\w+-2[0-1][0-9][0-9]|^2[0-1][0-9][0-9]-[0-1][0-9]-[0-2][0-9]
      mode: halt_before
      timeout_ms: 1000
    host_key: "host"
  • transforms/trans_add_filed.yaml
    inputs:
    - src_*
    type: remap
    source: |
      .hostip = "10.41.1.11"
      .service = "core_rh"

增加 hostip 和 service field，便于检索。

  • sinks/catalina-vlog.yaml 输出到 victorialogs
    inputs:
    - trans_*
    healthcheck:
      enabled: false
    type: elasticsearch
    endpoints:
    - http://10.41.201.5:9428/insert/elasticsearch/
    api_version: v8
    mode: bulk
    compression: none
    request:
      headers:
        AccountID: '0'
        ProjectID: '0'
    query:
      _stream_fields: host,file,hostip,service
      _msg_field: message
      _time_field: timestamp
  • sinks/catalina-openobserve.yaml 输出到 openobserve
    inputs:
    - trans_*
    healthcheck:
      enabled: false
    type: http
    uri: http://10.41.201.7:5080/api/default/core_rh/_json
    method: post
    auth:
      strategy: basic
      user: admin@my.com
      password: admin
    encoding:
      codec: json
      timestamp_format: rfc3339
  • sinks/test-console.yaml 输出到 console，用于调试
    inputs:
    - trans_*
    type: console
    target: stdout
    encoding:
      codec: json

inputs 以前面配置文件去除.yaml的文件名，作为 inputs的输入。如 catalina.yaml 以 catalina 作为 sinks 的inputs。支持 通配符。

3. 试运行

vector --config-dir /etc/vector

4. 修改服务管理文件

修改 /usr/lib/systemd/system/vector.service，增加 --config-dir /etc/vector。

[Unit]
Description=Vector
Documentation=https://vector.dev
After=network-online.target
Requires=network-online.target

[Service]
User=vector
Group=vector
ExecStartPre=/usr/bin/vector validate --config-dir /etc/vector
ExecStart=/usr/bin/vector --config-dir /etc/vector
ExecReload=/usr/bin/vector validate --config-dir /etc/vector
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
AmbientCapabilities=CAP_NET_BIND_SERVICE
EnvironmentFile=-/etc/default/vector
# Since systemd 229, should be in [Unit] but in order to support systemd <229,
# it is also supported to have it here.
StartLimitInterval=10
StartLimitBurst=5
[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl restart vector

https://www.xamrdz.com/backend/3dp1941883.html

相关文章：