Elastic：如何使用 Ansible自动化部署 Elastic Stack -Metricbeat（五）

在今天的文章中，我们来讲述一些如何使用 ansible 来部署 metricbeat。在进行这个练习之前，我希望你已经完成了之前的练习：

1. 如何使用 Ansible自动化部署 Elastic Stack - Overview（一）

2. 如何使用 Ansible自动化部署 Elastic Stack - Elasticsearch （二）

3. 如何使用 Ansible自动化部署 Elastic Stack - Kibana（三）

4. 如何使用 Ansible自动化部署 Elastic Stack -Security（四）

在我们部署在 Ubunut 机器上的 Elastic Stack 具有安全设置。我们重复之前的步骤来创建 role 及 playbook。

你可以在地址找到源码：https://github.com/liu-xiao-guo/elk-ansible

创建 metricbeat 角色

我们在 elasticsearch/roles 下使用如下的命令：

$ pwd
/Users/liuxg/ansible/elasticsearch/roles
$ ansible-galaxy init metricbeat
- Role metricbeat was created successfully

接下来，我们来配置这个 metricbeat role。我们现在 metricbeat 下的 defaults 目录下修改 main.yml 文件：

metricbeat/defaults/main.yml

---
# defaults file for metricbeat

#------------ Live reload settings ---------
reload_enabled: true
reload_period: 10s

#------------ Template settings ------------
index_number_of_shards: 1
index_codec: best_compression



#-----------  dashboard----------------------
setup_dashboards_enabled: true

tag: elk

#----------- Lopgging -----------------------
logging_level: debug

logging_to_files: true
logging_path: /var/log/metricbeat
logging_file_name: metricbeat
Logging_keepfiles: 7
logging_file_permissions: 0644

#---------- Monitoring ---------------------
monitoring_enabled: true

在上面的文件中，我们定义了一下变量这些变量将会在其它文件中被引用。接下来，我们在 templates 下创建一个叫做 metricbeat.yml 的文件：

metricbeat/templates/metricbeat.yml

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  reload.period: {{ reload_period }}

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: {{ index_number_of_shards }}
  index.codec: {{ index_codec }} 
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["{{ tag }}"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
setup.dashboards.enabled: true

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "{{ kibana_host }}:{{ kibana_port }}"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["{{ elastic_host }}:{{ elastic_port }}"]

  # Protocol - either `http` (default) or `https`.
  protocol: "{{ elastic_protocol }}"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "{{ elastic_username }}"
  password: "{{ elastic_password }}"

# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: {{ logging_level }}

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
logging.selectors: ["*"]

logging.to_files: {{ logging_to_files }}
logging.files:
path: {{ logging_path }}
name: {{ logging_file_name }}
keepfiles: {{ Logging_keepfiles }}
permissions: {{ logging_file_permissions }}

# ============================= X-Pack Monitoring ==============================
# Metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
monitoring.enabled: true

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
monitoring.elasticsearch:

# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

这个文件最原始的文件可以来自之前的一个手动安装的 metricbeat。在这个文件中它使用了我们之前在 credentials.yml 中定义的用户账号信息以及在刚才 defaults/main.yml 文件中定义的变量。

接着，我们创建 tasks。我们编辑 tasks/main.yml 文件：

metricbeat/tasks/main.yml

---
# tasks file for metricbeat

# ------Install Metricbeat--------


- name: Install Metricbeat
  apt:
   name: metricbeat
   update_cache: yes

# ----- Replacing the configuration file

- name: Replace default metricbeat configuration file
  template: 
   src: metricbeat.yml
   dest: /etc/metricbeat/metricbeat.yml

#--------- Starting metricbeat service

- name: Starting metricbeat
  service:
   name: metricbeat
   state: started
   enabled: yes

在上面，我们执行了一下的几个任务：

• 安装 metricbeat
• 覆盖 metricbeat 安装中的配置文件 metricbeat.yml
• 启动 metricbeat 服务

创建 playbook

接下来，我们为 metricbeat 创建一个 playbook，尽管我们可以添加到之前的 deploy-demo.yml 文件中。我们在 playboooks 目录下创建一个叫做 deploy-metricbeat.yml 的文件：

playbooks/deploy-metricbeat.yml

---

# This playbook  will deploy elk stack
- hosts: elk
  become: yes
  vars_files: 
  - ../vars/credentials.yml
  - ../vars/main.yml
  roles:
  - ../roles/metricbeat

我们使用如下的命令来进行部署：

$ pwd
/Users/liuxg/ansible/elasticsearch
$ ansible-playbook -K -i inventory/hosts.yml playbooks/deploy-metricbeat.yml

$ pwd
/Users/liuxg/ansible/elasticsearch
$ ansible-playbook -K -i inventory/hosts.yml playbooks/deploy-metricbeat.yml
BECOME password: 

PLAY [elk] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [192.168.0.4]

TASK [../roles/metricbeat : Install Metricbeat] ********************************
ok: [192.168.0.4]

TASK [../roles/metricbeat : Replace default metricbeat configuration file] *****
changed: [192.168.0.4]

TASK [../roles/metricbeat : Starting metricbeat] *******************************
changed: [192.168.0.4]

PLAY RECAP *********************************************************************
192.168.0.4                : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

从上面的输出中，我们可以看出来部署是成功的。

我们可以直接到 Ubuntu OS 中去查看：

service metricbeat status

上面显示服务已经被成功地启动。在默认的情况下，system 模块是被启动的。

我们登录 Kibana，然后点击 Dashboard，我们可以看到如下的画面：

我们看到收到的信息。

如果我们到 Ubuntu 机器上，运行如下的命令：

sudo metricbeat modules list

$ sudo metricbeat modules list
[sudo] password for liuxg: 
Enabled:
system

Disabled:
activemq
aerospike
apache
appsearch
aws
azure
beat
beat-xpack
ceph
ceph-mgr
cloudfoundry
cockroachdb
consul
coredns
couchbase
couchdb
docker
dropwizard
elasticsearch
elasticsearch-xpack
envoyproxy
etcd
golang
googlecloud
graphite
haproxy
http
ibmmq
iis
istio
jolokia
kafka
kibana
kibana-xpack
kubernetes
kvm
linux
logstash
logstash-xpack
memcached
mongodb
mssql
munin
mysql
nats
nginx
openmetrics
oracle
php_fpm
postgresql
prometheus
rabbitmq
redis
redisenterprise
sql
stan
statsd
tomcat
traefik
uwsgi
vsphere
windows
zookeeper

我们会发现只有 system 模块被启动。假如我们想要启动 Nginx 模块，我们可以做如下的操作。修改 tasks/main.yml 文件：

metricbeat/tasks/main.yml

---
# tasks file for metricbeat

# ------Install Metricbeat--------


- name: Install Metricbeat
  apt:
   name: metricbeat
   update_cache: yes

# ----- Replacing the configuration file

- name: Replace default metricbeat configuration file
  template: 
   src: metricbeat.yml
   dest: /etc/metricbeat/modules.d/nginx

# ----- enable Nginx module

- name: Enable Nginx module
  command: mv /etc/metricbeat/modules.d/nginx.yml.disabled /etc/metricbeat/modules.d/nginx.yml

#--------- Starting metricbeat service

- name: Starting metricbeat
  service:
   name: metricbeat
   state: started
   enabled: yes

在上面，我们直接把 nginx.yml.disabled 文件重新命名即可。重新部署：

ansible-playbook -K -i inventory/hosts.yml playbooks/deploy-metricbeat.yml

等上述的命令执行完后，我们在 Ubuntu OS 上重新使用命令：

sudo metricbeat modules list

我们会发现：

从上面可以看出来 nginx 及 system 两个模块同时被启动了。