Monitoring with Prometheus and Grafana

You can monitor your SUSE Manager environment using Prometheus and Grafana. SUSE Manager Server and Proxy are able to provide self-health metrics. You can also install and manage a number of Prometheus exporters on Salt clients.

Prometheus and Grafana packages are included in the SUSE Manager Client Tools for SUSE Linux Enterprise 12, SUSE Linux Enterprise 15 and openSUSE 15.x.

You need to install Prometheus and Grafana on a machine separate from the SUSE Manager Server. We recommend you use a managed Salt client as your monitoring server.

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to monitored clients. Clients must have corresponding open ports and be reachable over the network.

Prometheus and Grafana

Prometheus

Prometheus is an open-source monitoring tool that is used to record real-time metrics in a time-series database. Metrics are pulled via HTTP, enabling high performance and scalability.

Prometheus metrics are time series data, or timestamped values belonging to the same group or dimension. A metric is uniquely identified by its name and set of labels.

   metric name               labels                 timestamp     value
┌────────┴───────┐  ┌───────────┴───────────┐    ┌──────┴──────┐  ┌─┴─┐
http_requests_total{status="200", method="GET"}  @1557331801.111  42236

Each application or system being monitored must expose metrics in the format above, either through code instrumentation or Prometheus exporters.

Prometheus Exporters

Exporters are libraries that help with exporting metrics from third-party systems as Prometheus metrics. Exporters are useful whenever it is not feasible to instrument a given application or system with Prometheus metrics directly. Multiple exporters can run on a monitored host to export local metrics.

The Prometheus community provides a list of official exporters, and more can be found as community contributions. For more information and an extensive list of exporters, see https://prometheus.io/docs/instrumenting/exporters/.

Grafana

Grafana is a tool for data visualization, monitoring, and analysis. It is used to create dashboards with panels representing specific metrics over a set period of time. Grafana is commonly used together with Prometheus, but also supports other data sources such as ElasticSearch, MySQL, PostgreSQL, and Influx DB. For more information about Grafana, see https://grafana.com/docs/.

Set up the Monitoring Server

Install Prometheus

If your monitoring server is a SUSE Manager Salt client, you can install the Prometheus package using the SUSE Manager Web UI. Otherwise you can download and install the package on your monitoring server manually.

Procedure: Installing Prometheus using the Web UI
  1. In the SUSE Manager Web UI, open the details page of the system where Prometheus is to be installed, and navigate to the Formulas tab.

  2. Check the Prometheus checkbox to enable monitoring formulas, and click Save.

  3. Navigate to the Prometheus tab in the top menu.

  4. In the SUSE Manager Server section, enter valid SUSE Manager API credentials. Make sure that the credentials you have entered allow access to the set of systems you want to monitor.

  5. Customize any other configuration options according to your needs.

  6. Click Save Formula.

  7. Apply the highstate and confirm that it completes successfully.

  8. Check that the Prometheus interface loads correctly. In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, http://example.com:9090).

Procedure: Manually installing and configuring Prometheus
  1. On the monitoring server, install the golang-github-prometheus-prometheus package:

    zypper in golang-github-prometheus-prometheus
  2. Enable the Prometheus service:

    systemctl enable --now prometheus
  3. Check that the Prometheus interface loads correctly. In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, http://example.com:9090).

  4. Open the configuration file at /etc/prometheus/prometheus.yml and add this configuration information. Replace server.url with your SUSE Manager server URL and adjust username and password fields to match your SUSE Manager credentials.

    # {productname} self-health metrics
    scrape_configs:
    - job_name: 'mgr-server'
      static_configs:
        - targets:
          - 'server.url:9100'  # Node exporter
          - 'server.url:9187'  # PostgreSQL exporter
          - 'server.url:5556'  # JMX exporter (Tomcat)
          - 'server.url:5557'  # JMX exporter (Taskomatic)
          - 'server.url:9800'  # Taskomatic
        - targets:
          - 'server.local :80'    # Message queue
          labels:
            __metrics_path__: /rhn/metrics
    
    # Managed systems metrics:
    - job_name: 'mgr-clients'
      uyuni_sd_configs:
       - host: "http://server.url"
         username: "admin"
         password: "admin"
  5. Save the configuration file.

  6. Restart the Prometheus service:

    systemctl restart prometheus

For more information about the Prometheus configuration options, see the official Prometheus documentation at https://prometheus.io/docs/prometheus/latest/configuration/configuration/

Install Grafana

If your monitoring server is a SUSE Manager Salt client, you can install the Grafana package using the SUSE Manager Web UI. Otherwise you can download and install the package on your monitoring server manually.

Procedure: Installing Grafana via SUSE Manager Web UI
  1. In the SUSE Manager Web UI, open the details page of the system where Grafana is to be installed, and navigate to the Formulas tab.

  2. Check the Grafana checkbox to enable monitoring formulas, and click Save.

  3. Navigate to the Grafana tab in the top menu.

  4. In the Enable and configure Grafana section, enter the admin credentials you want to use to log in Grafana.

  5. On the Datasources section, make sure that the Prometheus URL field points to the system where Prometheus is running.

  6. Customize any other configuration options according to your needs.

  7. Click Save Formula.

  8. Apply the highstate and confirm that it completes successfully.

  9. Check that the Grafana interface is loading correctly. In your browser, navigate to the URL of the server where Grafana is installed, on port 3000 (for example, http://example.com:3000).

SUSE Manager provides pre-built dashboards for server self-health, basic client monitoring, and more. You can choose which dashboards to provision in the formula configuration page.

Procedure: Manually installing Grafana
  1. Install the grafana package:

    zypper in grafana
  2. Enable the Grafana service:

    systemctl enable --now grafana-server
  3. Check that the Grafana interface is loading correctly. In your browser, navigate to the URL of the server where Grafana is installed, on port 3000 (for example, http://example.com:3000).

monitoring grafana example

For more information on how to manually install and configure Grafana, see https://grafana.com/docs.

Configure SUSE Manager Monitoring

With SUSE Manager 4, you can enable the server to expose Prometheus self-health metrics, and also install and configure exporters on client systems.

Server Self Monitoring

The Server self-health metrics cover hardware, operating system and SUSE Manager internals. These metrics are made available by instrumentation of the Java application, combined with Prometheus exporters.

These exporter packages are shipped with SUSE Manager Server:

These exporter packages are shipped with SUSE Manager Proxy:

The exporter packages are pre-installed in SUSE Manager Server and Proxy, but their respective systemd daemons are disabled by default.

Procedure: Enabling Self Monitoring
  1. In the SUSE Manager Web UI, navigate to Admin  Manager Configuration  Monitoring.

  2. Click Enable services.

  3. Restart Tomcat and Taskomatic.

  4. Navigate to the URL of your Prometheus server, on port 9090 (for example, http://example.com:9090)

  5. In the Prometheus UI, navigate to menu:[Status > Targets] and confirm that all the endpoints on the mgr-server group are up.

  6. If you have also installed Grafana with the Web UI, the server insights will be visible on the SUSE Manager Server dashboard.

monitoring enable services

Only server self-health monitoring can be enabled using the Web UI. Metrics for a proxy are not automatically collected by Prometheus. To enable self-health monitoring on a proxy, you will need to manually install exporters and enable them.

Monitoring Managed Systems

Prometheus metrics exporters can be installed and configured on Salt clients using formulas. The packages are available from the SUSE Manager client tools channels, and can be enabled and configured directly in the SUSE Manager Web UI.

These exporters can be installed on managed systems:

When you have the exporters installed and configured, you can start using Prometheus to collect metrics from monitored systems. If you have configured your monitoring server with the Web UI, metrics collection will happen automatically.

Procedure: Configuring Prometheus Exporters on a Client
  1. In the SUSE Manager Web UI, open the details page of the client to be monitored, and navigate to the menu:Formulas tab.

  2. Check the Enabled checkbox on the Prometheus Exporters formula.

  3. Click Save.

  4. Navigate to the Formulas  Prometheus Exporters tab.

  5. Select the exporters you want to enable and customize arguments according to your needs.

  6. Click Save Formula.

  7. Apply the highstate.

monitoring configure formula

Monitoring formulas can also be configured for System Groups, by applying the same configuration used for individual systems inside the corresponding group.