Monitoring applications on Red Hat OpenShift Container Platform with Prometheus and Grafana

We can set up application monitoring on Red Hat OpenShift® Container Platform 4.19 by enabling monitoring for user-defined projects without the need to install an extra monitoring solution. We use Prometheus so that it monitors output metrics from the application while we use the Grafana dashboard to view the application metrics.

This solution deploys a second Prometheus operator instance inside the openshift-user-workload-monitoring namespace. This namespace is configured to monitor all namespaces except the openshift- prefixed namespaces, which are already monitored by the default platform monitoring for the cluster.

Ensure that the MicroProfile Metrics feature is built as part of our Liberty image.

Ensure that the Liberty image running the application is deployed to the Red Hat OpenShift Container Platform cluster inside your project namespace. Before we install application monitoring on Red Hat OpenShift Container Platform, ensure that an application that has a service endpoint for outputting metrics in Prometheus format is running:

  • The running application must be deployed to the Red Hat OpenShift Container Platform cluster inside your project namespace.

  • The Prometheus metrics endpoint must be exposed on the /metrics path.

We configure a monitoring stack. Then, we enable user-defined projects and deploy a service monitor to define the service endpoint that is monitored by the Prometheus instance. You create Grafana custom dashboards to view the application metrics and create custom queries. These Grafana custom dashboards must be deployed on their own Grafana instance using a community-powered Grafana Operator.

  1. Configure the monitoring stack.

    You create the cluster-monitoring and user-workload-monitoring-config configuration maps so we can enable monitoring for user-defined projects in the next step. When configuring the monitoring stack, a Grafana dashboard is created. However, you delete this Grafana dashboard because we cannot create custom queries since it is read-only.

  2. Enable monitoring for user-defined projects.

  3. Deploy a service monitor inside your project namespace to define the service endpoint that the Prometheus instance monitors and then enable Prometheus to scrape data.

    The WebsphereLibertyApplication custom resource creates a service monitor for you. Configure the service monitor so that it uses basic authentication credentials. These credentials are specified in the server.xml file for accessing the /metrics endpoint. Add the following basicAuth section to the monitoring definition in your WebsphereLibertyApplication custom resource, and replace the basic-auth name with your secret. 

      kind: WebSphereLibertyApplication
spec:
…
  monitoring:
    endpoints:
      - basicAuth:
  username:
    key: username
    name: basic-auth
  password:
    key: password
    name: basic-auth
        interval: 30s

  4. Confirm that your service is being monitored.

    Click Observe > Metrics and insert our own queries specific to the metrics for the application.

  5. Deploy a community-powered Grafana instance so that custom dashboards are created.

    Follow the Red Hat blog about Custom Grafana dashboards for Red Hat OpenShift Container Platform 4.

    Note that the Grafana Operator's apiVersion and Custom Resources were renamed since the creation of the Red Hat blog about Custom Grafana dashboards for Red Hat OpenShift Container Platform 4. When we are deploying the Grafana data source in the blog, use the following YAML instead. 

      kind: GrafanaDatasource
apiVersion: grafana.integreatly.org/v1beta1
metadata:
  name: prometheus-grafanadatasource
  namespace: my-grafana
spec:
  datasource:
    access: proxy
    editable: true
    isDefault: true
    jsonData:
      httpHeaderName1: 'Authorization'
      timeInterval: 5s
      tlsSkipVerify: true
    name: Prometheus
    secureJsonData:
      httpHeaderValue1: 'Bearer ${BEARER_TOKEN}'
    type: prometheus
    url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
instanceSelector:
    matchLabels:
      dashboards: grafana-a
  plugins:
    - name: grafana-clock-panel
      version: 1.3.0

    We can use sample Grafana dashboards for the custom Grafana dashboards. Look in the featureManager section of the server.xml file for either the mpMetrics feature or the umbrella microProfile feature to determine which dashboard to use.

    This Grafana instance is connected to Red Hat OpenShift monitoring in the openshift-monitoring namespace. CAUTION:Metrics queries that are run from Grafana dashboards might increase the load on the Prometheus service. Pay attention to the Prometheus CPU and memory usage when you deploy dashboards.

  6. Deploy the Grafana dashboard template to see all the services that are currently being monitored.

    Copy the following template into a file named grafana_dashboard.yaml. Ensure that the namespace in the .yaml file is configured to the namespace that you deployed in a previous step. 

      apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  labels:
    app: grafana
  name: template-dashboard
  namespace: prometheus-operator
spec:
  json: |
    {       "__inputs": [
 {    "name": "Prometheus",
   "label": "Prometheus",
   "description": "",
   "type": "datasource",
   "pluginId": "prometheus",
   "pluginName": "Prometheus"
       ],
      "__requires": [
 {    "type": "grafana",
   "id": "grafana",
   "name": "Grafana",
   "version": "5.2.0"
 },
 {    "type": "panel",
   "id": "graph",
   "name": "Graph",
   "version": "5.0.0"
 },
 {    "type": "datasource",
   "id": "prometheus",
   "name": "Prometheus",
   "version": "5.0.0"
 },
 {    "type": "panel",
   "id": "table",
   "name": "Table",
   "version": "5.0.0"
       ],
      "annotations": { "list": [
  {     "builtIn": 1,
    "datasource": "-- Grafana --",
    "enable": true,
    "hide": true,
    "iconColor": "rgba(0, 211, 255, 1)",
    "name": "Annotations & Alerts",
    "type": "dashboard"
  }
        ]
      },
      "editable": true,
      "gnetId": null,
      "graphTooltip": 0,
      "id": 1,
      "iteration": 1569353980677,
      "links": [],
      "panels": [
        {   "columns": [],
  "datasource": "Prometheus",
  "fontSize": "100%",
  "gridPos": {     "h": 6,
    "w": 24,
    "x": 0,
    "y": 2
  },
  "id": 26,
  "links": [],
  "options": {},
  "pageSize": null,
  "scroll": true,
  "showHeader": true,
  "sort": {     "col": 0,
    "desc": true
  },
  "styles": [
    {       "alias": "Time",
      "dateFormat": "YYYY-MM-DD HH:mm:ss",
      "pattern": "Time",
      "type": "date"
    },
    {       "alias": "Namespace",
      "pattern": "namespace",
      "type": "custom"
    },
    {       "alias": "Service",
      "pattern": "service",
      "type": "custom"
    },
    {       "alias": "Endpoint",
      "pattern": "endpoint",
      "type": "custom"
    },
    {       "alias": "",
      "mappingType": 1,
      "pattern": "Value",
      "type": "hidden",
      "unit": "short"
    }
  ],
  "targets": [
    {       "expr": "count(up) by (namespace, service, endpoint)",
      "format": "table",
      "instant": true,
      "intervalFactor": 1,
      "refId": "A"
    }
  ],
  "title": "Endpoints",
  "transform": "table",
  "type": "table"
        }
      ],
      "refresh": "10s",
      "schemaVersion": 19,
      "style": "dark",
      "tags": [],
      "time": { "from": "now-15m",
        "to": "now"
      },
      "timepicker": { "refresh_intervals": [
  "1m",
  "5m",
  "15m",
  "1h",
  "1d"
        ]
      },
      "title": "Template-Dashboard",
      "uid": "Template-Dashboard"
    }

  7. View the Grafana dashboard to use the application metrics that Prometheus gathered.

    Click Networking > Routes and then, click the link to the Grafana dashboard.