Analyzing application Logs on Red Hat OpenShift Container Platform with Loki, Vector, and the RHOCP Cluster Observability Operator

We can use Logging 6.0 with Loki, Red Hat OpenShift® Logging, and Cluster Observability operators to manage log storage, collection, and visualization on Red Hat OpenShift Container Platform 4.19.

Pod processes running in Kubernetes frequently produce application logs. To effectively manage the application log data and avoid loss of log data occurs when a pod stops, deploy log aggregation tools on the Kubernetes cluster. Log aggregation tools help you persist, search, and visualize the log data that is gathered from the pods across the cluster. For more information on the new changes introduced in Logging 6.0, see Upgrade to Logging 6.0.

Set up LokiStack and Cluster Log Forwarding

To set up LokiStack and the ClusterLogForwarder in the openshift-logging namespace, see Logging 6.0. After the LokiStack and the ClusterLogForwarder deployments are completed, the following pods run in the openshift-logging namespace:

logging-loki-compactor-0    1/1     Running   0  3h45m
logging-loki-distributor-74cb8f8854-5fr9t      1/1     Running   0  3h45m
logging-loki-distributor-74cb8f8854-p88cx      1/1     Running   0  3h45m
logging-loki-gateway-78888d6c56-428w9  2/2     Running   0  3h45m
logging-loki-gateway-78888d6c56-6wl5b  2/2     Running   0  3h45m
logging-loki-index-gateway-0   1/1     Running   0  3h45m
logging-loki-index-gateway-1   1/1     Running   0  3h45m
logging-loki-ingester-0     1/1     Running   0  3h45m
logging-loki-ingester-1     1/1     Running   0  3h44m
logging-loki-querier-57c8bd8c75-vcc4t  1/1     Running   0  3h45m
logging-loki-querier-57c8bd8c75-wwlrd  1/1     Running   0  3h45m
logging-loki-query-frontend-6bbb599859-rmsmk   1/1     Running   0  3h45m
logging-loki-query-frontend-6bbb599859-zhcfj   1/1     Running   0  3h45m
cluster-logging-operator-7b6bc9c48-2mjx2       1/1     Running   0  4h22m
collector-59dxs     1/1     Running   0  3h16m
collector-7p6h7     1/1     Running   0  3h16m
collector-jbwkn     1/1     Running   0  3h16m
collector-ntzm4     1/1     Running   0  3h16m
collector-vzlm4     1/1     Running   0  3h16m
collector-z75z5     1/1     Running   0  3h16m

For more information on configuring LokiStack storage, see Storing logs with LokiStack. For more information on configuring log forwarding, see Configure log forwarding.

Access object storage

We must have existing object storage to configure LokiStack. Loki Operator supports AWS S3, Azure, GCS, Minio, OpenShift Data Foundation, and Swift as options for LokiStack object storage. Create a logging-loki-s3 secret inside the openshift-logging namespace containing the fields needed for LokiStack to access your object storage. The following sample command creates a secret that allows access to an OpenShift Data Foundation managed S3 bucket that runs inside the cluster.

  oc create secret generic logging-loki-s3 \
    --from-literal=access_key_id=key \
    --from-literal=access_key_secret=secret \
    --from-literal=bucketnames=bucket-name \
    --from-literal=endpoint=endpoint \
    --namespace openshift-logging

Parsing JSON container logs

By default, if container logs are being output in JSON format, they are nested inside the Vector JSON document's message field. To solve this problem, add the following YAML to the ClusterLogForwarder custom resource:

 kind: ClusterLogForwarder
 spec:
   ...
   filterRefs:
   - parse-json
   ...
   pipelines:
   - name: default-logstore
     ...
     filters:
     - name: parse-json
       type: parse

The nested JSON container log in to a separate structured field inside the Vector JSON document, where the individual fields from the JSON container log can be accessed as structured. <field_name>.

Visualizing your logs using the Cluster Observability Operator's Logging UI plug-in

Add the label logFormat: liberty to our WebSphereLibertyApplication custom resource. This label filters through the application pod logs in Observe > Logs.

apiVersion: liberty.websphere.ibm.com/v1
kind: WebSphereLibertyApplication
metadata:
  name: <your-liberty-app>
  labels:
    logFormat: liberty

In the OpenShift Container Platform web console, go to Observe > Logs to view the application logs.

The following image shows the Logs page where we can view the application logs. Use the following LogQL query to filter for and format the application pod logs.

{ log_type="application" } | json | kubernetes_labels_logFormat="liberty" | line_format "[{{.structured_loglevel}}] {{.structured_message}}"

The following image shows the application logs filtered by the LogQL query.

To see other structured.* formatted individual fields from the application pod log, expand a log entry.

The following image shows the expanded application log entries.

We can now ingest, forward, and view the application logs using LokiStack, Vector, and the Cluster Observability Operator's Logging UI plug-in.