Troubleshooting and Managing Keyfactor ACME

To add NLog customization with a config map:

1. Create an nlogconfig.yaml file with contents similar with the following:

   Important:  When editing the file, be sure to preserve the indenting exactly as shown. YAML requires a very specific file layout to function. If the indenting (multiples of two spaces) or layout is incorrect, you will receive an error when trying to apply the file.

   apiVersion: v1
   data:
     nlog.config: |
       <?xml version="1.0" encoding="utf-8" ?>
       <nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           autoReload="true">
         <extensions>
           <add assembly="NLog.Extensions.Logging" />
         </extensions>
         <targets>
           <target xsi:type="AsyncWrapper" name ="ConsoleAsync">
             <target xsi:type="Console" name="console" layout="${longdate} ${uppercase:${activityId}} ${logger} [${level}] - ${message}"/>
           </target>
         </targets>
         <rules>
           <logger name="*" minlevel="Debug" writeTo="ConsoleAsync">
             <filters defaultAction="Log">
               <when condition="starts-with('${logger}', 'Microsoft.') and level &lt; LogLevel.Warn" action="Ignore" />
               <when condition="ends-with('${logger}', 'DeferedAuthenticationHandler') and level &lt; LogLevel.Warn" action="Ignore" /><when condition="ends-with('${logger}', 'StatusService') and level &lt; LogLevel.Warn" action="Ignore" /><when condition="ends-with('${logger}', 'NonceService') and level &lt; LogLevel.Warn" action="Ignore" />
             </filters>
           </logger>
         </rules>
       </nlog>
   kind: ConfigMap
   metadata:
     name: nlog.config

   Customize the filters and logging level to meet your needs. The example filters shown above will hide messages that tend to be very verbose and not generally needed in the course of regular troubleshooting.

2. Create a ConfigMap containing the NLog configuration based on the yaml file you created. For example:

   sudo kubectl apply --filename /opt/kyf_acme/nlogconfig.yaml --namespace keyfactor-acme

3. Edit your values file to add a volume and volumeMount for the ConfigMap of the NLog.config. For example, the following values file section shows the example root trusts volume (see Installing Keyfactor ACME in Containers Under Kubernetes) and the NLog.confg.

   Note:  The Keyfactor ACME server installed in container under Kubernetes does not have a separate log file for API and Configuration logging; the NLog.config file under Configuration manages all logging.

   
   volumes:
     - name: root-cas
       configMap:
         name: ca-roots
         items:
           - key: ca-certificates.crt
             path: ca-certificates.crt
     - name: nlog
       configMap:
         name: nlog.config
         items:
           - key: nlog.config
             path: NLog.config
   volumeMounts:
     - name: root-cas
       mountPath: /etc/ssl/certs/ca-certificates.crt
       subPath: ca-certificates.crt
     - name: nlog
       mountPath: /app/Configuration/NLog.config
       subPath: NLog.config
       readOnly: false

4. Load the new values, referencing the deployment name, namespace, your customized values file, the helm chart, and version. For example:

   sudo helm upgrade Helm_Deployment_Name --namespace keyfactor-acme --values values.yaml oci://repo.keyfactor.com/charts/command/acme --version 1.0.0

To add NLog customization with a PersistentVolume:

1. On your Kubernetes server, create an NLog.config file containing any customized filter rules you wish to apply and the configuration for any log aggregation solution you wish to use. For example, the following filters will hide messages that tend to be very verbose and not generally needed in the course of regular troubleshooting and configure the log aggregation solution to do a push to Loki by Grafana:

   <?xml version="1.0" encoding="utf-8" ?>
   <nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           autoReload="true">
     <extensions>
       <add assembly="NLog.Extensions.Logging" />
       <add assemblyFile="/app/Configuration/Loki/NLog.Loki.dll" />
     </extensions>
     <targets>
       <target xsi:type="AsyncWrapper" name ="ConsoleAsync">
         <target xsi:type="Console" name="console" layout="${longdate} ${uppercase:${activityId}} ${logger} [${level}] - ${message}"/>
       </target>
       <target xsi:type="loki"
           name="loki"
           endpoint="${environment:LOKI_ENDPOINT_URL}"
           tenant="nlog-forwarding">
         <layout xsi:type="JsonLayout" includeEventProperties="true">
           <attribute name="level" layout="${level} "/>
           <attribute name="source" layout="${logger}" />
           <attribute name="message" layout="${message}" />
           <attribute name="exception" encode="false">
             <layout xsi:type="JsonLayout">
               <attribute name="type" layout="${exception:format=type}" />
               <attribute name="message" layout="${exception:format=message}" />
               <attribute name="stacktrace" layout="${exception:format=tostring}" />
             </layout>
           </attribute>
         </layout>
         <label name="level" layout="${level:lowercase=true}" />
         <label name="server" layout="${hostname:lowercase=true}" />
         <label name="app" layout="acme" />
         <label name="helm" layout="${environment:HELM_NAME}" />
       </target>
     </targets>
     <rules>
       <logger name="*" minlevel="Trace" writeTo="loki">
         <filters defaultAction="Log">
           <when condition="starts-with('${logger}', 'Microsoft.') and level &lt; LogLevel.Warn" action="Ignore" />
           <when condition="ends-with('${logger}', 'DeferedAuthenticationHandler') and level &lt; LogLevel.Warn" action="Ignore" />
           <when condition="ends-with('${logger}', 'StatusService') and level &lt; LogLevel.Warn" action="Ignore" />
           <when condition="ends-with('${logger}', 'NonceService') and level &lt; LogLevel.Warn" action="Ignore" />
         </filters>
       </logger>
       <logger name="*" minlevel="Info" writeTo="ConsoleAsync">
         <filters defaultAction="Log">
           <when condition="starts-with('${logger}', 'Microsoft.') and level &lt; LogLevel.Warn" action="Ignore" />
           <when condition="ends-with('${logger}', 'DeferedAuthenticationHandler') and level &lt; LogLevel.Warn" action="Ignore" />
           <when condition="ends-with('${logger}', 'StatusService') and level &lt; LogLevel.Warn" action="Ignore" />
           <when condition="ends-with('${logger}', 'NonceService') and level &lt; LogLevel.Warn" action="Ignore" />
         </filters>
       </logger>
     </rules>
   </nlog>

   You don’t necessarily need to set any filters at the NLog.config level if you’re using a log aggregation solution, but the option is available. Notice that this NLog.config file contains two logger name sections—one directing log messages to Loki and the other continuing to write log messages to the default Kubernetes standard out. The filters can be configured separately for these. The HELM_NAME and LOKI_ENDPOINT An endpoint is a URL that enables the API to gain access to resources on a server._URL are environment variables referenced from your Helm values.yaml file.

   Note:  If you load a custom NLog.config file, the logging level set with the workloadDefaults > logLevel parameter will be ignored in favor of the logging level(s) set in the NLog.config file.

2. Create a PersistentVolume in Kubernetes to contain your NLog.config file, the libraries it references, and any supporting files. The steps for this will vary depending on your Kubernetes implementation and the intended storage location used by your PersistentVolume.

   Tip:  You can create a PersistentVolume in the local file system for testing purposes as follows:

   1. Create a directory that your PersistentVolume will reference and which will contain the files to be mounted via the persistent volume. For example:

      mkdir /opt/files/nlog

   2. Create a YAML file to define the PersistentVolume similar to the following:

      apiVersion: v1
      kind: PersistentVolume
      metadata:
        name: nlog-pv
      spec:
        storageClassName: manual
        capacity:
          storage: 10Gi
      accessModes:
        - ReadWriteOnce
      hostPath:
        path: "/opt/files/nlog"

   3. Create the PersistentVolume based on the yaml file you created. For example:

      sudo kubectl apply --filename=/opt/kyf_acme/nlog-pv.yaml

      PersistentVolumes are cluster-scoped resources, meaning they are not associated with a specific namespace.

   4. Create a YAML file to define a PersistentVolumeClaim associated with the PersistentVolume similar to the following:

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: nlog-pvc
      spec:
        storageClassName: manual
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

   5. Create the PersistentVolumeClaim based on the yaml file you created. For example:

      sudo kubectl apply --filename=/opt/kyf_acme/nlog-pvc.yaml --namespace keyfactor-acme

      Unlike PersistentVolumes, PersistentVolumeClaims are associated with a specific namespace.

   6. Place the files that should be mounted in the container into the directory you specified in the file system. For example:

      /opt/files/nlog/NLog.config

      /opt/files/nlog/NLog.Loki.dll

   Important:  This method should not be used for a production PersistentVolume.

3. Edit your values file to add an additionalEnvironmentVariables section, if one does not already exist, with a reference to your HELM_NAME and LOKI_ENDPOINT_URL values. Add a volume and volumeMount for the PersistentVolume containing the NLog.config file, any required libraries, and any supporting files.

   For example, the following values file section shows the example root trusts volume (see Installing Keyfactor ACME in Containers Under Kubernetes) and the NLog.config.

   • Your PersistentVolume claim name should match the claimName referenced here.

   • The HELM_NAME value is a reference name you can use from within Grafana to query logs.

   • The LOKI_ENDPOINT_URL should be set to the HTTP push endpoint for Loki without any path (server name or IP and port only).

   additionalEnvironmentVariables:
   
     - name: HELM_NAME
       value: ACME
     - name: LOKI_ENDPOINT_URL
       value: http://appsrvr185-loki.keyexample.com:32000
   
   volumes:
     - name: root-cas
       configMap:
         name: ca-roots
         items:
           - key: ca-certificates.crt
             path: ca-certificates.crt
     - name: nlog-custom-volume
       persistentVolumeClaim:
         claimName: nlog-pvc
   volumeMounts:
     - name: root-cas
       mountPath: /etc/ssl/certs/ca-certificates.crt
       subPath: ca-certificates.crt
     - name: nlog-custom-volume
       mountPath: /app/Configuration/NLog.config
       subPath: NLog.config
     - name: nlog-custom-volume
       mountPath: /app/Configuration/Loki/NLog.Loki.dll
       subPath: NLog.Loki.dll

4. Load the new values, referencing the deployment name, namespace, your customized values file, the helm chart, and version. For example:

   sudo helm upgrade Helm_Deployment_Name --namespace keyfactor-acme --values values.yaml oci://repo.keyfactor.com/charts/command/acme --version 1.0.0