Troubleshooting, Services, and Logs

Below are common issues with their corresponding resolution or investigation techniques. Additional details about your installation can also be found in the Windows Services settings on your server and in the AnyCAGateway REST log files.

Note:  These instructions apply only to Windows installations.

Open Windows Services on the AnyCAGateway REST and scroll down to locate the Keyfactor AnyCAGateway Servce (Name). The value in parentheses will be (Default) unless you supplied a different value in the -ServiceSuffix parameterClosed A parameter or argument is a value that is passed into a function in an application. (see -Service Suffix). If the service has not started, start the service.

Figure 773: AnyCAGateway REST Service

You can also track the path to the gateway installation on the services properties dialog.

Figure 774: Path to Executable

Note:  These instructions apply only to Windows installations.

To view the saved configuration and encrypted installation data:

  1. Navigate to the location of the AnyCAGateway REST configuration folder, by default at:

    C:\Program Files\Keyfactor\Keyfactor AnyCA Gateway\AnyGatewayREST\net8.0\configuration

  2. Open the folder to view the saved configuration and encrypted installation data, including:

    Figure 775: Configuration Files for a Client Certificate Authentication Implementation

When troubleshooting, both the AnyCA Gateway logs and the Keyfactor Command logs can be helpful resources.

Installations on Windows
Keyfactor Command Logs:
AnyCAGateway REST Logs:

By default, the AnyCAGateway REST installed on Windows generates logs at the Info logging level, stores logs for two days before deleting them, and places its log files in the following directory:

C:\Program Files\Keyfactor\Keyfactor AnyCA Gateway\AnyGatewayREST\net8.0\logs

Figure 776: AnyCAGateway REST Available Logs

If you wish to change these defaults:

  1. On the AnyCAGateway REST server where you wish to adjust logging, open a text editor (e.g. Notepad) using the “Run as administrator” option.

  2. In the text editor, browse to open the Nlog.config file on the configuration directory under the directory in which you installed the AnyCAGateway REST. By default, this is the following directory:

    C:\Program Files\Keyfactor\Keyfactor AnyCA Gateway\AnyGatewayREST\net8.0\Configuration
  3. Your Nlog.config file may have a slightly different layout than shown here, but it will contain the five fields highlighted in the below figure. The fields you may wish to edit are:

    • The logDirectory variable:

      name="logDirectory" value="logs/"

      Example new value:

      name="logDirectory" value="D:\GatewayLogs\logs/"
      Important:  If you choose to change the path for storage of the log files, you will may need to create the new directory (e.g. D:\GatewayLogs) and grant the service account under which the CA gateway service runs full control permissions on this directory, depending on what service account the gateway as running as. If the gateway is running as Network Service, it should have sufficient permissions to create the directory and switch the location to a local directory without this.

    • The path and file name of the active CA gateway log file:

      fileName="${logDirectory}/CA_Gateway_Log.txt"

    • The path and file name of previous days' CA gateway log files:

      archiveFileName="${logDirectory}/CA_Gateway_Log_Archive_{#}.txt"

      The CA gateway rotates log files daily and names the previous files using this naming convention.

    • The number of archive files to retain before deletion:

      maxArchiveFiles="2"

    • The level of log detail that should be generated:

      name="*" minlevel="Info"

      The default Info level logs error and some informational data but at a minimal level to avoid generating large log files. For troubleshooting, it may be desirable to set the logging level to Debug or Trace. Available log levels (in order of increasing verbosity) are:

      • OFF – No logging

      • FATAL – Log severe errors that cause early termination

      • ERROR – Log severe errors and other runtime errors or unexpected conditions that may not cause early termination

      • WARN – Log errors and use of deprecated APIs, poor use of APIs, almost errors, and other runtime situations that are undesirable or unexpected but not necessarily wrong

      • INFO – Log all of the above plus runtime events (startup/shutdown)

      • DEBUG – Log all of the above plus detailed information on the flow through the system

      • TRACE – Maximum log information—this option can generate VERY large log files

Figure 777: NLog.config File for the AnyCAGateway REST

Installations in Containers under Kubernetes

By default, the AnyCAGateway REST installed in containers under Kubernetes outputs logging information to the standard Kubernetes logging system and generates file system logs at the Info logging level. The logging level can be configured in the custom values file with the workloadDefaults > logLevel parameter. For a complete list of the parameters, see Values File Settings for Containers Under Kubernetes. The supported log levels are:

  • OFF

    No logging

  • FATAL

    Log severe errors that cause early termination

  • ERROR

    Log severe errors and other runtime errors or unexpected conditions that may not cause early termination

  • WARN

    Log errors and use of deprecated APIs, poor use of APIs, almost errors, and other runtime situations that are undesirable or unexpected but not necessarily wrong

  • INFO

    Log all of the above plus runtime events (startup/shutdown)

  • DEBUG

    Log all of the above plus detailed information on the flow through the system

  • TRACE

    Maximum log information—this option can generate VERY large log files

The maximum size and number of logs to retain are controlled with the following Kubernetes configuration settings:

  • containerLogMaxWorkers

  • containerLogMonitorInterval

For more information, see:

https://kubernetes.io/docs/concepts/cluster-administration/logging/#log-rotation

In addition, log customizations can be added using a custom nlog.config file to add, for example, log filters or reference third-party libraries or other files that support log aggregation.

For container installations under Kubernetes, custom NLog configuration can be added with either a config map or a persistent volume. If you’re just adding filters, a config map is the simpler route, though with a persistent volume subsequent changes to the nlog.config file will be loaded dynamically and will not require a restart or upgrade of the containers. If you need to include libraries or other files for log aggregation, you’ll need a persistent volume.

To add NLog customization with a config map:

  1. Create an nlogconfig.yaml file with contents similar with the following:

    Important:  When editing the file, be sure to preserve the indenting exactly as shown. YAML requires a very specific file layout to function. If the indenting (multiples of two spaces) or layout is incorrect, you will receive an error when trying to install.
    apiVersion: v1
data:
  nlog.config: |
    <?xml version="1.0" encoding="utf-8" ?>
    <nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        autoReload="true">
      <extensions>
        <add assembly="NLog.Extensions.Logging" />
      </extensions>
      <targets>
        <target xsi:type="AsyncWrapper" name ="ConsoleAsync">
          <target xsi:type="Console" name="console" layout="${longdate} ${uppercase:${activityId}} ${logger} [${level}] - ${message}"/>
        </target>
      </targets>
      <rules>
        <logger name="*" minlevel="Debug" writeTo="ConsoleAsync">
          <filters defaultAction="Log">
            <when condition="starts-with('${logger}', 'CertTransBlobEncoder') and level &lt; LogLevel.Info" action="Ignore" />
            <when condition="starts-with('${logger}', 'CertificateRequestDecode') and level &lt; LogLevel.Info" action="Ignore" />
            <when condition="ends-with('${logger}', 'EntrustCertificateRecord') and level &lt; LogLevel.Info" action="Ignore" />
          </filters>
        </logger>
      </rules>
    </nlog>
kind: ConfigMap
metadata:
  name: nlog.config

    Customize the filters and logging level to meet your needs. The example filters shown above will hide messages that tend to be very verbose and not generally needed in the course of regular troubleshooting.

    Note:  If you load a custom nlog.config file, the logging level set with the workloadDefaults > logLevel parameter will be ignored in favor of the logging level(s) set in the nlog.config file.

  2. Create a configmap containing the NLog configuration based on the yaml file you created. For example:

    sudo kubectl apply --filename /opt/kyf_gateway/nlogconfig.yaml --namespace keyfactor-gateway

  3. Edit your values file to add an additionalEnvironmentVariables section, if one does not already exist, with a reference to the path to your custom NLog.config file in the container and a volume and volumeMount for the config map of the NLog.config. For example, the following values file section shows the example root trusts volume (see Install Keyfactor Command in Containers Under KubernetesInstall AnyCAGateway REST in Containers Under Kubernetes) and the NLog.confg

    additionalEnvironmentVariables:
  - name: NLogConfigFile
    value: /app/Configuration/custom-nlog/nlog.config

volumes:
  - name: root-cas
    configMap:
      name: ca-roots
      items:
        - key: ca-certificates.crt
          path: ca-certificates.crt
  - name: nlog
    configMap:
      name: nlog.config
      items:
        - key: nlog.config
          path: nlog.config
volumeMounts:
  - name: root-cas
    mountPath: /etc/ssl/certs/ca-certificates.crt
    subPath: ca-certificates.crt
  - name: nlog
    mountPath: /app/Configuration/custom-nlog/nlog.config
    subPath: nlog.config
    readOnly: false

  4. Load the new values, referencing the deployment name, namespace, your customized values file, the helm chart, and version. For example:

    sudo helm upgrade Helm_Deployment_Name --namespace keyfactor-gateway --values values-local.yaml oci://repo.keyfactor.com/charts/command/anygateway-rest --version 1.0.1
Tip:  If you opt to use a persistent volume to configure your nlog.config, you may make subsequent changes to the nlog.config file stored in the persistent volume location without needing to restart or upgrade the containers. The NLog updates will be loaded dynamically. If your persistent volume is stored in a network file system, additional configuration settings are required for dynamic update.

To add NLog customization with a persistent volume:

  1. On your Kubernetes server, create an nlog.config file containing any customized filter rules you wish to apply and the configuration for any log aggregation solution you wish to use. For example, the following filters will hide the regular messages that appear in the log messages as the connection to the SQL server is checked to make sure it’s responding as expected and configure the log aggregation solution to do a push to Loki by Grafana:

    <?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        autoReload="true">
  <extensions>
    <add assembly="NLog.Extensions.Logging" />
    <add assemblyFile="/app/Configuration/Loki/NLog.Loki.dll" />
  </extensions>
  <targets>
    <target xsi:type="AsyncWrapper" name ="ConsoleAsync">
      <target xsi:type="Console" name="console" layout="${longdate} ${uppercase:${activityId}} ${logger} [${level}] - ${message}"/>
    </target>
    <target xsi:type="loki"
        name="loki"
        endpoint="${environment:LOKI_ENDPOINT_URL}"
        tenant="nlog-forwarding">
      <layout xsi:type="JsonLayout" includeEventProperties="true">
        <attribute name="level" layout="${level} "/>
        <attribute name="source" layout="${logger}" />
        <attribute name="message" layout="${message}" />
        <attribute name="exception" encode="false">
          <layout xsi:type="JsonLayout">
            <attribute name="type" layout="${exception:format=type}" />
            <attribute name="message" layout="${exception:format=message}" />
            <attribute name="stacktrace" layout="${exception:format=tostring}" />
          </layout>
        </attribute>
      </layout>
      <label name="level" layout="${level:lowercase=true}" />
      <label name="server" layout="${hostname:lowercase=true}" />
      <label name="app" layout="gateway" />
      <label name="helm" layout="${environment:HELM_NAME}" />
    </target>
  </targets>
  <rules>
    <logger name="*" minlevel="Trace" writeTo="loki">
      <filters defaultAction="Log">
        <when condition="starts-with('${logger}', 'CertTransBlobEncoder') and level &lt; LogLevel.Info" action="Ignore" />
        <when condition="starts-with('${logger}', 'CertificateRequestDecode') and level &lt; LogLevel.Info" action="Ignore" />
        <when condition="ends-with('${logger}', 'EntrustCertificateRecord') and level &lt; LogLevel.Info" action="Ignore" />
      </filters>
    </logger>
    <logger name="*" minlevel="Info" writeTo="ConsoleAsync">
      <filters defaultAction="Log">
        <when condition="starts-with('${logger}', 'CertTransBlobEncoder') and level &lt; LogLevel.Info" action="Ignore" />
        <when condition="starts-with('${logger}', 'CertificateRequestDecode') and level &lt; LogLevel.Info" action="Ignore" />
        <when condition="ends-with('${logger}', 'EntrustCertificateRecord') and level &lt; LogLevel.Info" action="Ignore" />
      </filters>
    </logger>
  </rules>
</nlog>

    Customize the filters and logging level(s) to meet your needs. The example filters shown above will hide messages that tend to be very verbose and not generally needed in the course of regular troubleshooting. You don’t necessarily need to set any filters at the nlog.config level if you’re using a log aggregation solution, but the option is available. Notice that this nlog.config file contains two logger name sections—one directing log messages to Loki and the other continuing to write log messages to the default Kubernetes standard out. The filters can be configured separately for these. The HELM_NAME and LOKI_ENDPOINTClosed An endpoint is a URL that enables the API to gain access to resources on a server._URL are environment variables referenced from your Helm values.yaml file.

    Note:  If you load a custom nlog.config file, the logging level set with the workloadDefaults > logLevel parameter will be ignored in favor of the logging level(s) set in the nlog.config file.

  2. Create a persistent volume in Kubernetes to contain your nlog.config file, the libraries it references, and any supporting files. The steps for this will vary depending on your Kubernetes implementation and the intended storage location used by your persistent volume.

  3. Edit your values file to add an additionalEnvironmentVariables section, if one does not already exist, with a reference to the path to your custom NLog.config file in the container and your HELM_NAME and LOKI_ENDPOINT_URL values. Add a volume and volumeMount for the persistent volume containing the nlog.config file, any required libraries, and any supporting files. For example, the following values file section shows the example root trusts volume (see Install Keyfactor Command in Containers Under KubernetesInstall AnyCAGateway REST in Containers Under Kubernetes) and the NLog.config.

    Your persistent volume claim name should match the claimName referenced here. The HELM_NAME value is a reference name you can use from within Grafana to query logs. The LOKI_ENDPOINT_URL should be set to the HTTP push endpoint for Loki without any path (server name or IP and port only).

    The NLogPoller__Enabled and NLogPoller__PollingInterval settings are only needed if your persistent volume is in a network file system (such as an Azure file share). These enable periodic checks for updates to the nlog.config file to allow for dynamic updates of the configuration. By default, this option is disabled. The polling interval sets the number of seconds between checks for updates. The default is 10 seconds. Local persistent volumes are updated dynamically without the need for these settings.

    additionalEnvironmentVariables:
  - name: NLogConfigFile
    value: /app/Configuration/Loki/nlog.config
  - name: HELM_NAME
    value: Gateway
  - name: LOKI_ENDPOINT_URL
    value: http://appsrvr185-loki.keyexample.com:32000
  - name: NLogPoller__Enabled
    value: "true"
  - name: NLogPoller__PollingInterval
    value: "15"

volumes:
  - name: root-cas
    configMap:
      name: ca-roots
      items:
        - key: ca-certificates.crt
          path: ca-certificates.crt
  - name: nlog-custom-volume
    persistentVolumeClaim:
      claimName: nlog-custom-pvc
volumeMounts:
  - name: root-cas
    mountPath: /etc/ssl/certs/ca-certificates.crt
    subPath: ca-certificates.crt
  - name: nlog-custom-volume
    mountPath: /app/Configuration/Loki

  4. Load the new values, referencing the deployment name, namespace, your customized values file, the helm chart, and version. For example:

    sudo helm upgrade Helm_Deployment_Name --namespace keyfactor-gateway --values values-local.yaml oci://repo.keyfactor.com/charts/command/anygateway-rest --version 1.0.1
Tip:  To view the logs in Grafana, you can use a query similar to this:
{app="gateway"} | json | line_format "{{.source}} {{.level_extracted}} {{.message}}" != "-EVENT" | server=~"restgateway1--anygateway-rest-portal.*"

The app label is defined in the nlog.config file. The server name is made up of the Helm deployment name, the application, and some additional IDs. The above query requests the app where the value is gateway, formats the returned results in a manner similar to expected nlog output, excludes messages tagged -EVENT, and includes only messages for the portal application for the Helm deployment restgateway1. Given a Helm deployment name of restgateway1, possible server tags would start with:

  • restgateway1-anygateway-rest-portal
  • restgateway1-anygateway-rest-dbmanagement (this container is short-lived)
Tip:  The AnyCAGateway REST will log Info-level messages when it starts up that contain the following information:
  • The version of AnyCAGateway REST being installed.
  • Any loaded AnyCAPlugin, with the version of plugin implementation.
  • The database connection string. If a password is part of the connection string, it will be masked out.

  • If the default AnyCAPlugin is in use, the service will log a message and warn the user to configure a plugin (see Configure the AnyCAPlugin).

    2023-08-30 12:15:13.4538 Keyfactor.AnyGatewayCore.NoOpAnyCAPlugin [Warn] - The current AnyCAPlugin is a placeholder. Please configure a AnyCAPlugin in the manifest.json.

    2023-08-30 12:15:13.4538 Keyfactor.AnyGatewayREST.Filters.APIExceptionFilter [Error] - Keyfactor encountered an error processing the request. Please view the server log for more detail. (Error Code: 0x80004001)

    2023-08-30 12:15:13.4538 Keyfactor.AnyGatewayREST.Filters.APIExceptionFilter [Error] - The method or operation is not implemented.

Note:  CA synchronization error handling and logging:
Tip:  For identity provider logging in the AnyCA Gateway log files, log messages will require either the DEBUG or TRACE log levels. In the event that there is an error when adding/updating an Identity provider, the log will log a WARN message indicating there was an issue.

The following are some error messages you might encounter and some possible resolutions.