Troubleshooting
Troubleshooting
The following error conditions and general troubleshooting tips may be helpful in resolving issues with the Keyfactor Command server. Generally speaking, issues on installation or upgrade are often related to SQL connectivity or permissions. Certificate enrollment Certificate enrollment refers to the process by which a user requests a digital certificate. The user must submit the request to a certificate authority (CA). issues are often related to Kerberos configuration problems.
It is often helpful to enable debug logging on the server. For information on configuring this, see Editing NLog.
Once the logging is set at debug or trace level, it can be helpful to watch the logs live while activity is going on. There are tools on Windows with functionality similar to the Linux tail function to watch the log in real time. Notepad++, for example, has this functionality built in. Be sure to review all the logs that could be relevant. For example, installation and configuration messages are found in the configuration log. Messages related to using the Management Portal can be found in both the portal log and the Keyfactor API A set of functions to allow creation of applications. Keyfactor offers the Keyfactor API, which allows third-party software to integrate with the advanced certificate enrollment and management features of Keyfactor Command. log.
Some messages in the Keyfactor API and orchestrators API logs include a correlation ID that helps to identify log messages that originated from the same request. The correlation ID is a randomly generated GUID that often appears just after the date in the log entry (C282ACA1-DED5-4F2E-B83B-F3F9E865E371 in the following example) and is the same for all log messages for the given request until the request completes.
2022-09-13 04:51:18.6884 C282ACA1-DED5-4F2E-B83B-F3F9E865E371 CSS.CMS.Web.KeyfactorApi.Controllers.Enrollment.Enrollment2Controller [Trace] - Starting PFX Enrollment Process 2022-09-13 04:51:19.0477 C282ACA1-DED5-4F2E-B83B-F3F9E865E371 Keyfactor.Command.Workflows.Engine.WorkflowGraph [Error] - Invalid O provided: Value must be Key Example, Inc or Key Example.
Below are some possible errors you might encountered and some suggested troubleshooting tips or solutions.
You many encounter this error when trying to install or upgrade to Keyfactor Command version 10 or later:
2022-03-04 09:58:55.7262 CSS.CMS.Install.ConfigurationWizard.ViewModels.DatabaseViewModel [Error] - Unable to configure database 2022-03-04 09:58:55.9821 CSS.CMS.Install.ConfigurationWizard.ViewModels.DatabaseViewModel [Error] - An error occurred while preparing the database at CSS.CMS.Install.ConfigurationWizard.ViewModels.DatabaseViewModel.b(Object A_0, RunWorkerCompletedEventArgs A_1) A connection was successfully established with the server, but then an error occurred during the login process. (provider: SSL Provider, error: 0 - The certificate chain was issued by an authority that is not trusted
Keyfactor Command version 10 requires an encrypted connection to the SQL server. If the SQL server is not configured correctly to receive a secure connection (is not configured with a valid certificate that is trusted by the Keyfactor Command server), you may receive this message.
For information about configuring TLS TLS (Transport Layer Security) and its predecessor SSL (Secure Sockets Layer) are protocols for establishing authenticated and encrypted links between networked computers. for SQL server, see:
You may encounter this error on an enrollment when the CA A certificate authority (CA) is an entity that issues digital certificates. Within Keyfactor Command, a CA may be a Microsoft CA or a Keyfactor gateway to a cloud-based or remote CA. rejects the request. If the request is clearly not excessively long, review the request for invalid characters. Be sure to also check the default subject (see the Subject Format application setting on the Application Settings: Enrollment Tab and the subject defaults in certificate templates for both Configuring System-Wide Settings and Enrollment Defaults Tab). Quotation marks should not be used in the fields of the default subject except in the case where these are part of the desired subject value, as they are processed as literal values. This is a change from earlier versions of Keyfactor Command where quotation marks were used around fields containing embedded commas.
This error can also appear of the CA receives an enrollment request with no subject at all.
You may encounter this error either in the Keyfactor Command Management Portal or when submitting a Keyfactor API request. This error is typically not accompanied by any error in the Keyfactor Command logs. This error can occur if the IIS WebDAV Publishing feature is installed on the Keyfactor Command server. Keyfactor Command is not compatible with this IIS feature. Uninstall the WebDAV Publishing feature, reboot if required, and try your command again.
This error may appear during certificate enrollment against a certificate authority A certificate authority (CA) is an entity that issues digital certificates. Within Keyfactor Command, a CA may be a Microsoft CA or a Keyfactor gateway to a cloud-based or remote CA. running the Keyfactor CA Policy Module if the license for the policy module has expired. See License Expiration Monitoring and Rotation for license update information.
You may occasionally see error messages in the service log similar to the following if you are running Keyfactor Command in a redundant environment:
2023-01-12 16:55:00.0618 Keyfactor.LockProviders.SqlLockProvider [Error] - Unable to acquire lock on resource 'TimerServiceJob_https://ejbca3_keyother_com:8443 - CorpIssuingCA2 - Differencing 1/13/2023 12:55:00 AM'. 2023-01-12 16:55:00.0618 b [Error] - An error occured attempting to produce an instance of 'NoOverlapJobLoggingWrapper`1': Unable to acquire lock on resource 'TimerServiceJob_https://ejbca3_keyother_com:8443 - CorpIssuingCA2 - Differencing 1/13/2023 12:55:00 AM'. at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Keyfactor.LockProviders.SqlLockProvider.AcquireLock(LockType type, String key) at b.NewJob(TriggerFiredBundle bundle, IScheduler scheduler) 2023-01-12 16:55:00.0618 Quartz.Core.ErrorLogger [Error] - An error occurred instantiating job to be executed. job= 'DEFAULT.CASynchronizationService-56' 2023-01-12 16:55:00.0618 Quartz.Core.ErrorLogger [Error] - An error occurred instantiating job to be executed. job= 'DEFAULT.CASynchronizationService-56' 2023-01-12 16:55:00.0618 Quartz.Simpl.RAMJobStore [Info] - All triggers of Job DEFAULT.CASynchronizationService-56 set to ERROR state.
These message indicate that the Keyfactor Command Service on two different Keyfactor Command servers both attempted to run the same job at the same time and this server was unable to acquire a lock to run that job—the other server ran the job instead. This normally does not indicate any problem. If these errors occur frequently, it may be helpful to increase the timeout value for the job locking mechanism. To do this:
- On each Keyfactor Command server, open a text editor (e.g. Notepad) using the “Run as administrator” option.
-
In the text editor, browse to open the CMSTimerService.exe.config file. This file is located in the Service directory within the install directory, which is the following directory by default:
C:\Program Files\Keyfactor\Keyfactor Platform\Service -
In the CMSTimerService.exe.config file, locate the Keyfactor.TimerJobs.LockTimeout key in the appSettings section and increase the value appropriately for your environment. The value is specified in milliseconds, so the default value of 5000 indicates that the Keyfactor Command Service will attempt to acquire a lock on the job for 5 seconds before producing an error if the lock cannot be obtained.
Figure 426: Adjust the Keyfactor.TimerJobs.LockTimeout Value
- Restart the Keyfactor Command Service (see Enable and Start the Keyfactor Command Service in the Keyfactor Command Server Installation Guide) to read the updated configuration.
On the Validation tab of the certificate details you will sometimes see a fail result for some of the validation tests. The following are some possible reasons why this might occur.
-
If you see both Full Chain and CRL Online in a fail state, this generally indicates that you have not imported the root and/or intermediate certificate for the certificate into the appropriate store on the Keyfactor Command server (see Configure Certificate Chain Trusts for CAs in the Keyfactor Command Server Installation Guide).
-
If you see just CRL Online in a fail state, this generally indicates that the Certificate Revocation List A Certificate Revocation List (CRL) is a list of digital certificates that have been revoked by the issuing Certificate Authority (CA) before their scheduled expiration date and should no longer be trusted. (CRL A Certificate Revocation List (CRL) is a list of digital certificates that have been revoked by the issuing Certificate Authority (CA) before their scheduled expiration date and should no longer be trusted.) for the CA could not be reached.
Important: Because a “+” (plus sign) in a URL can represent either a space or a “+” Keyfactor Command has chosen to read “+” as a space. For CRL URLs that require a “+” (plus sign), rather than a space, replace plus signs in your CRL's URL with “%2B
”. Only replace the plus signs you don't wish to be treated as a space. -
If you see Revocation Status in a fail state but CRL Online is in a pass state, this can indicate that the CRL is accessible but expired, that the CRL is not fully configured, or that the Authority Information Access The authority information access (AIA) is included in a certificate--if configured--and identifies a location from which the chain certificates for that certificate may be retrieved. (AIA The authority information access (AIA) is included in a certificate--if configured--and identifies a location from which the chain certificates for that certificate may be retrieved.) for the CA has not been configured correctly or could not be reached.
For EJBCA CAs, CRLs and AIA need to be configured both at the CA level and at the certificate profile level. One way to do this is:
- AIA: Set the path to the AIA in the CA issuer Default URI field in the CA. You can find this on the Fetch CA certificates page of your EJBCA public web. Check both the Authority Information Access box and the Use CA defined CA issuer box in each certificate profile.
- CRL: Set the path to the CRL distribution point (CDP) in the Default CRL Distribution Point field in the CA. If appropriate for your environment, set also the Default CRL Issuer and/or Default Freshest CRL Distribution Point (delta CRLs). Check both the CRL Distribution Points box and the Use CA defined CRL Distribution Point box in each certificate profile.
Figure 427: Certificate Validation Fails for Full Chain and CRL Online
If SSL TLS (Transport Layer Security) and its predecessor SSL (Secure Sockets Layer) are protocols for establishing authenticated and encrypted links between networked computers. jobs are taking longer to complete than expected and you check the log on the orchestrator Keyfactor orchestrators perform a variety of functions, including managing certificate stores and SSH key stores. and find messages similar to the following:
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Keyfactor.WindowsAgent.Jobs.GenericJobExecutor`7.f.h()
2021-08-24 17:22:48.4463 Keyfactor.WindowsAgent.Jobs.SSL.SslDiscovery [Info] - Splitting endpoint result batch of 29 into smaller pieces and retrying
You may want to make modifications to the IIS maximum request size settings on your Keyfactor Command server to allow it to accept larger incoming pieces of content to streamline SSL scanning. You can do this using the configuration editor built into the IIS management console. Make the setting changes at the Default Web Site level (or other web site, if you installed your Keyfactor Command in an alternate web site). There are three settings to change:
- system.webServer/security/requestFiltering/requestLimits/maxAllowedContentLength
- system.webServer/serverRuntime/uploadReadAheadSize
- system.web/httpRuntime/maxRequestLength
Set each system.webServer value to at least 1,000,000 bytes for best SSL scanning performance. The default value of 4096 KB for the maxRequestLength will probably be sufficient for SSL scanning in most environments, but if it has been reduced in your environment, you may need to increase it. (The system.webServer values are set in bytes while the system.web values are set in kilobytes.) If you are scanning networks with especially large numbers of returned certificates, you may need to increase all these values. Monitor the orchestrator logs after modifying the values to confirm that you have achieved the desired effect.
Figure 428: Modify IIS Settings for SSL Scanning: maxAllowedContentLength
Figure 429: Modify IIS Settings for SSL Scanning:uploadReadAheadSize
Figure 430: Modify IIS Settings for SSL Scanning: maxRequestLength