Troubleshooting

In this snippet you see a successful inventory showing keys found for the Linux users ginag and svc_greenchicken and a logon found for the Linux user zadams with no key found. You see that the server is configured in inventory and publish policy mode, since after performing the inventory the server went through the steps of publishing logons and keys. Details about these are not written to the log.

Tue Aug 11 18:07:45 UTC 2020 [Debug]: Sending request to 'https://keyfactor.keyexample.com/KeyfactorAgents/SshSync/1/Configure' with payload '{"SessionToken": "5451f7aa-4fd5-4bf5-a563-2e4f7bd3ed3f", "JobId": "b835bde8-8174-447a-b351-810e582148c0"}'
Tue Aug 11 18:07:45 UTC 2020 [Debug]: Configure Response for job with id 'b835bde8-8174-447a-b351-810e582148c0': {"Hostname":"appsrvr79.keyexample.com","InventoryCompleteEndpoint":"/SshSync/1/InventoryComplete", "Port":22,"AuditId":7642,"JobCancelled":false,"Result":{"Status":1,"Error":null}}
Tue Aug 11 18:07:46 UTC 2020 [Debug]: Using sshd_config file '/etc/ssh/sshd_config' on server 'appsrvr79.keyexample.com' for job with id 'b835bde8-8174-447a-b351-810e582148c0'
Tue Aug 11 18:07:46 UTC 2020 [Info]: Beginning local inventory job on server 'appsrvr79.keyexample.com' for job with id 'b835bde8-8174-447a-b351-810e582148c0'
Tue Aug 11 18:07:49 UTC 2020 [Debug]: Sending request to 'https://keyfactor.keyexample.com/KeyfactorAgents/SshSync/1/InventoryComplete' with payload '{"Status":2,"Results": [{
"user": "ginag",
"lastlogon": "",
"keys": [ "ssh-rsa AAAAB3NzaC1yc2EAAAA[truncated for display purposes]9M5vl6f Gina G. Gant" ]
},{
"user": "zadams",
"lastlogon": "",
"keys": []
},{
"user": "svc_greenchicken",
"lastlogon": "",
"keys": [ "ssh-rsa AAAAB3NzaC1yc2EAAAAD[truncated for display purposes]vicWhZOd John W. Smith" ]
}],"SessionToken": "5451f7aa-4fd5-4bf5-a563-2e4f7bd3ed3f","JobId": "b835bde8-8174-447a-b351-810e582148c0"}'
Tue Aug 11 18:07:49 UTC 2020 [Debug]: Inventory Complete Response for job with id 'b835bde8-8174-447a-b351-810e582148c0' on server 'appsrvr79.keyexample.com': {"SshDesiredState":[{"Username":"ginag","Keys":["ssh-rsa AAAAB3NzaC1yc2EAAAA[truncated for display purposes]9M5vl6f Gina G. Gant"]},{"Username":"zadams","Keys":[]},{"Username":"svc_greenchicken","Keys":["ssh-rsa AAAAB3NzaC1yc2EAAAAD[truncated for display purposes]vicWhZOd John W. Smith"]}],"Result":{"Status":1,"Error":null}}
Tue Aug 11 18:07:49 UTC 2020 [Info]: Enforcing publish policy on server 'appsrvr79.keyexample.com' for job with id 'b835bde8-8174-447a-b351-810e582148c0'
Tue Aug 11 18:07:52 UTC 2020 [Info]: Publishing logons on local server 'appsrvr79.keyexample.com' for job with id 'b835bde8-8174-447a-b351-810e582148c0'
Tue Aug 11 18:07:52 UTC 2020 [Info]: Published logons successfully on server 'appsrvr79.keyexample.com' for job 'b835bde8-8174-447a-b351-810e582148c0'
Tue Aug 11 18:07:52 UTC 2020 [Info]: Publishing keys on local server 'appsrvr79.keyexample.com' for job with id 'b835bde8-8174-447a-b351-810e582148c0'
Tue Aug 11 18:07:54 UTC 2020 [Info]: Published keys successfully on server 'appsrvr79.keyexample.com' for job 'b835bde8-8174-447a-b351-810e582148c0'
Tue Aug 11 18:07:54 UTC 2020 [Debug]: Sending request to 'https://keyfactor.keyexample.com/KeyfactorAgents/SshSync/1/Complete' with payload '{"SessionToken": "5451f7aa-4fd5-4bf5-a563-2e4f7bd3ed3f", "JobId": "b835bde8-8174-447a-b351-810e582148c0", "Status": 2}'
Tue Aug 11 18:07:54 UTC 2020 [Info]: Execution of 'b835bde8-8174-447a-b351-810e582148c0' on server 'appsrvr79.keyexample.com' complete.

During installation of the orchestrator, a local Linux user account should be created automatically as an identity under which the orchestrator service will operate. This allows the orchestrator to run as a non-root user. On servers on which you install the orchestrator directly, the following Linux user account is created:

On servers configured as remote control targets, the following Linux user account is created:

You can validate that the user has been created and has the correct configuration be reviewing the /etc/passwd file.

In a command shell, output the content of the /etc/passwd file to the screen:

In the output from this command, look for the entry for the keyfactor-bash or keyfactor-bash-orchestrator-svc user. It will look similar to one of these:

On the remote control target server, you should find an entry in the sshd_config file that directs the service account logon over to the install path for the client to find the authorized_keys file for the service account user, like so:

On both the orchestrator and remote control target servers, you should find a file in the /etc/sudoer.d directory named for the service name of the orchestrator or remote control target user (keyfactor-bash or keyfactor-bash-orchestrator-svc) and containing a list of commands the orchestrator is allowed to execute as root. For example:

The orchestrator connects to the remote control targets it is managing using SSH with a public key In asymmetric cryptography, public keys are used together in a key pair with a private key. The private key is retained by the key's creator while the public key is widely distributed to any user or target needing to interact with the holder of the private key. pair. On the orchestrator, the key pair In asymmetric cryptography, public keys are used together in a key pair with a private key. The private key is retained by the key's creator while the public key is widely distributed to any user or target needing to interact with the holder of the private key. is stored in the .ssh directory under the directory where the orchestrator is installed. By default, this is:

Both the private key Private keys are used in cryptography (symmetric and asymmetric) to encrypt or sign content. In asymmetric cryptography, they are used together in a key pair with a public key. The private or secret key is retained by the key's creator, making it highly secure. (id_rsa) and public key (id_rsa.pub) are found here.

In a command shell, output the content of the public key file to the screen:

On the remote control target, the public key of the key pair is stored in the authorized_keys file for the remote control target service account, which is found in the remote control install path. By default, this is:

In a command shell, output the content of the authorized_keys file to the screen:

Compare the public key string from the remote control target authorized_keys file to the public key string from the orchestrator id_rsa.pub file. They should match exactly. If they do not match, the remote control target is not using the correct public key, which will cause connection attempts made to it from the orchestrator to fail.

If the orchestrator is managing more than one server (remote control targets), it can be difficult to interpret the logs, because the orchestrator operates in a multi-threaded manner and log messages for jobs with different servers will be mixed together. Find a message related to the job you're interested in and look for the ID for that job. Then look for all other messages referencing that ID.

Look for error messages in the log. These should appear with the word Error in brackets just after the date like so:

Tue Aug 11 19:14:33 UTC 2020 [Error]: Error occurred during job with id 'b835bde8-8174-447a-b351-810e582148c0' on server 'appsrvr79.keyexample.com': An error occurred attempting to configure the job 'b835bde8-8174-447a-b351-810e582148c0'

This particular message doesn't tell you very much except that this job was unable to complete for some reason. If you look at the debug messages that appear immediately before and after the error message, they may provide more information.

This message indicates that the orchestrator was unable to make an SSH connection to the remote control target named in the message:

Mon Aug 10 23:36:10 UTC 2020 [Error]: Error occurred during job with id '3f04f552-05fd-4c90-b3b1-edeec70878bb' on server 'appsrvr80.ubuntu.keyexample.com': Unable to connect to 'appsrvr80.ubuntu.keyexample.com' on port '22' via SSH

This could happen for a number of reasons. Perhaps the hostname configured for the remote target is incorrect. Perhaps the public key on the remote target is incorrect. It can be helpful in this case to check the Linux syslog on the orchestrator for more context on the message. For example, this set of messages from the Linux syslog reveals that the public key on the target is invalid in some fashion:

Aug 11 13:03:04 appsrvr158 keyfactor-bash[29417]: Testing 'keyfactor-bash-orchestrator-svc' on server 'appsrvr80.keyexample.com' via SSH for job with id 'eeabd541-b9d2-46d2-a215-9cb99fed4adc'...
Aug 11 13:03:04 appsrvr158 keyfactor-bash-orchestrator.sh[932]: keyfactor-bash-orchestrator-svc@appsrvr80.keyexample.com: Permission denied (publickey).
Aug 11 13:03:30 appsrvr158 keyfactor-bash[29486]: Error occurred during job with id 'eeabd541-b9d2-46d2-a215-9cb99fed4adc' on server 'appsrvr80.keyexample.com': Unable to connect to 'appsrvr80.keyexample.com' on port '22' via SSH

For information on troubleshooting public key issues with remote control targets, see Validate Remote Control Target Public Key. For more information on troubleshooting remote control target issues in general, see Remote Control Target Logs. For information on what successful inventory and publish policy log messages look like, see Successful Inventory and Policy Publishing.

Unlike on the orchestrator itself, where you can enable debug logging to see a more detailed picture of what's going on when the orchestrator attempt to connect or run a job, on a remote control target, the only logs available are the SSH logs showing attempts by the orchestrator to make a remote connection into the target and then the commands the orchestrator runs from an SSH perspective. These logs are found in the Linux system log where SSH logs are consolidated. The name and location of this will vary by operating system, but it is often found in /var/log by default (auth.log or secure is common). A large number of entries are generated in the log on a successful connection for inventory or inventory and policy publishing, so it can be difficult to interpret the logs.

In these logs you can check to see if the orchestrator is successfully making an SSH connection. If it isn't, you may see some messages that will help determine why it isn't. If it's successfully making the initial connection but then failing further along in the process, this log may also help reveal that. Perhaps one of the commands that the service account needs to run isn't in the expected path, for example.

When the orchestrator first connects to the remote control target, the log entries on the target will look something like:

Aug 11 17:36:51 appsrvr80 sshd[95543]: Accepted publickey for keyfactor-bash-orchestrator-svc from 10.4.3.158 port 47778 ssh2: RSA SHA256:u5zNB4UEoPNcax5p4fBbkkWaoiWq6AcEkA65XdzUkM4
Aug 11 17:36:51 appsrvr80 sshd[95543]: pam_unix(sshd:session): session opened for user keyfactor-bash-orchestrator-svc by (uid=0)
Aug 11 17:36:51 appsrvr80 systemd-logind[656]: New session 13019 of user keyfactor-bash-orchestrator-svc.
Aug 11 17:36:51 appsrvr80 systemd: pam_unix(systemd-user:session): session opened for user keyfactor-bash-orchestrator-svc by (uid=0)
Aug 11 17:36:51 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/bin/cat /etc/ssh/sshd_config

An inventory of an authorized_keys file for a user will appear as a series of entries, something like:

Aug 11 18:11:28 appsrvr164 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/bin/test -f /home/jsmith/.ssh/authorized_keys
Aug 11 18:11:28 appsrvr164 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/bin/ls -l /home/jsmith/.ssh/authorized_keys
Aug 11 18:11:28 appsrvr164 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/bin/cat /home/jsmith/.ssh/authorized_keys

Removal of a rogue key A rogue key, in the context of Keyfactor Command, is an SSH public key that appears in an authorized_keys file on a server managed by the SSH orchestrator without authorization. on a remote control target under management (in inventory and publish policy mode) will appear as a series of entries where the authorized_keys file is removed, recreated and repopulated with any valid keys (none in this case), like:

Aug 12 09:01:24 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/usr/bin/test -f /home/jsmith/.ssh/authorized_keys
Aug 12 09:01:24 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/bin/rm /home/jsmith/.ssh/authorized_keys
Aug 12 09:01:25 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/usr/bin/test -f /home/jsmith/.ssh/authorized_keys
Aug 12 09:01:25 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/usr/bin/test -d /home/jsmith/.ssh
Aug 12 09:01:25 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/usr/bin/touch /home/jsmith/.ssh/authorized_keys
Aug 12 09:01:25 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/bin/chmod 640 /home/jsmith/.ssh/authorized_keys
Aug 12 09:01:25 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/bin/chown jsmith: /home/jsmith/.ssh/authorized_keys
Aug 12 09:01:25 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/usr/bin/flock /home/jsmith/.ssh/authorized_keys echo
Aug 12 09:01:25 appsrvr80 sudo: keyfactor-bash-orchestrator-svc : TTY=unknown ; PWD=/opt/keyfactor-bash-orchestrator-client ; USER=root ; COMMAND=/usr/bin/tee -a /home/jsmith/.ssh/authorized_keys

Here is an example of an error you might see when using a certificate store:

A "permission denied" error of this type is often the result of one of these issues:

• The service account used to authenticate the orchestrator into the target server does not have permissions or access to SFTP on that server (for an FTP certificate store).

• The service account does not have the proper permissions to read or write to the file and/or folder.

To troubleshoot permission denied issues:

1. Open a command prompt and type:

   sftp username@server IP or hostname

   or

   psftp username@server IP or hostname

   For example:

   psftp kyfuser@192.168.12.73

2. COnce connected to the target server, confirm that you can access the folder and files. For example, use the ls command to list the directory contents and check that the files are there and have correct permissions for your user.

   

   Figure 555: Check File Permissions for the User

3. If the service account user needs access to these files, you will need to change the permissions on them. This cannot be done over FTP/SFTP.

4. To correct the problem with the CRT file, you will need to add read permissions for the service account user, at a minimum. This can be done with the following chmod command:

   chmod 640 AppCert.crt

   This command grants the owner of the file (correct in this case) read and write permission on the file (permission level 6 = 4 (read) + 2 (write)), the group associated with the file read permissions (permission level 4), and all others no permissions (permission level 0). No users have execute permissions (permission level 1). A certificate file does not need to be executable.

5. To correct the problem with the PFX A PFX file (personal information exchange format), also known as a PKCS#12 archive, is a single, password-protected certificate archive that contains both the public and matching private key and, optionally, the certificate chain. It is a common format for Windows servers. file, you need to be acting as root, since only root presently has permissions to the file. You will probably want to change the file ownership to your service account user and then set the permissions appropriately as follows.

   Set the owner on the file with the chown command:

   sudo chown kyfuser AppCert.pfx

   Set the group on the file with the chgrp command (notice you don't need to use sudo here because you now have permissions on the file because you own it):

   chgrp kyfuser AppCert.pfx

   Set the permissions on the file with the chmod command:

   chmod 640 AppCert.pfx

Here is an example of some very similar errors you might see when trying to connect to a target machine to inventory a certificate store or execute a management or discovery job on a certificate store:

Messages of this type are generally the result of the target server being inaccessible. This might happen if the server was turned off or in maintenance mode. Perhaps there is a network problem routing to that server. If the certificate store has never worked in Keyfactor Command, perhaps there is a typo in the server name configuration.

You may encounter this error when doing an inventory of an IIS certificate store:

This is an indication that the certificate store you are inventorying contains more certificates (or more precisely, the certificates add up to a total number of bytes greater) than IIS on the Keyfactor Command server is configured to accept. To resolve this, adjust the values on the IIS server that control the upload limits. For example, the maxAllowedContentLength. See Monitoring Network Scan Jobs with View Scan Details in the Keyfactor Command Reference Guide) on fine tuning SSL monitoring for more information.

You may receive a 403.16 error while trying to authenticate an orchestrator to Keyfactor Command using certificate authentication. On the face of it, this error indicates that the chain for the certificate you're using to authenticate is not trusted by the Keyfactor Command server. First, check to be sure that your certificate is trusted by the Keyfactor Command server. But if your certificate is fully trusted and you're still getting this error, what then?

This error can indicate that the trusted root store on the Keyfactor Command server contains a certificate that is not a root certificate (for example, an intermediate certificate is accidentally in the root store). To check this, open the Local Computer certificates MMC on the Keyfactor Command server, drill down to Certificates under the Trusted Root Certificate Authorities and scan for any certificates where the Issued To does not match the Issued By. Remove any certificates you find like this.

If you receive an error similar to the following:

This may indicate that the Keyfactor Universal Orchestrator was installed without the Microsoft Visual C++ Redistributable x64 required to manage certificates from remote Microsoft CAs (see System Requirements).

If you receive an error similar to the following (some portions of message removed for clarity):

2023-02-15 11:54:27.6600   Keyfactor.Orchestrators.JobEngine.SessionJobExecutor [Error] - Error in SessionManager: Unable to register session.

The SSL connection could not be established, see inner exception.

The remote certificate is invalid because of errors in the certificate chain: RevocationStatusUnknown, OfflineRevocation

This may indicate that the Keyfactor Universal Orchestrator cannot access the CRL(s) for the SSL certificate used to secure the Keyfactor Command server (see System Requirements).

To check this:

1. Enable at least debug level logging (see Configure Logging for the Universal Orchestrator).
2. Either wait for the orchestrator to attempt to register again, or restart the orchestrator service (see Start the Universal Orchestrator Service) to force an immediate attempt to register.

3. Look in the logs for a log message similar to the following (referencing your Keyfactor Command server name):

   2023-02-15 12:08:14.6076 Keyfactor.Orchestrators.Core.Http.KeyfactorHttpClient [Debug] - Sending request to 'https://keyfactor.keyexample.com/KeyfactorAgents/Session/Register'

4. Visit the referenced URL (https://keyfactor.keyexample.com/KeyfactorAgents/Session/Register) in a browser on the orchestrator server. This should give you a response of:

   The requested resource does not support http method 'GET'.

5. In the browser, view details for the certificate (the exact method for this will vary depending on the browser) and check the CRL Distribution Points field in the certificate.

   

   Figure 557: Find the Certificate for the Keyfactor Command Web Site

6. In the same browser on the orchestrator server, attempt to browse to the URL for the CRL (assuming it's a URL).
7. If the CRL downloads without error, then likely CRL access is not the issue. Open the CRL and check the Next update date to see if it's in the past (indicating the CRL is out of date).