Certificate validation

Truststores and Certificate Bundles

In the scope of this document, a Truststore is a set of CA certificates that are used at run time to verify the authenticity of TLS server certificates, and a Certificate Bundle is a file containing one or more trusted CA certificates in PEM format.

Truststores are used by RADKit to verify TLS connections to RADKit Cloud, Cisco-operated services, network devices and other endpoints. The specific truststore used for a given TLS connection is determined by the context in which the connection is established. RADKit never uses the system-provided certificates for TLS certificate validation.

RADKit includes built-in certificate bundles curated by Cisco’s Security & Trust Organization (for more details, see Cisco PKI: Trusted Root Stores).

There are four built-in certificate bundles in RADKit:

Domain bundle: for RADKit Cloud and Cisco IQ TLS endpoints.
Core bundle: for communication with Cisco-operated services such as CXD or Cisco SSO.
Intersect bundle: a curated set of root certificates that can be used for other TLS connections, such as HTTP or Swagger connections to devices.
Union bundle: a less restrictive set of root certificates that includes every public CA that is trusted by any of the major Root Store programs.

RADKit currently has three truststores:

Domain truststore: used exclusively for TLS connections to RADKit Cloud; uses the Domain bundle.
Cisco truststore: used for TLS connections to Cisco-operated services; uses the Core bundle by default.
Devices truststore: used for TLS connections to devices and other endpoints (RADKit Service only); uses the Intersect bundle by default.

Each truststore points to one of the built-in bundles (called a “root bundle” in RADKit settings) and an extension bundle. The extension bundle is a user-provided bundle that can be added to the built-in bundle to extend the set of trusted root certificates for a given truststore. The extension bundle is optional and can be empty.

Extension bundles are located under the directory certificate_bundles in the RADKit data directory (e.g. /opt/radkit/certificate_bundles/ under Linux). Paths in the settings are relative to that directory.

In short, adding one or more root certificates to RADKit consists of these steps:

Create a new bundle file in PEM format containing the root certificate(s) you want to add.
Configure the corresponding truststore extension to point to that bundle in the RADKit settings.
Optionally deactivate the root bundle for that truststore to utilize the extension bundle only.

Each truststore has its own settings in the RADKit settings file. See truststore.radkit_root_bundle for details.

Example use-cases

Private enterprise CA

When RADKit Service connects to managed network devices, it uses the Devices truststore to validate the TLS server certificates presented by those devices.

The Devices truststore has two settings:

devices_root_bundle which refers to a built-in bundle (by default the Intersect bundle)
devices_extension_bundle which refers to an optional user-provided bundle (by default empty)

If network devices are using a private enterprise CA, this CA will not be part of the Intersect bundle.

In that case, the administrator can create a new bundle containing the private enterprise CA root certificate in the RADKit directory (for example /opt/radkit/certificate_bundles/ under Linux).

Let’s assume the new bundle is named enterprise_ca.pem; its full path is /opt/radkit/certificate_bundles/enterprise_ca.pem.

The administrator can then edit the RADKit settings file and set devices_extension_bundle to enterprise_ca.pem.

If the administrator wants to use only the private enterprise CA root certificate and not the built-in Intersect bundle, they can also set devices_root_bundle to None.

For example:

[truststore]
devices_root_bundle = "None"
devices_extension_bundle = "enterprise_ca.pem"

TLS transparent proxy

When RADKit Service is protected by a TLS transparent proxy, the TLS server certificate presented by the proxy will not be part of the Domain bundle.

In this case, the Domain truststore must be set up as it governs RADKit cloud connectivity. The domain truststore is made of two parts:

radkit_root_bundle which refers to an embedded bundle (by default the Domain bundle)
radkit_extension_bundle which refers to an optional user-provided bundle (by default empty)

Similarly to the enterprise CA example, the administrator can create a new bundle containing the TLS transparent proxy root certificate in the RADKit directory (for example /opt/radkit/certificate_bundles/ under Linux).

Let’s assume the new bundle is named proxy_ca.pem; its full path is /opt/radkit/certificate_bundles/proxy_ca.pem.

The administrator can then edit the RADKit settings file and set radkit_extension_bundle to proxy_ca.pem.

Optionally, if the administrator wants to use only the TLS transparent proxy root certificate and not the Domain root bundle, they can also set the radkit_root_bundle to None.

For example:

[truststore]
radkit_root_bundle = "None"
radkit_extension_bundle = "proxy_ca.pem"

Updating the Cisco bundles

Cisco periodically publishes updates to its curated root bundles. Each new version of RADKit will include the latest version of those bundles. For this reason and other security considerations, it is strongly recommended to upgrade RADKit regularly.

If nevertheless you wish to install a fresh bundle without waiting for a new RADKit version, you can do so by following the instructions below.

Note

To perform these operations, you will need a recent version of the OpenSSL CLI. The examples below are for macOS or Linux.

Download the updated certificate bundle(s) from https://www.cisco.com/security/pki/trs/readme.html:

Core Bundle: https://www.cisco.com/security/pki/trs/current/ios_core/ios_core.p7b

Intersect Bundle: https://www.cisco.com/security/pki/trs/current/ios/ios.p7b

Union Bundle: https://www.cisco.com/security/pki/trs/current/ios_union/ios_union.p7b

and also download the Cisco TRS Bundle Root CA certificate from https://www.cisco.com/security/pki/:

Cisco TRS Bundle Root CA: http://www.cisco.com/security/pki/certs/tbrca.pem

Convert each bundle to a set of PEM certificates:
```
$ openssl cms -purpose any -verify -inform der -in FILE.p7b -CAfile tbrca.pem \
| openssl pkcs7 -inform der -print_certs -quiet -out FILE.pem
```
where FILE.p7b should be replaced with the actual file name for the bundle that your are converting (e.g. ios_core.p7b), and FILE.pem should be replaced with the desired output file name with the .pem extension (e.g. ios_core.pem).
Move the resulting .pem file(s) to the certificate_bundles directory (see above):
```
$ mv ios_core.pem /opt/radkit/certificate_bundles/
```
Specify the path to the new bundle in the TOML settings, using it as an extension to the appropriate truststore(s). Note that if you have an existing extension, it should be merged with the PEM file you generated in step 2.
```
[truststore]
devices_root_bundle = "None"
devices_extension_bundle = "ios_core.pem"
```

Important notes:

Make sure to set the root bundle to None to stop using the obsolete built-in bundle.
Make sure to keep your bundles updated, or restore the default settings after you upgrade RADKit to benefit from the latest built-in bundles.
If you edit the TOML settings file directly, you will need to:
- stop the RADKit Service (this step is critical),
- edit the settings file, and
- restart RADKit Service for the changes to take effect.

Certificate revocation

Certificate revocation checks are carried out after each TLS handshake performed by RADKit. These checks are dictated by the settings within the active SSL/TLS context. Each TLS connection operates within its own SSL/TLS context, as described in Truststores and Certificate Bundles. The revocation status of the certificate chain presented by the peer during the TLS handshake is verified up to the first recognized and trusted Certificate Authority (CA) from the truststore tied to that SSL/TLS context.

Within the RADKit settings, users can adjust:

the revocation method(s) used to check the status of each certificate;
the fail-open/fail-close policy;
the settings of specific revocation methods.

RADKit enables the selection and ordering of methods for checking revocation status, as outlined in cloud_client.ocsp.methods. If issues like network interruptions occur during the verification of the certificate chain, RADKit will switch to the next available method. If a certificate is confirmed as revoked, the TLS connection is immediately cut off, with no switch to another method. If an error unrelated to certificate revocation is encountered and no other methods are left to try, the TLS connection will be terminated. Including the PASS method at the end allows non-revocation related errors to be overlooked as part of a fail-open policy.

Available methods:

STAPLER: Leverages the RADKit Proxy Stapler (a service in RADKit Cloud) to conduct OCSP revocation checks on the user’s behalf. Its main benefit is that it connects to the same host as the Forwarder and Access services, instead of a multitude of remote OCSP endpoints.
DIRECT: Directly carries out OCSP revocation checks by connecting to the OCSP servers listed in the certificates received during the TLS handshake. Uses the HTTP protocol as per the standard.
DIRECT_HTTPS: Functions the same as DIRECT, but forces the use of HTTPS for OCSP queries instead of HTTP.
PASS: Ignores all errors except those related to the revocation of a certificate. This should be added last if a fail-open policy is desired.