Set up mutual TLS with user-provided certificates

This page provides instructions for creating a root certificate and a signed intermediate certificate, and then uploading those certificates to a Certificate Manager TrustConfig resource. If you have existing certificates to upload, skip the steps that create new certificates.

You also create the network security resources required for configuring mutual TLS (mTLS) for Application Load Balancers. The instructions use OpenSSL to create the root and intermediate certificates.

Before you begin

Review the Mutual TLS overview.
Review the Manage trust configs.
Install the Google Cloud CLI. For a complete overview of the tool, see the gcloud CLI overview. You can find commands related to load balancing in the API and gcloud CLI reference.

If you haven't run the gcloud CLI previously, first run the gcloud init command to authenticate.
Enable APIs: Compute Engine API, Certificate Manager API, Network Security, and Network Services API.
If you are using global external Application Load Balancer or classic Application Load Balancer, make sure you have set up a load balancer with any of the following supported backends:
- VM instance group backends
- Cloud Storage buckets (Supported only if there is at least one backend service also attached to the load balancer, in addition to the backend bucket)
- Cloud Run, App Engine, or Cloud Functions
- Hybrid connectivity
If you are using regional external Application Load Balancer, cross-region internal Application Load Balancer, or regional internal Application Load Balancer, make sure you have set up a load balancer with any of the following supported backends:
- VM instance group backends
- Cloud Run
- Hybrid connectivity
Set your project.
gcloud
```
gcloud config set project PROJECT_ID
```

Permissions

To get the permissions that you need to complete this guide, ask your administrator to grant you the following IAM roles on the project:

To create load balancer resources such as TargetHTTPSProxy: Compute Load Balancer Admin (roles/compute.loadBalancerAdmin)
To use Certificate Manager resources: Certificate Manager Owner (roles/certificatemanager.owner)
To create security and networking components: Compute Network Admin (roles/compute.networkAdmin) and Compute Security Admin (roles/compute.securityAdmin)
To create a project (optional): Project Creator (roles/resourcemanager.projectCreator)

For more information about granting roles, see Manage access.

You might also be able to get the required permissions through custom roles or other predefined roles.

Generate a key and signed certificates

This section uses openssl commands to create root and intermediate certificates.

Use the following commands to generate a root certificate and a signed intermediate certificate with valid keyUsage and extendedKeyUsage fields.

Create a sample example.cnf file with the minimum configuration required to create valid signing certificates. You can edit this file if you want to set any additional fields on these certificates.

cat > example.cnf << EOF
[req]
distinguished_name = empty_distinguished_name
[empty_distinguished_name]
# Kept empty to allow setting via -subj command line arg.
[ca_exts]
basicConstraints=critical,CA:TRUE
keyUsage=keyCertSign
extendedKeyUsage=clientAuth
EOF

Create the root certificate.

openssl req -x509 \
    -new -sha256 -newkey rsa:2048 -nodes \
    -days 3650 -subj '/CN=root' \
    -config example.cnf \
    -extensions ca_exts \
    -keyout root.key -out root.cert

Create the signing request for the intermediate certificate.

openssl req \
    -new -sha256 -newkey rsa:2048 -nodes \
    -subj '/CN=int' \
    -config example.cnf \
    -extensions ca_exts \
    -keyout int.key -out int.req

Create the intermediate certificate.

openssl x509 -req \
    -CAkey root.key -CA root.cert \
    -set_serial 1 \
    -days 3650 \
    -extfile example.cnf \
    -extensions ca_exts \
    -in int.req -out int.cert

Generate a certificate and add it to an allowlist

This section uses openssl commands to create a sample certificate and add it to an allowlist.

Use the following commands to generate a certificate and add it to an allowlist.

   openssl req -x509 \
       -new -sha256 -newkey rsa:2048 -nodes \
       -days 3650 -subj '/CN=localhost' \
       -keyout allowlisted.key -out allowlisted.cert

Format the certificates

To include new or existing certificates in a TrustStore, format the certificates into a single line and store them in environment variables, so that they can be read into the YAML file. Use the following commands to format the certificates and store them in environment variables:

export ROOT_CERT=$(cat root.cert | sed 's/^[ ]*//g' | tr '\n' $ | sed 's/\$/\\n/g')

export INTERMEDIATE_CERT=$(cat int.cert | sed 's/^[ ]*//g' | tr '\n' $ | sed 's/\$/\\n/g')

To include new or existing certificates that are added to an allowlist in a trust config, format the certificates into a single line and store them in environment variables, so that they can be read into the YAML file. For certificates that are on an allowlist, use the following command to format the certificates into a single line and store them in the ALLOWLISTED_CERT environment variable.

export ALLOWLISTED_CERT=$(cat allowlisted.cert | sed 's/^[ ]*//g' | tr '\n' $ | sed 's/\$/\\n/g')

Create a TrustConfig resource

Create a Certificate Manager TrustConfig resource that represents your PKI. This example TrustConfig resource contains a trust store with two trust anchors and two intermediate CA certificates. It reads the certificate content from the environment variables created in the previous Format the certificates step.

To create a trust store with additional trust anchors or intermediate CA certificates, add pemCertificate rows in the appropriate section. If you have fewer trust anchors or intermediate CA certificates, remove the unneeded lines.

This example TrustConfig resource contains a certificate that is added to an allowlist. You can specify multiple certificates in an allowlist by using multiple instances of the pemCertificate field.

In the following steps, replace TRUST_CONFIG_NAME with the name of the TrustConfig resource:

To create the trust_config.yaml file with a trust store, use the following command:

cat << EOF > trust_config.yaml
trustStores:
- trustAnchors:
  - pemCertificate: "${ROOT_CERT?}"
  - pemCertificate: "${ROOT_CERT_2?}"
  intermediateCas:
  - pemCertificate: "${INTERMEDIATE_CERT?}"
  - pemCertificate: "${INTERMEDIATE_CERT_2?}"
EOF

Optional: To create the trust_config.yaml file with a certificate that is added to an allowlist, use the following command:
```
cat << EOF > trust_config.yaml
 allowlistedCertificates:
 - pemCertificate: "${ALLOWLISTED_CERT?}"
EOF
```
To create the Certificate Manager TrustConfig resources, use the gcloud certificate-manager trust-configs import command:
global
For external Application Load Balancers and cross-region internal Application Load Balancers, use this command:
```
gcloud certificate-manager trust-configs import TRUST_CONFIG_NAME  \
   --source=trust_config.yaml
```
regional
For regional external Application Load Balancers and regional internal Application Load Balancers, use this command:
```
gcloud certificate-manager trust-configs import TRUST_CONFIG_NAME  \
   --source=trust_config.yaml \
   --location=REGION
```

Create the Client Authentication resources

A Client Authentication (also called ServerTLSPolicy) resource lets you specify the server-side TLS mode and the TrustConfig resource to use when validating client certificates. When the client presents an invalid certificate or no certificate to the load balancer, the clientValidationMode specifies how the client connection is handled. For more information, see MTLS client validation modes.

When the clientValidationMode is set to ALLOW_INVALID_OR_MISSING_CLIENT_CERT, all requests are passed to the backend even if the validation fails or the client certificate is missing.
When the clientValidationMode is set to REJECT_INVALID, only requests that supply a client certificate that can be validated against a TrustConfig resource are passed to the backend.

To create the ServerTLSPolicy resource, complete the following steps:

Based on how you want to handle the connection, select one of the following options.

In the following steps, replace SERVER_TLS_POLICY_NAME with the name of the server TLS policy, and replace PROJECT_ID with the ID of your Google Cloud project.
- Option 1: clientValidationMode is set to ALLOW_INVALID_OR_MISSING_CLIENT_CERT.
  
  To create the server_tls_policy.yaml file, use the following command:
  global
  For external Application Load Balancers and cross-region internal Application Load Balancers, use the command:
```
cat << EOF > server_tls_policy.yaml
name: SERVER_TLS_POLICY_NAME
mtlsPolicy:
  clientValidationMode: ALLOW_INVALID_OR_MISSING_CLIENT_CERT
  clientValidationTrustConfig: projects/PROJECT_ID/locations/global/trustConfigs/TRUST_CONFIG_NAME
EOF
```
  regional
  For regional external Application Load Balancers and regional internal Application Load Balancers, use the command:
```
cat << EOF > server_tls_policy.yaml
name: SERVER_TLS_POLICY_NAME
mtlsPolicy:
  clientValidationMode: ALLOW_INVALID_OR_MISSING_CLIENT_CERT
  clientValidationTrustConfig: projects/PROJECT_ID/locations/REGION/trustConfigs/TRUST_CONFIG_NAME
EOF
```
- Option 2: clientValidationMode is set to REJECT_INVALID.
  
  To create the server_tls_policy.yaml file, use the following command:
  global
  For external Application Load Balancers and cross-region internal Application Load Balancers, use the command:
```
cat << EOF > server_tls_policy.yaml
name: SERVER_TLS_POLICY_NAME
mtlsPolicy:
  clientValidationMode: REJECT_INVALID
  clientValidationTrustConfig: projects/PROJECT_ID/locations/global/trustConfigs/TRUST_CONFIG_NAME
EOF
```
  regional
  For regional external Application Load Balancers and regional internal Application Load Balancers, use the command:
```
cat << EOF > server_tls_policy.yaml
name: SERVER_TLS_POLICY_NAME
mtlsPolicy:
  clientValidationMode: REJECT_INVALID
  clientValidationTrustConfig: projects/PROJECT_ID/locations/REGION/trustConfigs/TRUST_CONFIG_NAME
EOF
```
To create the ServerTlsPolicy resource, use the gcloud network-security server-tls-policies import command:
global
For external Application Load Balancers and cross-region internal Application Load Balancers, use the command:
```
gcloud network-security server-tls-policies import SERVER_TLS_POLICY_NAME \
  --source=server_tls_policy.yaml \
  --location=global
```
regional
For regional external Application Load Balancers and regional internal Application Load Balancers, use the command:
```
gcloud network-security server-tls-policies import SERVER_TLS_POLICY_NAME \
  --source=server_tls_policy.yaml \
  --location=REGION
```
Optional: List all the Client Authentication (ServerTlsPolicies) resources in the specified location of the current project.
Console
1. In the Google Cloud console, go to the Client authentication page.
  
  Go to Client authentication
2. All the ServerTlsPolicies resources are displayed.
gcloud
To list all the Client Authentication (ServerTlsPolicies) resources, use the gcloud network-security server-tls-policies list command:
```
gcloud network-security server-tls-policies list \
  --location=REGION
```
Replace the following:

REGION: use global for cross-region internal Application Load Balancer, global external Application Load Balancer, or classic Application Load Balancer. For regional external Application Load Balancer or regional internal Application Load Balancer, use the region where you configured the load balancer.

Set up mTLS for the load balancer

For mutual TLS authentication to work, after you set up a load balancer, you need to update the target HTTPS proxy by using the ServerTLSPolicy resource.

Ensure that you have already created the Client Authentication (ServerTLSPolicy) resource. For instructions, see Create the Client Authentication resources.
To list all the target HTTPS proxies in your project, use the gcloud compute target-https-proxies list command:
```
gcloud compute target-https-proxies list
```
Note the name of the target HTTPS proxy to attach the ServerTLSPolicy resource. This name is referred to as TARGET_HTTPS_PROXY_NAME in the following steps.
To export a target HTTPS proxy's configuration to a file, use the gcloud compute target-https-proxies export command.
global
```
  gcloud compute target-https-proxies export TARGET_HTTPS_PROXY_NAME \
      --destination=TARGET_PROXY_FILENAME \
      --global
  
```
Replace the following:
- TARGET_HTTPS_PROXY_NAME: the name of the target proxy.
- TARGET_PROXY_FILENAME: the name of a yaml file. For example, mtls_target_proxy.yaml.
regional
```
 gcloud compute target-https-proxies export TARGET_HTTPS_PROXY_NAME \
     --destination=TARGET_PROXY_FILENAME \
     --region=REGION
 
```
Replace the following:
- TARGET_HTTPS_PROXY_NAME: the name of the target proxy.
- TARGET_PROXY_FILENAME: the name of a yaml file. For example, mtls_target_proxy.yaml
- REGION: the region where you configured the load balancer.
List all the ServerTlsPolicies resources in the specified location of the current project.
Console
1. In the Google Cloud console, go to the Client authentication page.
  
  Go to Client authentication
2. All the ServerTlsPolicies resources are displayed.
gcloud
To list all the Client authentication (ServerTlsPolicies) resources, use the gcloud network-security server-tls-policies list command:
```
gcloud network-security server-tls-policies list \
  --location=REGION
```
Replace the following:

REGION: use global for cross-region internal Application Load Balancer, global external Application Load Balancer, or classic Application Load Balancer. For regional external Application Load Balancer or regional internal Application Load Balancer, use the region where you configured the load balancer.

Note the name of the ServerTlsPolicies resource to configure mTLS. This name is referred to as SERVER_TLS_POLICY_NAME in the next step.

To append the ServerTlsPolicy resource file TARGET_PROXY_FILENAME, use the following command. Replace PROJECT_ID with the ID of your Google Cloud project.

echo "serverTlsPolicy: //networksecurity.googleapis.com/projects/PROJECT_ID/locations/REGION/serverTlsPolicies/SERVER_TLS_POLICY_NAME" >> TARGET_PROXY_FILENAME

To import a target HTTPS proxy's configuration from a file, use the gcloud compute target-https-proxies import command.
global
```
   gcloud compute target-https-proxies import TARGET_HTTPS_PROXY_NAME \
       --source=TARGET_PROXY_FILENAME \
       --global
   
```
Replace the following:
- TARGET_HTTPS_PROXY_NAME: the name of the target proxy.
- TARGET_PROXY_FILENAME: the name of a yaml file. For example, mtls_target_proxy.yaml.
regional
```
   gcloud compute target-https-proxies import TARGET_HTTPS_PROXY_NAME \
       --source=TARGET_PROXY_FILENAME \
       --region=REGION
   
```
Replace the following:
- TARGET_HTTPS_PROXY_NAME: the name of the target proxy.
- TARGET_PROXY_FILENAME: the name of a yaml file. For example, mtls_target_proxy.yaml
- REGION: the region where you configured the load balancer.

Add mTLS custom headers

When you enable mTLS, you can pass information about the mTLS connection using custom headers. You can also enable logging so that mTLS connection failures are captured in the logs.

Add mTLS custom headers to backend services

For global external Application Load Balancers or classic Application Load Balancers, you can use custom headers to pass information about the mTLS connection to backend services.

To list all the backend services in the project, use the gcloud compute backend-services list command:
```
gcloud compute backend-services list
```
Note the name of the backend service to enable custom headers and logging. This name is referred to as BACKEND_SERVICE in the following step.

To update the backend service, use the gcloud compute backend-services update command:

gcloud compute backend-services update BACKEND_SERVICE \
  --global \
  --enable-logging \
  --logging-sample-rate=1 \
  --custom-request-header='X-Client-Cert-Present:{client_cert_present}' \
  --custom-request-header='X-Client-Cert-Chain-Verified:{client_cert_chain_verified}' \
  --custom-request-header='X-Client-Cert-Error:{client_cert_error}' \
  --custom-request-header='X-Client-Cert-Hash:{client_cert_sha256_fingerprint}' \
  --custom-request-header='X-Client-Cert-Serial-Number:{client_cert_serial_number}' \
  --custom-request-header='X-Client-Cert-SPIFFE:{client_cert_spiffe_id}' \
  --custom-request-header='X-Client-Cert-URI-SANs:{client_cert_uri_sans}' \
  --custom-request-header='X-Client-Cert-DNSName-SANs:{client_cert_dnsname_sans}' \
  --custom-request-header='X-Client-Cert-Valid-Not-Before:{client_cert_valid_not_before}' \
  --custom-request-header='X-Client-Cert-Valid-Not-After:{client_cert_valid_not_after}'

Add mTLS custom headers to URL map

For cross-region internal Application Load Balancer, regional external Application Load Balancer, or regional internal Application Load Balancer, you can use custom headers to pass information about the mTLS connection to the URL map.

To list all the URL maps in the project, use the gcloud compute url-maps list command:

   gcloud compute url-maps list

Note the name of the URL map to enable custom headers and logging. This name is referred to as URL_MAP_NAME in the following step.

global

To edit the URL map for a cross-region internal Application Load Balancer, use the gcloud compute url-maps edit command:

   gcloud compute url-maps edit URL_MAP_NAME --global

Following is a sample YAML file that shows you how to use variables in custom request headers (requestHeadersToAdd). You can use the same variables to send custom response headers (responseHeadersToAdd).

   headerAction:
      requestHeadersToAdd:
      - headerName: "X-Client-Cert-Present"
        headerValue: "{client_cert_present}"
      - headerName: "X-Client-Cert-Chain-Verified"
        headerValue: "{client_cert_chain_verified}"
      - headerName: "X-Client-Cert-Error"
        headerValue: "{client_cert_error}"
      - headerName: "X-Client-Cert-Hash"
        headerValue: "{client_cert_sha256_fingerprint}"
      - headerName: "X-Client-Cert-Serial-Number"
        headerValue: "{client_cert_serial_number}"
      - headerName: "X-Client-Cert-SPIFFE"
        headerValue: "{client_cert_spiffe_id}"
      - headerName: "X-Client-Cert-URI-SANs"
        headerValue: "{client_cert_uri_sans}"
      - headerName: "X-Client-Cert-DNSName-SANs"
        headerValue: "{client_cert_dnsname_sans}"
      - headerName: "X-Client-Cert-Valid-Not-Before"
        headerValue: "{client_cert_valid_not_before}"
      - headerName: "X-Client-Cert-Valid-Not-After"
        headerValue: "{client_cert_valid_not_after}"
      - headerName: "X-Client-Cert-Issuer-Dn"
        headerValue: "{client_cert_issuer_dn}"
      - headerName: "X-Client-Cert-Subject-Dn"
        headerValue: "{client_cert_subject_dn}"
      - headerName: "X-Client-Cert-Leaf"
        headerValue: "{client_cert_leaf}"
      - headerName: "X-Client-Cert-Chain"
        headerValue: "{client_cert_chain}"

regional

To edit the URL map for a regional external Application Load Balancer or a regional internal Application Load Balancer, use the gcloud compute url-maps edit command:

   gcloud compute url-maps edit URL_MAP_NAME --region=REGION

   defaultService: regions/REGION/backendServices/BACKEND_SERVICE_1
      name: regional-lb-map
      region: region/REGION
   headerAction:
      requestHeadersToAdd:
      - headerName: "X-Client-Cert-Present"
        headerValue: "{client_cert_present}"
      - headerName: "X-Client-Cert-Chain-Verified"
        headerValue: "{client_cert_chain_verified}"
      - headerName: "X-Client-Cert-Error"
        headerValue: "{client_cert_error}"
      - headerName: "X-Client-Cert-Hash"
        headerValue: "{client_cert_sha256_fingerprint}"
      - headerName: "X-Client-Cert-Serial-Number"
        headerValue: "{client_cert_serial_number}"
      - headerName: "X-Client-Cert-SPIFFE"
        headerValue: "{client_cert_spiffe_id}"
      - headerName: "X-Client-Cert-URI-SANs"
        headerValue: "{client_cert_uri_sans}"
      - headerName: "X-Client-Cert-DNSName-SANs"
        headerValue: "{client_cert_dnsname_sans}"
      - headerName: "X-Client-Cert-Valid-Not-Before"
        headerValue: "{client_cert_valid_not_before}"
      - headerName: "X-Client-Cert-Valid-Not-After"
        headerValue: "{client_cert_valid_not_after}"
      - headerName: "X-Client-Cert-Issuer-Dn"
        headerValue: "{client_cert_issuer_dn}"
      - headerName: "X-Client-Cert-Subject-Dn"
        headerValue: "{client_cert_subject_dn}"
      - headerName: "X-Client-Cert-Leaf"
        headerValue: "{client_cert_leaf}"
      - headerName: "X-Client-Cert-Chain"
        headerValue: "{client_cert_chain}"

Sign a client key with the intermediate certificate

This section provides an additional configuration option to generate a leaf certificate. If you have already created a TrustConfig resource by using intermediate certificates (int.cert and int.key), use the following instructions:

Create a client key configuration file.

cat > client.config << EOF
[req]
default_bits              = 2048
req_extensions            = extension_requirements
distinguished_name        = dn_requirements
prompt                    = no

[extension_requirements]
basicConstraints          = critical, CA:FALSE
keyUsage                  = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage          = clientAuth

[dn_requirements]
countryName               = US
stateOrProvinceName       = California
localityName              = San Francisco
0.organizationName        = example
organizationalUnitName    = test
commonName                = test.example.com
emailAddress              = [email protected]

EOF

If you want to have a SPIFFE identity attached:

Add a subjectAltName to your [extension_requirements] section as follows:
```
subjectAltName            = @sans_list
```

Add a new section at the bottom of the client.config file with the following:

[sans_list]
URI.1                     = spiffe://example.com/test-identity

Sign the key.

openssl req -new -keyout client.key -out client.csr -config client.config

openssl x509 -req -in client.csr -out client.cert -extfile client.config -extensions extension_requirements -days 365 -CA int.cert -CAkey int.key

To test send a curl request to the load balancer's IP address.
```
curl -v -k --key client.key --cert client.cert https://IP_ADDRESS
```
Replace IP_ADDRESS with the load balancer's IP address.

What's next

Set up mutual TLS with a private CA

Set up mutual TLS with user-provided certificates

Before you begin

gcloud

Permissions

Generate a key and signed certificates

Generate a certificate and add it to an allowlist

Format the certificates

Create a TrustConfig resource

global

regional

Create the Client Authentication resources

global

regional

global

regional

global

regional

Console

gcloud

Set up mTLS for the load balancer

global

regional

Console

gcloud

global

regional

Add mTLS custom headers

Add mTLS custom headers to backend services

Add mTLS custom headers to URL map

global

regional

Sign a client key with the intermediate certificate

What's next