Manage agent pools with the Terraform Cloud Operator v2
Note
HCP Terraform Free Edition includes one self-hosted agent. Refer to HCP Terraform pricing for details.
The Terraform Cloud Operator for Kubernetes lets you create and manage HCP Terraform agents, agent pools, and tokens through a single Kubernetes custom resource.
HCP Terraform Agents let HCP Terraform communicate with isolated, private, or on-premises infrastructure. By deploying agents within a specific network segment, you can connect your environment and HCP Terraform, which lets Terraform provision and manage private infrastructure.
In this tutorial you will use the Terraform Cloud Operator for Kubernetes to create and manage an HCP Terraform Agent. The operator manages the lifecycle of agents, making it easier to scale them quickly and securely.
Prerequisites
This tutorial assumes that you are familiar with the standard Terraform workflow, HCP Terraform, and basic Kubernetes usage.
For this tutorial, you will need:
- An HCP Terraform organization owner account
- kubectl
- A running Kubernetes cluster v1.16+ with the Terraform Cloud Operator for Kubernetes installed
If you do not have the Terraform Cloud Operator for Kubernetes installed already, follow the Deploy Infrastructure with the Terraform Cloud Operator for Kubernetes tutorial through the Deploy the operator step and do not delete the Kind cluster or any of the resources created.
Clone repository
In your terminal, clone the Learn Terraform Cloud Operator for Kubernetes repository.
$ git clone https://github.com/hashicorp/learn-terraform-kubernetes-operator
Review configuration
Navigate into the v2
directory in the repository.
$ cd learn-terraform-kubernetes-operator/v2
Navigate into the operator
directory, which contains configuration files for the operator.
$ cd operator
Open the agentpool.yml
file. This file defines an HCP Terraform agent pool with one agent. Replace ORGANIZATION-NAME
with your own HCP Terraform organization and save the file.
operator/agentpool.yml
apiVersion: app.terraform.io/v1alpha2
kind: AgentPool
metadata:
name: agent-pool-demo
spec:
organization: ORGANIZATION_NAME
token:
secretKeyRef:
name: terraformrc
key: token
name: agent-pool-demo
agentTokens:
- name: agent-pool-demo-token
agentDeployment:
replicas: 1
spec:
containers:
- name: tfc-agent
image: "hashicorp/tfc-agent:1.13.1"
Configure HCP Terraform access
The operator authenticates to HCP Terraform using an HCP Terraform Team API token, which it accesses as a Kubernetes secret.
To create the token, navigate to your HCP Terraform organization's Settings page, then click Teams.
If you use a free HCP Terraform organization, HCP Terraform shows the team settings for the "owners". This is the only available team and it has full access to the HCP Terraform API.
If you use HCP Standard, Plus, or Terraform Enterprise, HCP Terraform shows the team management page. Click Create a team and name the team
k8sop-agents
. Under Settings permissions, enable the Manage agent pools permission. Click Update team organization access to save your settings.
Click on the API tokens option in the left navigation, and then choose the Team Tokens tab.
Click Create a team token. If you use a free HCP Terraform organization, choose the owners
team. If you use HCP Standard or Plus, choose your k8sop-agents
team. Choose an Expiration of 30 days and then click Create.
Click Copy token to copy the token string. You will use this token in the next section.
Tip
We recommend specifying an expiration for all team tokens.
Warning
The Team token has global privileges. Ensure that the Kubernetes cluster using this token has proper role-based access control to limit access to the secret, or store it in a secret manager with access control policies.
Copy this token and store it somewhere safe.
Create the agent pool
Create an environment variable to configure the Kubernetes namespace to deploy to named NAMESPACE
and set it to edu
.
$ export NAMESPACE=edu
Create a secret containing your HCP Terraform Team API token. Replace APITOKEN
with the token that you created earlier.
$ kubectl create secret generic -n $NAMESPACE tfc-owner --from-literal=token=APITOKEN
Note
HCP Terraform will not accept this API key once it expires. You can use the Vault Secrets Operator and Terraform Cloud secret engine to automatically rotate this token an update the Kubernetes secret.
Apply the AgentPool specification to the namespace.
$ kubectl apply -n $NAMESPACE -f agentpool.yml
Review the operator logs to confirm you launched the agent and agent pool.
$ kubectl logs deployment/terraform-operator-hcp-terraform-operator -n tfc-operator-system -f
2023-03-28T12:58:05Z INFO Reconcile Agent Pool {"agentpool": "edu/agent-pool-demo", "msg": "status.agentPoolID is empty, creating a new agent pool"}
2023-03-28T12:58:05Z INFO Reconcile Agent Pool {"agentpool": "edu/agent-pool-demo", "msg": "successfully created a new agent pool with ID apool-REDACTED"}
Verify the agent pool
Navigate to your HCP Terraform organization settings. Go to the Agents page and verify that you created a new agent pool named agent-pool-demo
with one agent.
Enable autoscaling
Note: HCP Terraform Free Edition includes one self-hosted agent. Refer to HCP Terraform pricing for details.
Provide the .spec.autoscaling
configuration in your AgentPool
specification so that the operator scales the number of agents based on the number of pending Terraform workloads.
Open the agentpool.yml
file and add the following configuration.
apiVersion: app.terraform.io/v1alpha2
kind: AgentPool
spec:
##...
autoscaling:
minReplicas: 0
maxReplicas: 1
targetWorkspaces:
- name: greetings
cooldownPeriod:
scaleUpSeconds: 30
scaleDownSeconds: 30
The minReplicas
and maxReplicas
fields define the number of agents that the operator will deploy. By default, the operator will wait 300 seconds between scaling events, but you can configure the wait time with the scaleUpSeconds
and scaleDownSeconds
fields.
The optional targetWorkspaces
configuration takes an array of workspaces to enable autoscaling on. If you omit this field, the operator will determine when to scale the agent pods based on the load of all workspaces configured to use the agent pool. You can identify workspaces with the following identifiers:
name
: The name of the workspace (Example:greetings
).id
: The ID of the workspace (Example:ws-NUVHA9feCXzAmPHx
).wildcardName
: A pattern to search for workspace names (Example:west-development-*
).
In the above example, the operator will only scale the pods for this agent pool based on the load of the greetings
workspace.
Apply the updated AgentPool
spec.
$ kubectl apply -n $NAMESPACE -f agentpool.yml
Open the workspace.yml
file and configure the workspace to use the agent pool by adding the .spec.agentPool
configuration with the agent pool name.
apiVersion: app.terraform.io/v1alpha2
kind: Workspace
spec:
##...
agentPool:
name: agent-pool-demo
Next, update the workspace executionMode
and set the vale to agent
.
apiVersion: app.terraform.io/v1alpha2
kind: Workspace
spec:
##...
executionMode: agent
Finally, apply the updated Workspace
spec.
$ kubectl apply -n $NAMESPACE -f workspace.yml
With no pending Terraform workloads, the operator automatically scales the number of agents to match the minReplicas
defined in your configuration. In this case, the operator stops all running agents. Watch the pods in the operator namespace and verify that the agent pods are stopped.
$ kubectl -n $NAMESPACE get pods -w
##...
edu agents-of-agent-pool-demo-78477fbb6f-2bsch 0/1 Terminating 0 9m25s
Clean up resources
First, destroy the agent pool and agent by removing the resource definition from your cluster.
$ kubectl delete -n $NAMESPACE agentpool agent-pool-demo
Next, delete the Kubernetes resources. Navigate to the root directory.
$ cd ..
Destroy the namespace, secrets, and the operator. Confirm the deletion by typing yes
when prompted.
$ terraform destroy
##...
Plan: 0 to add, 0 to change, 5 to destroy.
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
##...
Destroy complete! Resources: 5 destroyed.
Finally, if you are running on Kind, delete the cluster.
$ kind delete cluster --name terraform-learn
Deleting cluster "terraform-learn" …
Next steps
In this tutorial, you used the Terraform Cloud Operator for Kubernetes to create an HCP Terraform agent pool. You can modify and extend the example configuration to deploy multiple agent pools, and scale the number of agents in each agent pool.
Visit the following resources to learn more about the Terraform Cloud Operator for Kubernetes.
- Read the Deploy infrastructure with the Terraform Cloud Operator v2 tutorial.
- To learn more about the operator and its design, check out the hashicorp/hcp-terraform-operator repository.
- To discover more about managing Kubernetes with Terraform, review the HashiCorp Kubernetes tutorials.