Manage multiple environments with Terraform workspaces

For any organization scaling its infrastructure, managing multiple environments like development, staging, and production is a fundamental challenge. A common anti-pattern for beginners is duplicating the entire Terraform configuration for each environment. This leads to code duplication, configuration drift, and a high-maintenance nightmare. Fortunately, HashiCorp provides a built-in solution to this problem: Terraform workspaces. This feature allows you to use a single set of configuration files to manage multiple, distinct sets of infrastructure resources, primarily by isolating their state files.

This comprehensive guide will dive deep into what Terraform workspaces are, how to use them effectively, and their best practices. We’ll explore practical examples, variable management strategies, and how they fit into a modern CI/CD pipeline, empowering you to streamline your environment management process.

What Are Terraform Workspaces?

At their core, Terraform workspaces are a mechanism to manage multiple, independent state files for a single Terraform configuration. When you run terraform apply, Terraform writes metadata about the resources it created into a file named terraform.tfstate. A workspace provides a separate, named “space” for that state file.

This means you can have a single directory of .tf files (your main.tf, variables.tf, etc.) and use it to deploy a “dev” environment, a “staging” environment, and a “prod” environment. Each of these will have its own state file, completely isolated from the others. Running terraform destroy in the dev workspace will not affect the resources in your prod workspace, even though they are defined by the same code.

The ‘default’ Workspace

If you’ve never explicitly used workspaces, you’ve still been using one: the default workspace. Every Terraform configuration starts with this single workspace. When you run terraform init and terraform apply, the state is written to terraform.tfstate directly in your root directory (or in your configured remote backend).

How Workspaces Manage State

When you create a new workspace (e.g., dev), Terraform no longer writes to the root terraform.tfstate file. Instead, it creates a new directory called terraform.tfstate.d. Inside this directory, it will create a folder for each workspace, and each folder will contain its own terraform.tfstate file.

For example, if you have dev and prod workspaces, your directory structure might look like this (for local state):

.
├── main.tf
├── variables.tf
├── terraform.tfvars
├── terraform.tfstate.d/
│   ├── dev/
│   │   └── terraform.tfstate
│   └── prod/
│       └── terraform.tfstate

If you are using a remote backend like an S3 bucket, Terraform will instead create paths or keys based on the workspace name to keep the state files separate within the bucket.

Why Use Terraform Workspaces? (And When Not To)

Workspaces are a powerful tool, but they aren’t the solution for every problem. Understanding their pros and cons is key to using them effectively.

Key Advantages

• Code Reusability (DRY): The most significant benefit. You maintain one codebase for all your environments. A change to a security group rule is made once in main.tf and then applied to each environment as needed.
• Environment Isolation: Separate state files prevent “cross-talk.” You can’t accidentally destroy a production database while testing a change in staging.
• Simplicity for Similar Environments: Workspaces are ideal for environments that are structurally identical (or very similar) but differ only in variables (e.g., instance size, count, or name prefixes).
• Rapid Provisioning: Quickly spin up a new, temporary environment for a feature branch (e.g., feat-new-api) and tear it down just as quickly, all from the same configuration.

Common Pitfalls and When to Avoid Them

• Overuse of Conditional Logic: If you find your .tf files littered with complex if statements or count tricks based on terraform.workspace, you may be forcing a single configuration to do too much. This can make the code unreadable and brittle.
• Not for Different Configurations: Workspaces are for deploying the *same* configuration to *different* environments. If your prod environment has a completely different architecture than dev (e.g., uses RDS while dev uses a containerized SQLite), they should probably be separate Terraform configurations (i.e., in different directories).
• Large-Scale Complexity: For very large, complex organizations, managing many environments with subtle differences can still become difficult. At this scale, you might consider graduating to a tool like Terragrunt or adopting a more sophisticated module-based architecture where each environment is a separate root module that calls shared, versioned modules.

Getting Started: A Practical Guide to Terraform Workspaces

Let’s walk through a practical example. We’ll define a simple AWS EC2 instance and deploy it to both dev and prod environments, giving each a different instance type and tag.

Step 1: Basic CLI Commands

First, let’s get familiar with the core terraform workspace commands. Initialize a new Terraform directory to get started.

# Show the current workspace
$ terraform workspace show
default

# Create a new workspace
$ terraform workspace new dev
Created and switched to workspace "dev"

# Create another one
$ terraform workspace new prod
Created and switched to workspace "prod"

# List all available workspaces
$ terraform workspace list
  default
* prod
  dev

# Switch back to the dev workspace
$ terraform workspace select dev
Switched to workspace "dev"

# You cannot delete the 'default' workspace
$ terraform workspace delete default
Error: Failed to delete workspace: "default" workspace cannot be deleted

# You also cannot delete the workspace you are currently in
$ terraform workspace delete dev
Error: Failed to delete workspace: cannot delete current workspace "dev"

# To delete a workspace, switch to another one first
$ terraform workspace select prod
Switched to workspace "prod"

$ terraform workspace delete dev
Deleted workspace "dev"!

Step 2: Structuring Your Configuration with terraform.workspace

Terraform exposes the name of the currently selected workspace via the terraform.workspace interpolation. This is incredibly useful for naming and tagging resources to avoid collisions.

Let’s create a main.tf file.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# We will define variables later
variable "instance_type" {}
variable "ami_id" {
  description = "The AMI to use for our instance"
  type        = string
  default     = "ami-0c55b159cbfafe1f0" # An Amazon Linux 2 AMI
}

resource "aws_instance" "web_server" {
  ami           = var.ami_id
  instance_type = var.instance_type # This will come from a variable

  tags = {
    # Use the workspace name to differentiate resources
    Name        = "web-server-${terraform.workspace}"
    Environment = terraform.workspace
  }
}

output "instance_id" {
  value = aws_instance.web_server.id
}

output "instance_public_ip" {
  value = aws_instance.web_server.public_ip
}

Notice the tags block. If we are in the dev workspace, the instance will be named web-server-dev. In the prod workspace, it will be web-server-prod. This simple interpolation is the key to managing resources within a single AWS account.

Step 3: Managing Environment-Specific Variables

This is the most critical part. Our dev environment should use a t3.micro, while our prod environment needs a t3.medium. How do we manage this?

There are two primary methods: using map variables or using separate .tfvars files.

Method 1: Using a Map Variable (Recommended)

This is a clean, self-contained approach. We define a map in our variables.tf file that holds the configuration for each environment. Then, we use the terraform.workspace as a key to look up the correct value.

First, update variables.tf (or add it):

# variables.tf

variable "ami_id" {
  description = "The AMI to use for our instance"
  type        = string
  default     = "ami-0c55b159cbfafe1f0" # Amazon Linux 2
}

# Define a map variable to hold settings per environment
variable "env_config" {
  description = "Configuration settings for each environment"
  type = map(object({
    instance_type = string
    instance_count = number
  }))
  default = {
    "default" = {
      instance_type = "t3.nano"
      instance_count = 0 # Don't deploy in default
    },
    "dev" = {
      instance_type = "t3.micro"
      instance_count = 1
    },
    "prod" = {
      instance_type = "t3.medium"
      instance_count = 2 # Example of changing count
    }
  }
}

Now, update main.tf to use this map. We’ll also add the count parameter.

# main.tf (updated)

# ... provider block ...

resource "aws_instance" "web_server" {
  # Use the workspace name to look up the correct config
  count         = var.env_config[terraform.workspace].instance_count
  instance_type = var.env_config[terraform.workspace].instance_type

  ami = var.ami_id

  tags = {
    Name        = "web-server-${terraform.workspace}-${count.index}"
    Environment = terraform.workspace
  }
}

output "instance_public_ips" {
  value = aws_instance.web_server.*.public_ip
}

Now, let’s deploy:

# Make sure we have our workspaces
$ terraform workspace new dev
$ terraform workspace new prod

# Initialize the configuration
$ terraform init

# Select the dev workspace and apply
$ terraform workspace select dev
Switched to workspace "dev"
$ terraform apply -auto-approve

# ... Terraform will plan to create 1 t3.micro instance ...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Outputs:
instance_public_ips = [
  "54.1.2.3",
]

# Now, select the prod workspace and apply
$ terraform workspace select prod
Switched to workspace "prod"
$ terraform apply -auto-approve

# ... Terraform will plan to create 2 t3.medium instances ...
# This plan is totally independent of the 'dev' state.
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
Outputs:
instance_public_ips = [
  "34.4.5.6",
  "34.7.8.9",
]

You now have two separate environments deployed from the same code, each with its own state file and configuration.

Method 2: Using -var-file

An alternative (and simpler for some) approach is to create separate variable files for each environment.

Create dev.tfvars:

# dev.tfvars
instance_type = "t3.micro"
instance_count = 1

Create prod.tfvars:

# prod.tfvars
instance_type = "t3.medium"
instance_count = 2

Your main.tf would just use these variables directly:

# main.tf (for -var-file method)

variable "instance_type" {}
variable "instance_count" {}
variable "ami_id" { default = "ami-0c55b159cbfafe1f0" }

resource "aws_instance" "web_server" {
  count         = var.instance_count
  instance_type = var.instance_type
  ami           = var.ami_id
  
  tags = {
    Name        = "web-server-${terraform.workspace}-${count.index}"
    Environment = terraform.workspace
  }
}

When you run apply, you must specify which var file to use:

# Select the workspace first!
$ terraform workspace select dev
Switched to workspace "dev"

# Then apply, passing the correct var file
$ terraform apply -var-file="dev.tfvars" -auto-approve

# --- Repeat for prod ---

$ terraform workspace select prod
Switched to workspace "prod"

$ terraform apply -var-file="prod.tfvars" -auto-approve

This method is clear, but it requires you to remember to pass the correct -var-file flag every time, which can be error-prone. The map method (Method 1) is often preferred as it’s self-contained and works automatically with just terraform apply.

Terraform Workspaces in a CI/CD Pipeline

Automating Terraform workspaces is straightforward. Your pipeline needs to do two things:

1. Select the correct workspace based on the branch or trigger.
2. Run plan and apply.

Here is a simplified example for GitHub Actions that deploys dev on a push to the main branch and requires a manual approval to deploy prod.

# .github/workflows/terraform.yml

name: Terraform Deploy

on:
  push:
    branches:
      - main
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment to deploy'
        required: true
        type: choice
        options:
          - prod

jobs:
  deploy-dev:
    name: 'Deploy to Development'
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      TF_WORKSPACE: 'dev'

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2

    - name: Terraform Init
      run: terraform init

    - name: Terraform Workspace
      run: terraform workspace select $TF_WORKSPACE || terraform workspace new $TF_WORKSPACE

    - name: Terraform Plan
      run: terraform plan -out=tfplan

    - name: Terraform Apply
      run: terraform apply -auto-approve tfplan

  deploy-prod:
    name: 'Deploy to Production'
    if: github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'prod'
    runs-on: ubuntu-latest
    environment: production # GitHub environment for secrets and approvals
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_PROD_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_PROD_SECRET_ACCESS_KEY }}
      TF_WORKSPACE: 'prod'

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2

    - name: Terraform Init
      run: terraform init

    - name: Terraform Workspace
      run: terraform workspace select $TF_WORKSPACE || terraform workspace new $TF_WORKSPACE

    - name: Terraform Plan
      run: terraform plan -out=tfplan

    - name: Terraform Apply
      run: terraform apply -auto-approve tfplan

Terraform Workspaces vs. Git Branches: Clearing the Confusion

A very common point of confusion is how workspaces relate to Git branches. They solve different problems.

• Git Branches are for code isolation. You use a branch (e.g., feature-x) to develop a new feature without affecting the stable main branch.
• Terraform Workspaces are for state isolation. You use a workspace (e.g., prod) to deploy the *same* stable main branch code to a different environment, keeping its state separate from dev.

Anti-Pattern: Using Git branches to manage environments (e.g., a dev branch and a prod branch that have different .tf files). This leads to massive configuration drift and makes merging changes from dev to prod a nightmare.

Best Practice: Use Git branches for feature development. Use Terraform workspaces for environment management. Your main branch should represent the code that defines *all* your environments, with differences handled by variables.

Best Practices for Using Terraform Workspaces

• Use Remote State: Always use a remote backend (like S3, Azure Blob Storage, or Terraform Cloud) for your state files. This provides locking (preventing two people from running apply at the same time) and durability. Remote backends fully support workspaces.
• Keep Environments Similar: Workspaces are best when environments are 90% the same. If prod is radically different from dev, they should be in separate root modules (directories).
• Prefix Resource Names: Always use ${terraform.workspace} in your resource names and tags to prevent clashes.
• Use the Map Variable Method: Prefer using a variable "env_config" map (Method 1) over passing -var-file flags (Method 2). It’s cleaner, less error-prone, and easier for CI/CD.
• Secure Production: Use CI/CD with branch protection and manual approvals (like the GitHub Actions example) before applying to a prod workspace. Restrict direct CLI access to production state.
• Know When to Graduate: If your env_config map becomes gigantic or your main.tf fills with count and if logic, it’s time to refactor. Move your configuration into shared Terraform modules and consider a tool like Terragrunt for orchestration.

Frequently Asked Questions

Q: Are Terraform workspaces the same as Terragrunt workspaces?

A: No, this is a common source of confusion. Terragrunt uses the term “workspaces” in a different way, more akin to how Terraform uses root modules. The feature discussed in this article is a built-in part of Terraform CLI, officially called “Terraform CLI workspaces.”

Q: How do I delete a Terraform workspace?

A: You must first switch to a *different* workspace, then run terraform workspace delete <name>. You cannot delete the workspace you are currently in, and you can never delete the default workspace. Remember to run terraform destroy *before* deleting the workspace if you want to destroy the resources it managed.

Q: What’s the difference between Terraform workspaces and modules?

A: They are complementary. A module is a reusable, packaged set of .tf files (e.g., a module to create an “RDS Database”). A workspace is a way to manage separate states for a *single* configuration. A good pattern is to have a single root configuration that calls multiple modules, and then use workspaces to deploy that entire configuration to dev, staging, and prod.

Q: Is the ‘default’ workspace special?

A: Yes. It’s the one you start in, and it cannot be deleted. Many teams avoid using the default workspace entirely. They immediately create dev or staging and work from there, leaving default empty. This avoids ambiguity.

Conclusion

Terraform workspaces are a fundamental and powerful feature for managing infrastructure across multiple environments. By isolating state files, they allow you to maintain a single, clean, and reusable codebase, adhering to the DRY (Don’t Repeat Yourself) principle. When combined with a robust variable strategy (like the environment map) and an automated CI/CD pipeline, workspaces provide a simple and effective solution for the dev/staging/prod workflow.

By understanding their strengths, and just as importantly, their limitations, you can effectively leverage Terraform workspaces to build a scalable, maintainable, and reliable infrastructure-as-code process. Thank you for reading the DevopsRoles page!

DevOps, Terraform