Deploying an S3 Terraform Remote Backend... With Terraform

Deploying

Terraform tracks deployed resources using a state file. By default, this file is stored locally on the machine that performs the deployment. If you're working alone, or deployments are always performed by the same person, this may be adequate. In larger teams, where multiple people need to manage the existing structure, we need to move this state file to a shared environment.

Terraform supports a number of remote state storage options, but for this example we'll be using an S3 bucket on AWS. The remote backend is defined using a backend block, alongside the other Terraform configuration;

terraform {
  backend "s3" {
    bucket = "s3-terraform-state-test-bucket"
    key    = "terraform.tfstate"
    region = "us-east-1"
  }
}

As we're all in on deploying our infrastructure using Terraform, your next question may be; "Can I deploy this S3 bucket to store my Terraform state using Terraform"?
Deploying infrastructure in one step and then using properties of that deployed infrastructure during the deployment of other infrastructure is a common thing to do in Terraform, which may lead you to try something like this;

resource "aws_s3_bucket" "state" {
  bucket = "terraform-state-bucket"
}

terraform {
  backend "s3" {
    bucket = aws_s3_bucket.state.id
    key    = "terraform.tfstate"
    region = "us-east-1"
  }
}

Unfortunately, if you try to run terraform apply using this configuration, you'll be greeted with an error telling you "Variables may not be used here";

Initializing the backend...
╷
│ Error: Variables not allowed
│
│   on main.tf line 8, in terraform:
│    8:     bucket = aws_s3_bucket.state.name
│
│ Variables may not be used here.
╵

Thankfully, all we need to do is break down the deployment and utilisation of our state bucket into two steps. First, comment out the backend block, leaving just the definition of the S3 bucket itself;

resource "aws_s3_bucket" "state" {
  bucket = "terraform-state-bucket"
}

#terraform {
#  backend "s3" {
#    bucket = aws_s3_bucket.state.id
#    key    = "terraform.tfstate"
#    region = "us-east-1"
#  }
#}

Running terraform apply will now create the S3 bucket. This will create a local state file, but once everything is deployed we can move that state file to our newly created bucket. Uncomment the backend configuration and change the bucket definition to match the name of the bucket we deployed;

resource "aws_s3_bucket" "state" {
  bucket = "terraform-state-bucket"
}

terraform {
  backend "s3" {
    bucket = "terraform-state-bucket" # Note this has changed to match the name of the bucket above
    key    = "terraform.tfstate"
    region = "us-east-1"
  }
}

Finally, run terraform init. You'll be asked if you want to copy the local state file to our new remote backend.
Note: if the AWS profile you're using to deploy these resources isn't the default, you'll need to append -backend-config="profile=aws-profile-name" to that terraform init call, otherwise you'll get an "Access Denied" error.

> Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

That's it! We've now deployed an S3 bucket to AWS and migrated our Terraform state. Any future deployments will now use this remote state file, whether performed by us or anyone else on the team.

Destroying

If you ever need to destroy all the infrastructure Terraform has deployed with this configuration, you'll need to perform these steps in reverse. First, comment out our backend configuration again, and run terraform init -migrate-state. You'll be asked to confirm you want to copy the existing remote state to a local file;

Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "s3" backend to the
  newly configured "local" backend. No existing state was found in the newly
  configured "local" backend. Do you want to copy this state to the new "local"
  backend? Enter "yes" to copy and "no" to start with an empty state.

Once this is done you can run terraform destroy to remove the S3 bucket and any other resources that were deployed by Terraform.