People love to say things like “let’s move to the cloud”, “get started in minutes” or “just put it into BigQuery”, but developers understand that there is much more to it than that. The journey to move completely to the cloud is one that takes a lot of care and consideration. However, in many cases for large enterprise organisations it requires strong adherence to IT governance and security policies, especially when deploying to Production.

At Liveli and NGIS, we spend time helping large companies with the prerequisites and best practices they need to adopt cloud technology – particularly when it comes to moving geospatial workloads and data to the cloud. For example, a business may want to utilise BigQuery within Google Cloud Platform (GCP) to perform spatial analyses in a highly integratable and scalable way, or rather deploy Virtual Machines (VM) running proprietary GIS software. 

When it comes to running these workloads in production, the process to provision the individual services manually is relatively straight forward, or at least well-defined. But, how do you do this in an automated and repeatable fashion? This is where using products like HashiCorp Terraform can help. 

What is Terraform? 

Terraform is an open source tool that lets you provision infrastructure on any cloud or on-premise. This enables users to deploy GCP resources – such as VM, containers, storage, etc, with declarative configuration files. 

This Infrastructure-as-Code (IaC) approach promotes DevOps best practices for change management and lets you manage your infrastructure just like you would software.

In order to execute the deployment of these resource templates, Terraform allows for integration between several GitOps tools, such as GitHub, Azure DevOps, GCP Cloud Build. HashiCorp also provides their own cloud-based continuous deployment tool called Terraform Cloud.

With all these great tools at your disposal, sometimes the process of utilising them together presents roadblocks. Here is an issue we encountered and how we solved it.

Integrating Terraform with GitHub and Google Cloud

We wanted to use Terraform Cloud to automatically deploy GCP resources defined in a GitHub repo into our GCP organisation. The first step is integrating these three tools to create a DevOps pipeline, securely.

There is a lot of documentation regarding these individual tools, but nothing that described this process end-to-end. So here it is, for you! 

1. Set up a Terraform Cloud workspace

The first thing you need is a Terraform Cloud (TFC) account.

Create an account, or sign in to TFC at https://app.terraform.io/app. Ensure a TFC organisation has been created, or create a new one.

Create a new TFC workspace, which will be used to manage the automated ‘terraform plan’ and ‘terraform apply’ actions to GCP.  

During the creation, you will need to configure the version control settings, which in our case is to integrate with a GitHub repo.
 

Choose the repository that hosts your Terraform source code.

Under Advanced options, leave defaults, all git commits will trigger a run automatically.
 

Click “Create workspace”. Now you have a Terraform workspace. But you can’t deploy to GCP just yet. 

We need to configure this TFC workspace to be authenticated with GCP and be authorised to provision resources in GCP.

2. Create a GCP project and service account

You will need to create a GCP service account for which a key will be generated and used by Terraform Cloud. This service account will be granted GCP IAM policies in order to authorise Terraform on what it can provision in GCP and where (e.g organisation, folder or project-level).

The only way to create a service account is to create a GCP project and then the service account within it.

Within the GCP cloud console, create a new project. We shall refer to this as the “seed” project. Create a service account inside the seed project.
 

Create a private key for the service account.

The key will automatically be downloaded to your computer as a JSON file. Keep this safe.

3. Grant permission to the service account

You will now need to apply IAM policies to the service account to authorise it to provision resources in GCP.

IMPORTANT: The policies that you grant to the service account are dependent on your use case. For example, you may wish to create projects and organisation-level resources using Terraform, or you may have more restrictive purposes, e.g. provisioning specific resources into GCP projects. 

Read the guide on Using resource hierarchy for access control for more information on IAM policies.

4. Configure Terraform with service account key

Now that you have created a service account within GCP, created a key and also granted this account permission to provision resources in GCP – you are now ready to configure Terraform Cloud authentication to GCP.

Back in TFC, click “Configure variables”.

Scroll down to the “Environment variables” section and click “Add variable”.
 

IMPORTANT: TFC doesn’t support any newline characters in environment variables, therefore, before adding it you must remove the newline characters. If you don’t do this authentication to GCP will fail.

5. Prepare service account key

Open the service account JSON key you downloaded from GCP previously, using Visual Studio Code (you can, of course, perform this with other editors).

Using Visual Studio Code, open the .json file.

Press CMD+F to bring up the find dialog box. Click the arrow on the left hand side to use the Find + Replace tool.

In the find input field, press CTRL + enter to make a new line. Leave the replace field blank. Now click on the ‘replace all’ button in the bottom right.
 

This is the key file content before (don’t worry, this key doesn’t exist anymore so don’t bother trying to use it):
 

This is how it should look after:
 

6. Finalise Terraform environment variables

Now that you have prepared the key, you can resume entering the environment variable into TFC.

For the Key, enter “GOOGLE_CREDENTIALS”.

For the Value, paste in the prepared key (no newline characters). Check the “Sensitive” option to hide the key.
 

BONUS: Depending on what resources you are deploying and how your Terraform code is written, you may want to include variables in Terraform to be used at runtime, rather than storing sensitive information within your code base. An example of this could be the GCP organisation ID, or your GCP billing account number. To add these, navigate to the Variables menu in TFC and add the key values.

7. Deploy resources to GCP from Terraform Cloud

You are now ready to deploy your resources to GCP. Perform a “Queue Plan” and “Apply” from TFC.
 

You can now check GCP to see that your resources are deployed.

Summary

You have now created a IaC pipeline to deploy GCP resources using Terraform, using an integration between GitHub, TFC and GCP. This of course, is only one element of the Infrastructure-as-Code journey with Terraform, however, it is an essential part of creating a secure and automated process.

If you would like to know more about how you can automate resource deployment to GCP, take a look at Google’s Cloud Foundation Toolkit. From here you can browse several reference templates for provisioning GCP resources.

Additionally, if you would like to learn more about Terraform and Google integration libraries, check out the Terraform Provider Registry for Google Cloud Platform.

If you are interested in learning more about how you can automate your cloud deployments, please reach out to the team at team@liveli.com.au.