preloader
blog-post

How to create Snowflake resources using Terraform

author image
Table of Contents

The Terraform Snowflake Provider allows to provision Snowflake resources using declarative Infrastructure as Code(IaC). In this article, we will see how to provision some common resources in Snowflake. The resources will follow the naming standard described inĀ Snowflake object naming conventions. Code used in this article can be found here.

This blog will not teach basics of Terraform or Snowflake. It assumes that you know the basics and shows how to quickly spin up a working setup to create Snowflake resources using Snowflake.

Prerequisite

  • Download and install docker for your platform. Click here for instructions
  • Create a Snowflake account for the demo. Click here for instructions
  • Create Terraform cloud account for the demo. Click here for instructions

Account setup

Terraform

  • Login into your Terraform cloud account and navigate to User Settings –> Tokens and create a new token. This new token will be used by Terraform to maintain the state file in cloud

  • Create workspace in Terraform cloud with proper postfix for required environment names. Here is an example of workspace naming

    It’s not mandatory to use Terraform cloud for storing the state, you can use other services like Amazon S3, Azure Blob Storage, Google Cloud Storage, etc. for storing the remote state

Snowflake

  • We will need a Snowflake user and role to create the rest of the resources

  • Login into Snowflake and execute below SQL to create the user and role

    -- Create role and grant required access for TF to operate
USE ROLE SECURITYADMIN;
CREATE ROLE IF NOT EXISTS ENTECHLOG_TERRAFORM_ROLE;

CREATE USER IF NOT EXISTS ENTECHLOG_TF_USER DEFAULT_ROLE = ENTECHLOG_TERRAFORM_ROLE;
GRANT ROLE ENTECHLOG_TERRAFORM_ROLE TO USER ENTECHLOG_TF_USER;

GRANT CREATE ROLE ON ACCOUNT TO ROLE ENTECHLOG_TERRAFORM_ROLE;
GRANT CREATE USER ON ACCOUNT TO ROLE ENTECHLOG_TERRAFORM_ROLE;
GRANT MANAGE GRANTS ON ACCOUNT TO ROLE ENTECHLOG_TERRAFORM_ROLE;

GRANT ROLE ENTECHLOG_TERRAFORM_ROLE TO ROLE SECURITYADMIN;
GRANT ROLE ENTECHLOG_TERRAFORM_ROLE TO ROLE SYSADMIN;

USE ROLE ACCOUNTADMIN;
GRANT CREATE INTEGRATION ON ACCOUNT TO ROLE ENTECHLOG_TERRAFORM_ROLE;

USE ROLE SYSADMIN;  
GRANT CREATE DATABASE ON ACCOUNT TO ROLE ENTECHLOG_TERRAFORM_ROLE;
GRANT CREATE WAREHOUSE ON ACCOUNT TO ROLE ENTECHLOG_TERRAFORM_ROLE;

Development environment setup

For the purpose of the demo we will use a docker container called developer-tools which has Terraform and tools required for the demo.

Start container

  • Clone developer-tools repo

    git clone https://github.com/entechlog/developer-tools.git

  • cd into developer-tools directory and create a copy of .env.template as .env. For the purpose of demo, we don’t have to edit any variables

    cd developer-tools

  • Start the container

    docker-compose -f docker-compose-reg.yml up -d --build

Validate container

  • Validate the containers by running

    docker ps

  • SSH into the container

    docker exec -it developer-tools /bin/bash

  • Validate terraform version by running below command

    terraform --version

Terraform Resources

  • Clone the snowflake-examples repo to get started

  • cd into snowflake-examples/snow-objects/terraform. This directory contains .tf with the required resource definition. All configuration can be specified in a single file, but for the ease of understanding and managing them in long run, we will create multiple .tf files

  • The first file to look at is providers.tf. Provider allow Terraform to interact with Snowflake. Here, instead of hard coding the Snowflake credentials, we will make use of Terraform variables

    terraform {
  required_providers {
    snowflake = {
      source  = "Snowflake-Labs/snowflake"
      version = "0.47.0"
    }
  }
}

provider "snowflake" {
  account  = var.snowflake_account
  region   = var.snowflake_region
  username = var.snowflake_user
  password = var.snowflake_password
  role     = var.snowflake_role
}

  • Variables are defined inside a file called variables.tf in the same directory. The values for variables are specified in terraform.tfvars for local runs. Create terraform.tfvars from terraform.tfvars.template and update with the details about Snowflake account. When using Terraform cloud, values should be specified in workspace variables as shown below

  • main.tf file contains the resource blocks for declaring resources. The resources can be Snowflake user, database, schema, tables, etc. Please see Snowflake provider documentation for the list of supported Snowflake resources. Here is an example for creating Snowflake user, role using Terraform

    //***************************************************************************//
// Create Snowflake user
//***************************************************************************//

// Managing Snowflake users using Terraform will put the password in the terraform state file
// This is not recommended method for creating users
// Rather create users without password and set them latter OR use more secure options like SCIM
// https://docs.snowflake.com/en/user-guide/scim.html

resource "snowflake_user" "demo_user" {
  name         = lower("DEMO_USER")
  login_name   = "DEMO_USER"
  comment      = "Snowflake user account for demo"
  password     = "demouser"
  disabled     = false
  display_name = "Demo User"
  email        = "demo_user@example.com"
  first_name   = "Demo"
  last_name    = "User"

  default_warehouse = snowflake_warehouse.dev_entechlog_demo_wh_s.name
  default_role      = snowflake_role.entechlog_demo_role.name

  must_change_password = false
}

//***************************************************************************//
// Create Snowflake role
//***************************************************************************//

resource "snowflake_role" "entechlog_demo_role" {
  name    = "ENTECHLOG_DEMO_ROLE"
  comment = "Snowflake role used for demos"
}

//***************************************************************************//
// Create Snowflake role grants
//***************************************************************************//

resource "snowflake_role_grants" "entechlog_demo_role_grant" {
  role_name = snowflake_role.entechlog_demo_role.name
  roles     = ["SYSADMIN"]
  users = [
    "${snowflake_user.demo_user.name}"
  ]
}

  • Create the resources by applying the terraform template

    # install custom modules
terraform init -upgrade

# format code
terraform fmt -recursive

# plan to review the summary of changes
terraform plan

# apply the changes to target environment
terraform apply

Terraform Modules

  • To create multipleĀ users using the above approach, we will have to create multiple resource blocks. This makes the code lengthy. To overcome this, we can create Terraform modules. As per official documentation, ā€œAĀ moduleĀ is a container for multiple resources that are used together. You can use modules to create lightweight abstractions, so that you can describe your infrastructure in terms of its architecture, rather than directly in terms of physical objects.ā€

  • Here some general design considerations

    • Idea is to have 3 workspaces in terraform, say snowflake-dev, snowflake-stg and snowflake-prd
    • Development and Stage deployments happen when code is merged to develop branch.
    • Production deployments happen only when code is merged to main branch.
    • Common resources like user, roles are created during development deployment
    • Deployment trigger is controlled by “VCS branch” configuration in Terraform workspace settings
    • These design considerations are configured in snowflake-examples/snow-objects/terraform/modules/01-snowflake-config.tf

  • cd into subdirectories inside snowflake-examples/snow-objects/terraform/modules view module definitions. Every module definition will have providers.tf, variables.tf, main.tf and output.tf. Here is an example for module definition for Snowflake user inside user directory

  • providers.tf contains the providers details

      terraform {
    required_providers {
      snowflake = {
        source  = "Snowflake-Labs/snowflake"
        version = "0.47.0"
      }
    }
  }

  • variables.tf contains the input variable details. Here we are using a variable named user_map of datatype map. This allows to accept multiple users as input to this module

      variable "user_map" {
    type = map(any)
  }

  • main.tf contains the resource definition. Here we are looping over input variable user_map and accessing key and value in them to create the required users

      resource "snowflake_user" "user" {

    for_each = var.user_map

    name         = lower(each.key)
    login_name   = lower(each.key)
    disabled     = false
    display_name = "${title(each.value.first_name)} ${title(each.value.last_name)}"
    email        = lookup(each.value, "email", "NONE") == "NONE" ? "" : each.value.email
    first_name   = title(each.value.first_name)
    last_name    = title(each.value.last_name)

    default_warehouse = lookup(each.value, "default_warehouse", "NONE") == "NONE" ? "" : each.value.default_warehouse
    default_role      = lookup(each.value, "default_role", "NONE") == "NONE" ? "PUBLIC" : each.value.default_role

    must_change_password = false
  }

  • output.tf contains the output variable details

      output "user" {
    value = snowflake_user.user
  }

  • After creating the module, here is how we invoke them. The module block contains source, which points to the directory with the module definition. Looking at the below example, we have created multiple users with a couple of lines of code which points to the reusable resource block rather than defining the resource block multiple times

    module "all_service_accounts" {
  source = "./user"
  user_map = {
    "${lower(var.env_code)}_entechlog_demo_user" : { "first_name" = "Demo", "last_name" = "User" },
    "${lower(var.env_code)}_entechlog_dbt_user" : { "first_name" = "dbt", "last_name" = "User" },
    "${lower(var.env_code)}_entechlog_atlan_user" : { "first_name" = "Atlan", "last_name" = "User" },
    "${lower(var.env_code)}_entechlog_kafka_user" : { "first_name" = "Kafka", "last_name" = "User", default_role = "${upper(var.env_code)}_ENTECHLOG_KAFKA_ROLE" }
  }
}

  • By using this approach, you can create modules and simplify the Terraform usage

Connecting Terraform to version control provider

In the previous example, we applied the terraform changes manually. To automate the process, we can connect our Terraform cloud workspace to a version control provider.

  • Navigate to version control properties of the workspace to connect to the version control provider. Here we are going to use GitHub

  • Update Terraform Working Directory to the directory which contains the module blocks. In this example, it would be snow-objects/terraform/modules/

  • Set the Apply Method as Auto apply

  • Set the Run Triggers as Only trigger runs when files in specified paths change, this would be snow-objects/terraform/modules/

  • Enable Automatic speculative plans in the version control properties so Terraform plans will be auto generated for all PR’s with changes to modules directory

  • Once the PR is reviewed and approved, changes will be automatically deployed to Snowflake

Hope this was helpful. Did I miss something ? Let me know in the comments OR in the forum section.

This blog represents my own viewpoints and not of my employer, Snowflake. All product names, logos, and brands are the property of their respective owners.

References

Share this blog:
Comments

Related Articles

All rights reserved to Entechlog Ā©