Efficient resource management with Terraform in a mono-repo environment

This article presents a structured approach to using Terraform within a mono repo for managing multiple domains, each with distinct environments and resources. By implementing this strategy, teams can achieve autonomous domain management, environment-specific deployments, and streamlined state handling using GitHub Actions for continuous integration and delivery (CI/CD).

Overview of the strategy

The goal is to utilize a single repository (mono-repo) to manage several autonomous business domains — each with its own set of environments (development, pre-production, and production) and resources. The key features of this strategy include:

• Domain-specific state management: Each domain maintains its own independent Terraform state, allowing for isolated and efficient management of resources.
• Environment layers: Each domain contains three environment layers (dev, pprd, prd), supporting progressive changes and robust testing.
• CI/CD with GitHub actions: Automation is handled via GitHub Actions, enabling consistent deployments and infrastructure updates across all domains and environments.

The directory structure for Terraform mono-repo

/terraform-monorepo
|
|-- /domains
|   |-- /domain1
|   |   |-- main.tf          # General resources applicable to all environments
|   |   |-- variables.tf     # Variables definition used across all environments
|   |   |-- outputs.tf       # Output definitions
|   |   |-- backend.tf       # Backend configuration for the domain
|   |   |
|   |   |-- /env             # Environment-specific configurations
|   |       |-- /dev
|   |       |   |-- main.tf
|   |       |
|   |       |-- /pprd
|   |       |   |-- main.tf
|   |       |
|   |       |-- /prd
|   |           |-- main.tf
|   |
|   |-- /domain2
|   |   |-- ... (similar structure as domain1)
|   |
|   |-- /domain3
|       |-- ... (similar structure as domain1)
|
|-- /modules
|   |-- /network
|   |   |-- main.tf
|   |   |-- variables.tf
|   |   |-- outputs.tf
|   |
|   |-- /compute
|       |-- main.tf
|       |-- variables.tf
|       |-- outputs.tf
|
|-- /common
|   |-- terraform.tfvars (optional, for common variable values)
|   |-- common_variables.tf (defines common variables)
|
|-- README.md
|-- .gitignore
|-- .github
    |-- /workflows
        |-- terraform.yml

Explanation of key components

• /domains/: This is the top-level directory for all the business domains. Each sub-directory represents a separate domain.
• /domain1, /domain2, etc.: Each domain directory contains subdirectories for each environment. This separation ensures that configurations can be environment-specific and managed independently.
• /dev, /pprd, /prd: Subdirectories for different environments (development, pre-production, and production).

Terraform workflow design

The following is a basic CI/CD workflow setup using GitHub Actions, designed to initiate Terraform commands based on specific domain and environment inputs.

# Terraform Workflow Definition
# This workflow allows for manual triggering via GitHub Actions UI,
# where you specify the domain and environment for which to run Terraform.
# It supports multiple domains and environments within a mono-repo.
name: Terraform Workflow

# Trigger for the workflow: manual invocation with two required inputs:
# - domain: specifies the business domain to target (e.g., 'domain1').
# - env: specifies the environment within the domain to target (e.g., 'dev').
on:
  workflow_dispatch:
    inputs:
      domain:
        description: 'Domain (e.g., domain1)'  # Description for the UI
        required: true                        # This input must be provided
      env:
        description: 'Environment (e.g., dev)' # Description for the UI
        required: true                         # This input must be provided

# Jobs to be executed as part of this workflow
jobs:
  # Job for executing Terraform commands
  terraform:
    name: 'Terraform'  # Name of the job displayed in the Actions UI
    runs-on: ubuntu-latest  # The type of virtual host machine to use

    # Default settings for all run commands in this job
    defaults:
      run:
        shell: bash  # Use Bash shell for all run commands
        # Set the working directory dynamically based on the input domain and environment
        working-directory: ./domains/${{ github.event.inputs.domain }}/${{ github.event.inputs.env }}

    # Steps define a sequence of tasks that will be executed as part of the job
    steps:
    # Step 1: Check out the repository at the latest commit on this branch
    - name: Checkout
      uses: actions/checkout@v2  # Uses the v2 version of the checkout action

    # Step 2: Set up Terraform CLI
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1  # Uses the v1 version of the setup-terraform action

    # Step 3: Initialize a new or existing Terraform working directory
    - name: Terraform Init
      run: terraform init  # Command to initialize the Terraform environment
      env:
        GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}  # Use Google credentials stored in GitHub Secrets

    # Step 4: Generate and show an execution plan
    - name: Terraform Plan
      run: terraform plan  # Command to create an execution plan
      env:
        GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}  # Use Google credentials stored in GitHub Secrets

Explanation

1. Workflow dispatch: This GitHub Actions feature allows manual triggering of the workflow with specific inputs for domain and environment. It provides flexibility and control over when and where to deploy changes.
2. Working directory configuration: The working-directory is dynamically set based on the domain and environment inputs. This ensures that Terraform commands run in the correct context.
3. Terraform setup and initialization: The workflow sets up Terraform and initializes it within the specified directory, preparing the environment for further actions like plan and apply.
4. State management: To maintain an individual state for each domain (not per environment), adjustments in the Terraform configuration files are required to specify the backend configuration appropriately.

How it looks like in GitHub

Best practices for managing Terraform in a mono-repo

1. Isolate state files: Use distinct backend configurations for each domain to isolate state files. This reduces risk and increases the clarity of state management.
2. Directory structure: Organize the repository into subdirectories for each domain and within those, further subdirectories for each environment. This structure supports the logical separation of resources and simplifies navigation.
3. Securing secrets: Utilize GitHub secrets to manage sensitive information such as cloud credentials securely.
4. Modular design: Adopt a modular approach in Terraform configurations to reuse common components across different environments and domains, reducing code duplication and improving maintainability.
5. Continuous Integration: Integrate linting and compliance checks into the CI/CD pipeline to ensure that Terraform configurations are both syntactically correct and adhere to best practices.

Conclusion

Implementing Terraform in a mono-repo setup using the described CI/CD strategy can significantly enhance the manageability, scalability, and reliability of infrastructure deployments. By following the outlined practices and utilizing GitHub Actions for automation, teams can achieve more efficient and error-free operations, ensuring that each domain’s infrastructure is correctly and independently managed while maintaining overall coherence and control within a unified repository structure.

See ya 🤟