Peerobyte / Community / Questions & Answers / Best Practices for Terraform Project Structure

Best Practices for Terraform Project Structure (Edited)

Before, I managed all the infrastructure of my Terraform project by writing all the resources in a single main.tf file.

Now I want to split the terraform configuration by distributing it from one directory to several subfolders, for example:

		
			
		
		terraform-project/
├── network/
│   ├── main.tf
│   └── variables.tf
├── compute/
│   ├── main.tf
│   └── variables.tf
├── main.tf
└── terraform.tfvars

The problem occurs when I use terraform command:

		
			
		
		terraform apply network

Terraform creates a plan that will remove resources defined outside the network folder. How can I organize the repository so that changes affect only the needed group of resources and the current state is not broken?
Please tell me about best practices.

Answers

You're gonna think I'm a bore, but I highly recommend exploring the official terraform documentation. It provides a thorough introduction to getting started with Terraform and offers detailed guidance on structuring your terraform project—from basic simple structure to full production environments.

Sometimes using modules makes things too complex. Consider using Terragrunt.

Terragrunt is a tool for improving Terraform projects and managing infrastructure. I’ve seen that it makes working on big projects much easier.

It helps you manage your state in one place. You can set up remote storage like S3 or GCS with little hassle, which reduces conflicts and makes everything more reliable.

It also follows the idea of DRY (Don’t Repeat Yourself). This means you can put common settings into one template, so you don’t need to write the same Terraform code over and over. This really speeds up the deployment process.

Another cool thing is that it makes working with variables and configurations flexible. You can easily change settings for different environments like development, staging, or production, which makes scaling your project a lot simpler.

And lastly, it helps keep your code organized. With Terragrunt, even complex repositories become easier to understand and manage.

Bro, this looks like an ad ;)

Yeah, in a way. I like this tool. It helps me working with terraform and configuration management, and I recommend it.

I'm preparing to launch my Terraform workspaces in AWS. What challenges might I encounter when using Terragrunt, especially concerning the management of multiple accounts, scalability, configuration errors, and setup complexity?

I've been using Terraform with Terragrunt on Amazon Web Services (AWS) and haven't experienced any issues that are specific to the platform.

However, it's worth noting that there are some inherent nuances in using Terragrunt.

For example, if you need to manage multiple accounts, the issue of properly configuring roles and permissions immediately arises. If everything isn't set up perfectly, you may encounter problems when working with remote state (like S3 and DynamoDB). It's crucial to establish proper trust relationships between accounts to ensure everything works as intended.

Another challenge, especially if you have a large Terraform project structure, is scaling. Terragrunt allows for inheriting configurations, which may seem convenient at first, but when you have many environments, you end up with a very tangled file hierarchy. If there's an error in the base configuration, it might "leak" into several workspaces at once, making it hard to pinpoint the problem.

It's also worth noting that setting up Terragrunt itself requires extra effort. For someone accustomed to standard Terraform, the directory structure and inheritance logic can seem complicated. Initially, you end up spending a lot of time parsing through the Terragrunt and Terraform documentation and setting things up properly so you don't have to fix a bunch of minor issues later on.

Overall, Terragrunt is a powerful tool, but it demands precision and careful attention, especially when working in AWS with multiple accounts and multiple environments. If you properly configure permissions, maintain a clear structure within your Terraform configuration, and prepare for the nuances of inheritance, you can work quite effectively with it.

I don't think I need Terragrunt yet, but thanks anyway. If my project grows, I'll be upfront about this tool.

Just moving files into subfolders is not enough because Terraform only works with files in the current folder. The key solution is to change your code into modules.

In your case, you can create a separate folder for modules so that the project structure looks like this:

		
			
		
		terraform-project/
├── modules/
│   ├── network/
│   │   ├── main.tf
│   │   └── variables.tf
│   └── compute/
│       ├── main.tf
│       └── variables.tf
├── main.tf
└── terraform.tfvars

Then, in the root main.tf, include these modules with module blocks. This approach helps you manage the state in one place and avoids unwanted changes when applying changes in separate modules.

I can add that if you already have your infrastructure deployed, you can move it to a new modular structure without breaking the existing resources. Use the command:

		
			
		
		terraform state mv

This command lets you carefully move resources to a new location in the Terraform state. It helps keep the infrastructure working while you reorganize your Terraform configuration.

By default, Terraform only reads files in the current Terraform directory. Moving part of the configuration into subfolder without declaring it a Terraform module may cause Terraform to not see some of the code and try to delete the associated resources. Always set up the different sections as Terraform modules and include them with module blocks in your main Terraform file. This way, all resources stay in one state.