In the first phase of this project, I built a pipeline that could automatically deploy virtual machines to Proxmox using GitLab and OpenTofu. Getting a VM to appear in Proxmox from code was already a big step, but once that worked, I immediately started thinking about what the next version should look like.
The reality is that my environments is not made up of one type of resource. I use virtual machines, but I also use LXC containers. More importantly, I did not want automation that could only create infrastructure. I wanted to be able to manage the entire lifecycle.
That meant being able to create resources, change them, track them, and eventually remove them without having to jump into the Proxmox interface and start clicking buttons.
The goal is to move closer to managing the complete lifecycle of my Proxmox infrastructure through Git. Infrastructure is defined in configuration files that can be versioned and reused. Getting there required me to rethink quite a few things from the first version.
Where the first version fell apart
After I finished the first version of the pipeline, I started thinking harder about how I actually wanted this infrastructure to be built. My first instinct was to keep everything inside a single OpenTofu project, exactly the way I showed it before. But the more I worked with it, the more obvious it became that it was not going to scale.
The main problem was that every time I wanted to deploy a VM, the pipeline also evaluated my container configuration. And it happened in reverse too. If I wanted to deploy a container, the pipeline still evaluated my VM configuration and tried to reconcile both at the same time. It worked. It just wasn't what I wanted.
Restructuring the repository
Before we get into that separation, let's look at the overall repository structure, because I reorganized things there as well. Everything now lives inside a top level lab folder. Inside that I have an infrastructure folder, and inside that I have folders for virtualization, servers, and documentation. All of my hypervisor automation lives in the virtualization folder, and that is where the infrastructure-as-code repository sits. That repository is really the first major infrastructure repo for this environment.
Under the terraform folder there is now a directory for virtual machines, a directory for containers, and a third directory called destroy that we will get to shortly.
Both the VM and container directories contain the same core files: backend.tf, provider.tf, variables.tf, terraform.tfvars, and outputs.tf, plus either vms.tf or containers.tf depending on which one you are looking at.
The reason for that separation is that it gives each deployment its own isolated state. My VM deployments now know nothing about my containers, and my container deployments know nothing about my VMs.
Walking through the files
I changed a fair number of things inside these files. The container side and the VM side are nearly identical, so I will focus on containers here. The same ideas apply to both.
The variables file defines every variable used in the deployment. I cleaned it up and added a few new ones: container name, container ID, SSH public key, password, gateway, and IP address. Worth flagging that this is temporary. I am using GitLab CI/CD variables for now and they work fine at this stage, but a dedicated secrets manager is coming soon because I need something better for handling keys and credentials as this grows.
The tfvars file assigns the actual values. Name, ID, IP, gateway, template, storage, bridge, node. The variables file defines the shape, this file fills it in. The one line worth calling out is the template, which just points at an LXC template I downloaded through the hypervisor's own template section. Download it, save it to your storage, reference it here.
The provider file barely changed. I bumped the version and left everything else alone. The outputs file was empty last time. Now it prints useful details after something is created, which gives me visibility straight from the pipeline log.
The backend file is the important addition, and I had nothing like it before. It tells OpenTofu where to store its state. State is an inventory of everything OpenTofu has built. It tracks every resource, so OpenTofu knows what already exists and what needs to be created, updated, or destroyed. Without it, OpenTofu has no memory of what it built, which means it has no idea what to remove when you ask it to tear something down. That is what makes automated destruction possible.
State can live locally on the runner or on external storage. I went with the GitLab remote HTTP backend so it stays centralized and every pipeline run works from the same source of truth, no matter where it executes.
The resource file, either containers.tf or vms.tf, defines the thing being deployed. CPU, memory, disk, network, and the rest. The variables above get wired in here.
Rethinking the pipeline
Splitting the project in two meant the pipeline had to change with it. Instead of one pipeline doing everything, I now have separate jobs for VMs and containers, each with its own validation, plan, deploy, and destroy stages. The file has grown quite a bit, and I have more ideas queued up. Here are the changes that matter.
The pipeline watches specific files. Instead of evaluating everything on every commit, it triggers on what actually changed. Touch the VM resource file and only the VM jobs run. Touch the container file and only the container jobs run. This is the change I said was coming last time, and it keeps deployments isolated instead of firing jobs I never asked for.
State is split too. VMs get their own state, containers get their own. That makes the environment easier to manage and cuts down the chance of one deployment stepping on the other.
The manual approval step is gone. I originally added that gate so I could watch every run while I was still building this out. It behaved, so I removed it. Changes land on main, GitLab validates, plans, and deploys, and I am not in the loop.
The pipeline handles more configuration. Username, password, and SSH public key now come in through GitLab CI/CD variables and get set during provisioning, so there is no manual setup after a machine boots. This is another area I expect to rework once the secrets manager is in place.
Destroy is a first class workflow. Commit a name to a file in the destroy directory, push to main, and the pipeline first verifies that the resource actually exists in state. If it checks out, it generates a destroy plan and removes it. That validation step is what gives me confidence I am deleting exactly what I meant to delete.
Only main deploys. Not a big deal right now since I am working from main anyway, but once I start branching to test ideas, I want the hypervisor updated only after changes have been reviewed and merged. Easier to build the rule in now than to remember it later.
Watching it actually work
With everything in place, I ran the full loop end to end. I started with a container. Opened the resource file, changed the CPU core count, committed straight to main. The pipeline picked it up right away and moved through validation, plan, and apply. Nothing on the hypervisor at first, then the container appeared, built itself, and booted. No clicking, no manual steps, just a commit.
The VM side stayed quiet, exactly as it should have. The separation was doing its job. To tear the container back down, I grabbed its name, pasted it into the container file in the destroy directory, and committed. New pipeline, and the container removed itself.
Then the same loop for a VM. Change the resource file, commit, watch it build and boot with the SSH key already installed and the password already set. Paste the name into the VM destroy file, commit, watch it shut down and disappear.
Where this goes next
It took a lot of reading, testing, and validating to get here, but the environment finally feels streamlined. I knew from the beginning that I wanted to clean up my lab as easily as I create it, and this workflow gets me there. Automated deployment, automated destruction, and every line of code that drives both sitting in version control. That is a win.
Next I want to shift focus. I can deploy VMs and containers, but I don't want to stop at "it exists." What should every system look like the moment it comes online? What security tooling is installed, what certificates does it hold, what monitoring agent is already running, what baseline configuration does it receive? The goal is that all of that happens automatically as part of the deployment, whether through another pipeline, configuration management, or something else.
I am also weighing whether to separate my management plane from my lab plane. I am not sure I want security tooling and core management services living on the same host that runs the CI/CD environment. I am leaning toward a small dedicated host for that, but I want to think it through before committing.
That is the theme going forward. I don't want to build just to build. I want security designed in from the ground up rather than bolted on, and I want to treat this like a real production environment, because that is when you stop asking how to install a service and start asking how everything connects and how you secure it.
