Welcome back to another blog in the DevOps series. In the last post, I got my GitLab structure set up to act as the central, golden repo for this environment, and I also got a cloud-init template, an API token, and a dedicated user configured on Proxmox. That work was all groundwork. This post picks up exactly where that one left off.
Here's what this one covers. First, setting up variables in GitLab to securely store the API credentials from the previous post. Second, getting a GitLab runner installed on Proxmox so GitLab actually has something to execute jobs on. Third, walking through the Infrastructure as Code repo, and how I'm thinking about structuring things. And fourth, testing the pipeline end to end to see if it actually spins up a VM the way it's supposed to.
Why Variables Instead of Hardcoded Secrets
The first thing I needed to do was get my variables set up in GitLab. The reason is simple: I don't want to hardcode an API token and secret directly into pipeline files.
CI/CD variables are key-value pairs that GitLab stores and makes available to pipelines and jobs. Instead of putting sensitive information directly in code, you store it as a variable and reference that variable by name in the pipeline. Anyone looking at the code only sees the variable name, never the actual value, which keeps things clean and secure.
GitLab gives you a few behaviors to choose from when defining a variable:
- Masked, so the value is hidden in job logs and never exposed in output.
- Protected, so the variable is only available on protected branches or tags.
- Hidden, so the value isn't displayed in the UI after it's saved.
- File type, where instead of being passed as an environment variable, the value is written to a temporary file. This is useful for things like certificates or SSH keys, which I'll likely touch on later in the series once I start pushing keys out to VMs.
I do plan to stand up a dedicated secrets manager down the road for more granular control, but for now, GitLab's built-in CI/CD variables get the job done.
For this setup, I created three variables at the project level, meaning scoped to a single repo: one for the API token ID, one for the API token secret, and one for the URL of the hypervisor itself. The token ID and secret came straight from the API token I generated in the last post.
The names you give these variables need to be exact and memorable, because you'll reference those exact names later in the pipeline. Get the name wrong, and the pipeline simply won't be able to talk to anything.
Getting a Runner in Place
With variables configured, the next piece is a GitLab runner living on Proxmox. A runner is a lightweight agent that executes the jobs defined in a CI/CD pipeline. Think of it as the worker that picks up jobs from GitLab and actually runs them. Runners can operate at different scopes too: a shared runner usable across multiple projects, a project-specific runner dedicated to one project, or a group runner shared within a GitLab group.
Alongside runners sits the concept of executors, which is worth understanding before you set anything up. If a runner is the worker, the executor answers the question of where that work actually happens. A few of the common options:
- Shell, where the job runs directly on the server. Fast and simple to set up.
- Docker, where the job runs inside a freshly created container that disappears once the job is done.
- Kubernetes, where pods spin up inside a cluster to run jobs.
- VirtualBox, where the job runs inside a virtual machine.
- SSH, where the runner logs into another machine and executes the job there.
- Custom, where you build your own setup for full control.
For this build, I went with the Shell executor. Eventually I'd like to build a custom Docker image with all the tools I need, push it to a registry, and switch to the Docker executor, but that's later in the series.
To get a runner in place, I started with a clean server. Since I already had a cloud-init template ready from the last post, I just cloned it, gave it a name, chose a full clone, picked storage, and bumped the disk size up a bit since the runner needs some room to work.
Here's the part that's easy to miss: even though I'm using the Shell executor, Docker still needs to be installed on that runner server. That's because the pipeline itself uses Docker to pull down and run a Terraform or OpenTofu image. Docker has to be present for that to work, regardless of which executor you're using.
Once Docker was installed and running, I added the GitLab runner package repository, installed the runner software, and confirmed the service was healthy. From there, it's a matter of registering the runner against the GitLab instance, which is really just telling GitLab "I've got a worker ready in this environment, here's what it is, put it to use."
Registering a runner happens at whatever scope you choose, in this case the project level. When you create it, you'll set a tag, and that tag is genuinely important, because it's what ties jobs in your pipeline back to this specific runner. Remember whatever tag you choose, because it shows up again in the pipeline configuration.
I left the option to run untagged jobs unchecked, since I only want this runner picking up jobs I've explicitly tagged for it. You can also add a description, choose to pause, protect, or lock the runner to the project, and set a maximum job timeout if you want one.
Once you create the runner, GitLab hands you a registration command along with the instance URL and a token. Before copying any of that, make sure you've selected the correct operating system in the UI, or you'll get instructions that don't match your server.
Back on the server, running the registration command walks you through a short series of prompts: the GitLab instance URL, the registration token, a local name for the runner, and finally which executor to use. I chose Shell again here. It's not the cleanest long-term setup, but it's a solid starting point, and something I plan to revisit once the custom Docker image is ready.
Once registration finishes, hopping back into GitLab's runner list shows the new runner online and ready to pick up jobs. At this point, GitLab has everything it needs on both sides: variables holding credentials, and a runner ready to execute.
Walking Through the Repo Structure
With the runner and variables in place, it's time to look at how the actual Infrastructure as Code repo is put together. This is where all the logic lives, and there are a handful of key files worth knowing at a glance. The full contents of each one are laid out in the Command Reference Guide at the end of this post.
At the root of the repo sits the pipeline definition file, essentially the brain of the whole operation. It defines three stages that run in order, validate, plan, and apply, and tells GitLab exactly what each one should do. It's a living file that will keep evolving as the environment grows, and as far as I know, a repo can only have one of these.
Alongside it, I created a dedicated Terraform/OpenTofu directory to keep the actual infrastructure code organized and separate from the pipeline logic. Inside it are five files, each with its own job: an outputs file that starts out empty as a placeholder for information the tool can hand back after a run, a provider file that tells the tool which provider to use and how to connect to the hypervisor, a variable values file holding the settings specific to your own environment, a variable declarations file defining every variable the configuration expects, and a VM definition file, arguably the most important one in the repo, where the actual VM gets described. There's also a small placeholder file whose only job is to give Git something to track in what would otherwise be an empty folder.
Put together, each file plays its part: the declarations describe what's needed, the values and pipeline variables supply the actual data, the provider file handles the connection, the VM file defines what gets built, and the outputs file will eventually surface the results back. Every variable and value throughout all of this needs to reflect your own environment, not mine.
Running the Pipeline End to End
With everything wired up, here's how it all flows together. Pushing a commit to the repo triggers GitLab to pick it up and run the pipeline, which reads the pipeline definition file to know exactly what to do. It spins up the OpenTofu (or Terraform) Docker image on the runner and works through the stages in order.
First, validate confirms the configuration is clean and free of errors. Then plan connects to the hypervisor using the stored credentials and generates a detailed execution plan based on what's defined in the VM file, saving that plan as an artifact to hand off.
Finally, apply waits for manual approval, and once given, takes that saved plan and executes it against the hypervisor, cloning the template and standing up the VM exactly as defined. Code goes in, a VM comes out. That's the whole point of the setup.
Worth clarifying, since I go back and forth on terminology: OpenTofu is an open source fork of Terraform, and the commands are essentially identical. Wherever you see an OpenTofu command in the pipeline, the equivalent Terraform command works exactly the same way. Swap freely if that's your preference.
Right now, any push to this repo triggers a pipeline run regardless of what changed, which is fine while I'm still building and testing, but something I'll need to refine as the repo grows. GitLab does support scoping pipeline triggers to specific file changes or branches, and that's on the list for later.
Pushing a small edit kicks off the pipeline, validate and plan pass automatically, and the apply stage sits waiting for a manual green light. Approving it, the VM appears on the hypervisor, boots up, and becomes reachable, all without touching the hypervisor console directly. If any stage fails along the way, the pipeline is generally good about surfacing a clear error and detailed logs pointing to exactly what needs fixing before it'll move forward.
From here on, spinning up a new VM is just a matter of editing the VM definition file (more memory, more cores, more storage, whatever's needed), committing, and letting the pipeline do the rest.
What's Next
Everything is now wired up so that a simple push can spin up a new VM without touching the hypervisor console at all. There's still cleanup to do and things to tighten as I go, but this is solid progress. Next up in the series: automating container creation, and automating the teardown of these resources as well. There's a lot more to build, but there's no rush, no chasing the newest shiny tool for its own sake. Just building a solid foundation, one step at a time.
