Networking is one of those things that can either make or break your lab environment. As your setup grows, having a solid network structure becomes more and more critical. I recently ran into an issue that forced me to stop everything I was doing and upgrade a significant part of my network before I could continue building.
This wasn't an upgrade because I wanted the latest and greatest gear. This was necessary. I had hit a real bottleneck, and I needed to fix the structure of my network before moving forward. I want to make that clear because I'm a big believer in using what you have until you outgrow it, and that's exactly what happened here.
What Forced My Hand
When I first built this environment, I started with what I already had and only bought what was necessary to get things running. My previous setup had two main network components: a 24-port switch with combo and SFP ports (all running at standard speeds), and a compact firewall device with dual ports handling all my routing. These devices are genuinely good for a small network. But as the lab started to grow, I started hitting limitations hard.
The Firewall Bottleneck
The main bottleneck came from the firewall. In this environment, the firewall handled all routing, including inter-VLAN routing. Anytime devices on different VLANs talk to each other, traffic flows like this.
device to switch, switch to firewall, firewall back to switch, switch back to device. All of that was flowing over a single connection between the switch and firewall. The main problem? As I created more VLANs and more internal services started communicating across them, that single link began to feel like a real bottleneck on top of normal network traffic.
The NIC Stability Issues
Then there were the NIC stability issues. I'm not throwing shade here, but certain network interface controllers aren't known for being the most stable under load, especially in firewall and routing scenarios. I started experiencing random internet drops when traffic increased.
I was so frustrated that I wanted to rip those ports out. I went so far as planning to add a separate network card and bypass them entirely. But I took a step back and thought, you know what? It's time to upgrade.
The Performance Ceiling
The last bottleneck was the performance ceiling. The firewall simply wasn't powerful enough to keep adding services and advanced features without impacting performance. If you start loading up your firewall with multiple services, you drastically impact the performance of the device itself, which makes complete sense. It's a great device, but it's not powerful enough when you want it to do more as your environment grows.
I knew some of these limitations from the start, especially the performance ceiling. But like I said before, I'm a big believer in starting small, keeping costs down, and scaling when you actually hit the wall. And I officially hit that wall.
Why Upgrading the Firewall Alone Wasn't Enough
Here's the thing: upgrading the firewall alone wouldn't solve the problem. If I increased the capability of the firewall but the switch facilitating communication between the firewall and my internal network was still limited, I'd still be stuck. I'd have upgraded my firewall to be capable of so much more, and then my next bottleneck would be the switch.
This ended up becoming more than a firewall upgrade, it became a network architectural upgrade. The mindset shifted from "I need to upgrade my firewall" to "I need to upgrade the entire traffic flow."
Building a 10Gb Backbone
I decided I wanted my internal network backbone to support higher speeds to futureproof myself a bit. This included upgrading the firewall and the different components that support the traffic flow pipeline.
Here's an important thing to keep in mind: not every device in this environment is capable of using those speeds. A network is only as fast as the slowest link in the path.
While not every device will run at maximum capacity, building the core this way ensures that inter-VLAN traffic flows freely. Internal data transfers to things like backup storage, VMs, and NAS can move much more quickly, and the network stops being the limiting factor. The environment becomes futureproof, so to speak.
Now that you understand why I had to make these changes (because I find it frustrating when people make changes without explaining the reasoning), let's walk through what was added and how everything connects.
The New Firewall: Built for Growth
The main catalyst for this whole upgrade was the firewall. I decided to get something more powerful that would serve me now but also well into the future, so I wouldn't have to change my firewall as I upgrade and move things around.
I went with the Protectli VP6630. This thing is a beast for a homelab firewall. It's running an Intel Core i3 processor with 32GB of Crucial DDR5 RAM (2x16GB) and a 1TB Samsung 980 NVMe drive for storage. The real star of the show, though, is the network connectivity: two Intel 10GbE SFP+ ports and four Intel 2.5GbE copper ports. Really powerful. The reason I went bigger is because I'm setting up a whole bunch of monitoring and security features on the firewall, so I needed something capable of handling that without breaking a sweat.
Here's how I set it up: I use one of the 2.5GbE copper ports to connect directly to my ISP modem. It didn't make sense to use one of the SFP+ ports for this because even if I decide to get more bandwidth from my ISP, the 2.5GbE port can max that out. This gives me headroom for future bandwidth upgrades without needing to change anything. Then there's a DAC cable connected to the main switch using one of the Intel 10GbE SFP+ ports. We'll look at that in detail in a moment.
I wanted to use the SFP ports between the firewall and the switch. The reason? There's a lot of inter-VLAN communication happening, and I wanted something that would allow high-speed traffic between my firewall and switch with low latency. When there's heavy inter-VLAN communication, I don't want to run into any bottleneck.
The DAC cable provides that connection from the firewall to the switch, ensuring that traffic flows smoothly without adding unnecessary latency or heat (which transceivers can generate). This was the first major change, and I'm really happy with it because I'm already doing so much more on my firewall thanks to this device. It's very powerful and very capable.
The Main Switch: Layer 3 Capability for the Future
The next thing I had to change was the switch. I went with the UniFi Pro Max 24-port Layer 3 Switch (non-PoE version). Though I'm not currently using it for layer three routing, just for layer two purposes. My firewall is still doing all my routing, which gives me more flexibility and power.
The reason I chose this switch is because it offers dual SFP+ 10GbE ports for the connection between my firewall and switch, which is exactly what I needed for that high-speed backbone. But what I really love about this switch is the port segmentation: it has eight 2.5GbE ports and sixteen 1GbE ports. This allows me to separate my management traffic on the 1GbE ports while having devices capable of pushing more bandwidth connected to the 2.5GbE ports. It's a small thing, but it makes network organization and cable management so much cleaner.
The dual SFP+ 10GbE ports are perfect for building that pipeline I was after. One connects to my firewall via DAC cable for that low-latency, high-throughput connection. The other connects downstream to my high-speed switch, creating a proper 10GbE path through the network. This is connected to a patch panel capable of handling high-speed traffic using CAT6A cabling, which in turn connects to my other networking gear.
The Secondary Switches: Purpose-Built Connectivity
I'm using a transceiver connection (though I prefer DAC cables when possible because transceivers can get hot) to connect to my next switch, which is the UniFi Pro XG 8-Port 10GbE PoE++ switch. I'm using a transceiver temporarily until I receive shorter DAC cables that will allow me to connect things in a better manner.
This switch is where all my trunk ports connect. It has eight RJ45 10GbE PoE++ ports and two SFP+ 10GbE ports. I wanted all my trunk ports connected to a switch capable of handling 10GbE throughput so they can have that flow of traffic, even though not all of them are capable of pushing full 10GbE speeds themselves. The PoE++ capability is a huge bonus.
I also have a MikroTik CRS305 switch that basically allows me to extend my network if I need to. It has four SFP+ 10GbE ports and one 1Gbit copper ethernet port for management access. It's the smallest 10GbE switch I could find that would let me extend my network while maintaining high-speed connectivity. Ideally, in the future if I need to add another device, I can just run a DAC cable and use this to extend my network without getting more equipment.
Understanding the Real Limitations
One thing to keep in mind is: the speed of your network isn't just determined by your switch and firewall. It's also limited by the NIC in your device, the speed of your storage, and the weakest link in the path.
If your network can handle maximum speeds but your device only has a certain capacity NIC, then that link is limited to that capacity. If you're transferring data over a fast network but your drive can only write at a slower speed, then your storage becomes the bottleneck. The reality is that you can't just magically force hardware to get faster than it's designed to be.
Those are bottlenecks I'm perfectly fine living with right now. Understanding these limitations is crucial when designing your network because it helps you make informed decisions about where to invest and what actually matters for your use case.
Why Inter-VLAN Communication Matters
Before we wrap up, I want to talk about VLAN communication because this is one of those topics that's very important when designing a network, but I feel like it doesn't get discussed nearly enough.
When you create VLANs, you're creating separate networks. Anytime a device in one VLAN needs to talk to a device in another VLAN, that traffic has to be routed between VLANs. That's inter-VLAN communication. The device responsible for that routing is whatever's handling your layer three routing, which could be a layer three switch or, in my case, my firewall.
Since my firewall is doing my VLAN routing, anytime inter-VLAN traffic happens, the flow looks like this: the device sends traffic to my switch, the switch sends it to my firewall, my firewall makes the routing decision, sends that traffic back to my switch, and then my switch takes it and sends it to the correct device in the correct VLAN.
Think about this: in a network with multiple VLANs, lots of internal services talking to each other, IDS and IPS running security inspections on the firewall, and normal internet traffic also flowing through the same device, you quickly realize that the firewall is doing a lot of work.
Not only does it need to be powerful enough to handle all that processing, but the link between your switch and your firewall also needs to be fast enough to handle that constant back-and-forth traffic. If it's not, you'll start experiencing slowdowns in your inter-VLAN communication.
This is where a layer three switch comes into play. Both of the new switches I added are layer three capable. What that means is the switch can perform inter-VLAN routing locally instead of sending all that traffic up to the firewall. With layer three switch routing, the flow looks like this: the device sends information to the switch, the switch routes it to the correct VLAN, and then sends it back to the device in that VLAN. The traffic never has to go all the way up to the firewall and back down.
To be clear, the switch is still doing layer three routing here. It's just optimized for moving traffic quickly, not for doing deep security inspection like the firewall would. That's typically faster and more efficient because the switch is built specifically for moving traffic. It's not doing IDS, IPS, or deep packet inspection. It reduces the load on your firewall and reduces unnecessary traffic on the uplink between the switch and the firewall.
Making the Design Decision
This is where design decisions come into play. If the switch is handling your inter-VLAN routing, that traffic is not going through your firewall's deep inspection engine. You still have control. Layer three switches can do things like ACLs, VLAN rules, and basic traffic policy, but you lose things like IDS, IPS inspection, deep packet inspection, and advanced firewall-level filtering.
Here's the tradeoff: firewall routing equals deep inspection plus routing. Layer three switch routing equals fast routing plus basic control.To be honest, there's no single right answer here. Right now, I like things to be routed at the firewall level, and that's why I've designed my network to facilitate that. It all depends on how you want traffic to flow and how you want to structure your network.
One thing I did was get layer three switches, so if in the future I decide that things are getting too heavy on the uplink to my firewall and I want my switch to handle routing, I've futureproofed myself. I already have switches that can do that, so I can just make the switch with configuration changes. I don't need new hardware.
You have to decide how you want your traffic to flow and then structure your network that way. The reality is that a well-structured network isn't just about buying the fastest hardware. It's about understanding how traffic flows through your environment before you even start building services on top of it.
That's one of the reasons I decided to take a step back and redesign things. I wanted the flow set up the way I envisioned it, and I wanted to futureproof myself so that if something changes, I can go back in and make configuration adjustments to accommodate new traffic patterns.
VLAN routing plays a huge role in that. When you're designing your network, think about how you want traffic to flow, especially traffic flowing between your VLANs.
Final Thoughts
This upgrade took some time. I'm not going to lie. Really taking everything apart and putting it back together was a process. But as I said earlier, it was a necessary upgrade that needed to happen before I could move forward. I also wanted to make sure I properly documented the environment before moving on, which is why things took a bit longer. But now that this is sorted, it's back to real building. I have so many ideas, so many projects I want to tackle.
Thanks for coming along on this journey with me. Understanding why these changes were necessary and how they impact the overall environment is just as important as the changes themselves. A well-designed infrastructure is the foundation that lets you build amazing things on top of it.
Stay curious, keep learning, and don't be afraid to rebuild when you've outgrown what you have. Sometimes taking a step back to strengthen your foundation is the best way to move forward. Peace!
