Three NICs, but Proxmox Disagreed

Three months ago I tried to convince Proxmox that my homelab had four NICs instead of three.¹ It did not believe me. Here’s what I learned about Linux bridge interfaces, the price of stubbornness, and why I keep doing this to myself.

The setup was straightforward in the way that misconfigurations always are at the start. I had a [REPLACE — which node: e.g., “fresh Proxmox install on one of the EliteDesks”], a [REPLACE — what the actual physical layout was: number of physical ports, USB-NIC dongles, VLAN trunks, whatever], and a plan: one management interface, one general LAN bridge, one VLAN trunk to the OPNsense router, and a fourth interface dedicated to [REPLACE — what the fourth NIC was supposed to do]. The plan had four lines on a piece of paper. The machine had three lines on its motherboard. I did not, at the time, find this distinction load-bearing.

The first symptom was that ip link showed three physical interfaces, and Proxmox’s web UI showed three physical interfaces, and dmesg showed three NICs being initialized. I assumed I was missing a driver. I was not missing a driver. I was missing a NIC.

[REPLACE — the middle of the story: what I tried before I understood the problem. Examples worth choosing from: spent N hours convinced the kernel was hiding a port; created a bridge with a non-existent slave and watched Proxmox fail to apply the config in a way that did not surface the actual error; tried to bond two ports together; checked the BIOS for a disabled onboard NIC; emailed someone who knew better.]

The lesson, in the order I learned it, was this: a Linux bridge is a piece of software, but a Linux bridge with no physical slave is just a piece of software pointing at nothing. Proxmox is happy to let you create one; OPNsense is happy to take the trunk; the packets just stop in the bridge and look at you. The fix was, of course, to use the three NICs I had, the way I had them, and accept that the fourth interface was going to live as a tagged VLAN over an existing trunk rather than as a fifth ethernet cable on the back of the machine.

I keep doing this to myself because it is the cheapest form of education I know of. The bridge interfaces, the VLAN tagging, the bonded LACP I almost reached for and didn’t need — none of these are concepts I had to learn for work, and all of them are concepts I have since used at work, with the steady confidence of someone who has already made the embarrassing mistake on their own time.

If you’re new to Proxmox networking and reading this because the search engine has correctly guessed your problem: count the physical ports before you draw the network diagram. Look at the back of the machine. Trust the kernel before you trust the plan.

Footnotes

1. Reader, it had three. ↩