I struggled to title this blog post because the purpose of this implementation for me was to modify my existing vSphere environment so that I could host VMs in their own segregated network. Really, you don’t need that requirement for this to be useful – this post also documents how to segment your network with VLANs while using a virtual pfSense firewall. I found it confusing enough that I thought it’d make for a good blog post regarding a VM of pfSense and VLANs.
I wanted to be able to spin up a small Linux VM and let a friend remote into it to test an application but I didn’t want him being able to run free on my network. There’s many ways to accomplish this, so don’t take my instructions below as the sole method. Because I run a virtual pfSense firewall, the networking portion of this is a little more confusing than if the firewall were physical.
Originally, I had this concept working by simply creating an additional vSwitch on my ESXi host. I then added a NIC to my pfSense VM and connected it to a port group on the newly created vSwitch. After creating VMs and putting them in this new port group, I was able to control access to and from this new interface in pfSense. However, if you read the Configuration Maximums documentation for vSphere 6.0 you’ll find that there is a 10 NIC limit per VM so this method will not scale well at all. So, while this would work, I could potentially run into issues with total NIC counts on the pfSense VM and, even worse, each time I added a NIC I had to reboot the firewall because FreeBSD does not support hot-add of NICs yet. I don’t suspect needing that many NICs/networks on my firewall, but it’s not best-practice if we’re working toward real-world concepts.
Working at a cloud hosting company, I am very familiar with creating a multi-tenant setup in vSphere. However, the configurations I am accustomed to always involve a physical firewall, router, or Layer 3 switch at the top of the infrastructure making it more straightforward for me to understand. Consider the following basic diagram:
The above diagram is pretty conventional – all trunk lines will carry the necessary VLANs into the hosts; just substitute your favorite hardware, maybe add a secondary firewall, maybe put them on their own switch stack, maybe not, you get the idea. From there, a port group will be created on the vSwitch with the associated uplinks and that port group will be tagged for the appropriate VLAN. The VLAN will traverse from the firewall all the way down to the VM within the host. Access to and from the VLAN will be controlled by the firewall as a sub-interface and every VM on that VLAN/port group will use the sub-interface as its gateway. But, what happens when you move pfSense inside the a host as a VM? Well, functionally, not much – but, conceptually it is a little confusing at first (or was to me). I’ll post a diagram I came up with to best describe what’s going on in my working setup:
As you can see, the diagram above is a bit busier. That’s mainly because I decided to include more detail in order to better explain the solution. Starting at the top, you’ll find my internet connection (Verizon Fios) lands on my Dell PowerConnect 5448. I have the Fios ONT box land on the switch in a small 3-port VLAN because I want the ability to fail the firewall over to another host if I have to – having the switch involved means I don’t have to have the connection from the ONT plugged directly into a single host. We need to create a VLAN and place a few ports on it because we don’t want the DHCP from Verizon Fios being available to any old device on the switch. Note that the WAN VLAN does not leave the Dell PowerConnect switch. You’ll see above that vmnic6 on my ESXi host connected to the switch as part of the small VLAN. The PowerConnect 5448 also has all of the VLANs created on it that I am planning to use. To start, I’ve created VLAN2, VLAN51, and VLAN200.
Let’s make it work!
The ESXi host connects to the switch over two physical NICs (vmnic0 and vmnic4). The ports that these two interfaces connect to are configured as trunk ports and have VLANs 2, 51, and 200 added to the trunk. Because the pfSense VM has a NIC on vSwitch0, which links to the PowerConnect switch via vmnic0/4, all you need to do is create port groups tagged with the VLANs created on the switch and then create a sub-interface in pfSense tagged for the same VLANs and you’ll be ready to go. Below are screenshots showing that I’ve labeled interfaces g3 and g4 on my PowerConnect switch for the associated vmnics along with the VLAN configuration :
It’s always a good idea to label your ports with descriptions so that you can configure them later. With Cisco switches you can enable CDP and then identify which vmnic goes to which port of your switch. With Dell you can enable LLDP but I don’t currently have it enabled. In order to identify the ports you can use the show bridge address-table command on a PowerConnect switch to show the MAC address table and correlate the MACs of your ESXi host to the ports on your switch. If you have two NICs in an active/active configuration you may need to set one NIC to “standby” or “unused” for the remaining MAC to show up. The MAC table will only show information once traffic is initiated and the switch learns the MAC.
There’s on “gotcha” that threw me for a second. Your pfSense is going to be straddling two port groups up front, right? If you’ve already deployed pfSense it should have a NIC on your default VM port group and your WAN port group, as shown below:
Please ignore Network Adapter 3 above as that was for testing and I haven’t rebooted the VM to remove it yet. That said, here’s where the visualization aspect gets tricky. If the pfSense inside interface is on the “LAN” port group, and that port group isn’t tagged for any VLANs (or if it is tagged for a single VLAN) then how will pfSense know about the other VLANs we want to create? Remember, we are trying to not add more NICs to pfSense, so how do we get the single “LAN” NIC to see more than just the default (VLAN1) VLAN? Glad you asked! See the screenshot below:
Remember earlier when we focused on how a physical setup would trunk the VLANs all the way down to the VMs? Well, we still need to do that – except we can’t make a “trunk” port going into a vSwitch in the conventional sense. Instead, we can tag the port group that a VM is a member of for “All (4095)” VLANs – in doing this, the pfSense will receive any packets upstream that are tagged for any VLAN. This step is crucial as it basically “trunks” all VLANs through to the pfSense VM “LAN” NIC. Something worth noting is that “All (4095)” VLANs include the default VLAN1/untagged traffic. I have a bunch of VMs already on the “LAN” portgroup when it did not have a VLAN ID set. Setting it the “All (4095)” did not disturb the VMs already there.
Once you’ve got your trunk interfaces configured all the way to your ESXi host(s) and have your pfSense inside interface port group setup for all VLANs, you’re ready to create your new port groups. It’s important to know where to create the port groups or this won’t work. If you have more than one vSwitch with uplinks to your physical switch, make sure you’re using the right vmnics/vSwitch for this. To create the port groups on vSwitch0, just open your vSphere Web Client, select your host, go to Manage, then the Networking tab, Virtual Switches, and click the symbol for “Add Host Networking” – when the window opens, select the bottom option for creating “Virtual Machine Port Group for a Standard Switch“:
On the next window you’ll simply select the existing standard vSwitch (vSwitch0 in this case):
Then, when prompted to create the new Port Group, just give it a name that makes sense (you’ll be selecting this “network” when you configure VMs you wish to associate) and give it the proper VLAN ID that you’ve already created:
Once you’ve named and set the VLAN ID for the new port group, click next and Finish and you’ll have your new port group waiting for you. Sticking with my VLANs of 51 and 200, see the screenshot below:
Above you’ll see that I created a “Conway DR” and “HPS Network” port group each tagged for VLAN 51 and 200, respectively.
The next step is to tell pfSense what the heck to do with traffic that is tagged for these VLANs. To do that, we’ll log into our pfSense GUI. Once logged in, we’ll hover over Interfaces and select (assign):
If you refer back to the second diagram I posted in this entry you’ll note that the diagram shows pfSense with vmx0 and vmx1 interfaces. Specifically, it shows that pfSense has vmx1 on the “LAN” port group. I made it easy in pfSense by labeling vmx1 as “LAN” but you may not have done that – either way it’s important to know which interface corresponds to the port group that was set for the VLAN ID of “All (4095)“. Once you’ve identified that, click the VLANs tab:
You can see above that I already have my two VLANs created and they are associated with the vmx1 interface. In order to create that, you simply click the “+” symbol at the bottom of the window which will open a window where you specify the interface to associate the VLAN with (important!), the VLAN ID, and a description (note, VLAN 999 does not exist on my switch this is just a demonstration):
Once you click save, you’re done. You’ll see the VLAN is created but we’re not done yet. We have to go back to the Interface Assignments tab so that we can create a sub-interface from the newly created/associated VLAN:
As above, click the drop-down, select your new VLAN you created, and hit the “+” symbol on the right. The result will be a new interface with the VLAN associated with it, called Optx:
Once you see the new interface, simply click on it (it’s a URL), and enable the interface, set the name of the interface, and create a static IPv4 address/subnet for this interface to use:
That’s it! Now, you have a port group associated with VLAN 999, a sub-interface in pfSense associated with VLAN 999, and when you hover over Firewall and then select Rules you will see the interface as a tab and you can create rules! You can also setup a DHCP server for this new interface if you choose.
From here, you’d simply create a new VM and when you go to configure the NIC you will see your newly created port group in the drop down – simply select the port group you want, IP the VM (or let DHCP do it), and you’re set. Remember that the gateway for the VM will be the sub-interface IP we set in the screenshot above (10.0.99.1 in the example).
Going forward, you could create a new port group/sub-interface for each “client” or network you want to isolate (remembering to trunk the VLAN from the switch all the way into the pfSense VM) and create rules as needed. Or, you could create a “client network” and simply drop whatever VMs are not part of your personal/management network into that port group and IP them accordingly. There’s potential for the VMs to see each other this way, but the access rules can be configured so that they won’t be able to see anything outside of their own VLAN. If you were a large cloud host you could create these many “client networks” with 24-bit masks so as to keep broadcast domains small, for example.
The whole point of doing it this way is not only to make it easier to manage from a vSphere perspective, but so to also not run up against the NIC limitation at the VM level. And, as mentioned before, adding/removing NICs from a FreeBSD VM will require the VM be rebooted for the changes to take effect. The “sub-interface” method is much more production-friendly and most similarly matches the first diagram in this blog entry dealing with conventional physical network equipment.
I hope you found this entry useful – let me know if anything needs clarification or is incorrect! One last thing I wanted to add – I did this configuration 100% remote. Everything from identifying the ports on the switch to making them trunks, to adding sub-interfaces is all non-invasive (if you do it right). If you think ahead, you can move everything around to a properly secured/zoned VLAN setup without disturbing any network connectivity already in place!