Multi-tenant/VLANs behind a virtualized pfSense firewall in ESXi

I struggled to title this blog post because the purpose of this implementation for me was to modify my existing vSphere environment so that I could host VMs in their own segregated network.  Really, you don’t need that requirement for this to be useful – this post also documents how to segment your network with VLANs while using a virtual pfSense firewall.  I found it confusing enough that I thought it’d make for a good blog post regarding a VM of pfSense and VLANs.

Why bother?

I wanted to be able to spin up a small Linux VM and let a friend remote into it to test an application but I didn’t want him being able to run free on my network.  There’s many ways to accomplish this, so don’t take my instructions below as the sole method.  Because I run a virtual pfSense firewall, the networking portion of this is a little more confusing than if the firewall were physical.

Originally, I had this concept working by simply creating an additional vSwitch on my ESXi host.  I then added a NIC to my pfSense VM and connected it to a port group on the newly created vSwitch.  After creating VMs and putting them in this new port group, I was able to control access to and from this new interface in pfSense.  However, if you read the Configuration Maximums documentation for vSphere 6.0 you’ll find that there is a 10 NIC limit per VM so this method will not scale well at all.  So, while this would work, I could potentially run into issues with total NIC counts on the pfSense VM and, even worse, each time I added a NIC I had to reboot the firewall because FreeBSD does not support hot-add of NICs yet.  I don’t suspect needing that many NICs/networks on my firewall, but it’s not best-practice if we’re working toward real-world concepts.

Working at a cloud hosting company, I am very familiar with creating a multi-tenant setup in vSphere.  However, the configurations I am accustomed to always involve a physical firewall, router, or Layer 3 switch at the top of the infrastructure making it more straightforward for me to understand.  Consider the following basic diagram:

Cloud Example

The above diagram is pretty conventional – all trunk lines will carry the necessary VLANs into the hosts; just substitute your favorite hardware, maybe add a secondary firewall, maybe put them on their own switch stack, maybe not, you get the idea.  From there, a port group will be created on the vSwitch with the associated uplinks and that port group will be tagged for the appropriate VLAN.  The VLAN will traverse from the firewall all the way down to the VM within the host.  Access to and from the VLAN will be controlled by the firewall as a sub-interface and every VM on that VLAN/port group will use the sub-interface as its gateway.  But, what happens when you move pfSense inside the a host as a VM?  Well, functionally, not much – but, conceptually it is a little confusing at first (or was to me).  I’ll post a diagram I came up with to best describe what’s going on in my working setup:

Lab Example

As you can see, the diagram above is a bit busier.  That’s mainly because I decided to include more detail in order to better explain the solution.  Starting at the top, you’ll find my internet connection (Verizon Fios) lands on my Dell PowerConnect 5448.  I have the Fios ONT box land on the switch in a small 3-port VLAN because I want the ability to fail the firewall over to another host if I have to – having the switch involved means I don’t have to have the connection from the ONT plugged directly into a single host.  We need to create a VLAN and place a few ports on it because we don’t want the DHCP from Verizon Fios being available to any old device on the switch.  Note that the WAN VLAN does not leave the Dell PowerConnect switch.  You’ll see above that vmnic6 on my ESXi host connected to the switch as part of the small VLAN.  The PowerConnect 5448 also has all of the VLANs created on it that I am planning to use.  To start, I’ve created  VLAN2, VLAN51, and VLAN200.

Let’s make it work!

The ESXi host connects to the switch over two physical NICs (vmnic0 and vmnic4).  The ports that these two interfaces connect to are configured as trunk ports and have VLANs 2,  51,  and 200 added to the trunk.  Because the pfSense VM has a NIC on vSwitch0, which links to the PowerConnect switch via vmnic0/4, all you need to do is create port groups tagged with the VLANs created on the switch and then create a sub-interface in pfSense tagged for the same VLANs and you’ll be ready to go.  Below are screenshots showing that I’ve labeled interfaces g3 and g4 on my PowerConnect switch for the associated vmnics along with the VLAN configuration :

KRC Switch 1

KRC Switch 2

It’s always a good idea to label your ports with descriptions so that you can configure them later.  With Cisco switches you can enable CDP and then identify which vmnic goes to which port of your switch.  With Dell you can enable LLDP but I don’t currently have it enabled.  In order to identify the ports you can use the show bridge address-table command on a PowerConnect switch to show the MAC address table and correlate the MACs of your ESXi host to the ports on your switch.  If you have two NICs in an active/active configuration you may need to set one NIC to “standby” or “unused” for the remaining MAC to show up.  The MAC table will only show information once traffic is initiated and the switch learns the MAC.

There’s on “gotcha” that threw me for a second.  Your pfSense is going to be straddling two port groups up front, right?  If you’ve already deployed pfSense it should have a NIC on your default VM port group and your WAN port group, as shown below:

pfSense NIC settings

pfSense NIC settings

Please ignore Network Adapter 3 above as that was for testing and I haven’t rebooted the VM to remove it yet.  That said, here’s where the visualization aspect gets tricky.  If the pfSense inside interface is on the “LAN” port group, and that port group isn’t tagged for any VLANs (or if it is tagged for a single VLAN) then how will pfSense know about the other VLANs we want to create?  Remember, we are trying to not add more NICs to pfSense, so how do we get the single “LAN” NIC to see more than just the default (VLAN1) VLAN?  Glad you asked!  See the screenshot below:

vSwitch port group set up as a "trunk"

vSwitch port group set up as a “trunk”

Remember earlier when we focused on how a physical setup would trunk the VLANs all the way down to the VMs?  Well, we still need to do that – except we can’t make a “trunk” port going into a vSwitch in the conventional sense.  Instead, we can tag the port group that a VM is a member of for “All (4095)” VLANs – in doing this, the pfSense will receive any packets upstream that are tagged for any VLAN.  This step is crucial as it basically “trunks” all VLANs through to the pfSense VM “LAN” NIC.  Something worth noting is that “All (4095)” VLANs include the default VLAN1/untagged traffic.  I have a bunch of VMs already on the “LAN” portgroup when it did not have a VLAN ID set.  Setting it the “All (4095)” did not disturb the VMs already there.

Once you’ve got your trunk interfaces configured all the way to your ESXi host(s) and have your pfSense inside interface port group setup for all VLANs, you’re ready to create your new port groups.  It’s important to know where to create the port groups or this won’t work.  If you have more than one vSwitch with uplinks to your physical switch, make sure you’re using the right vmnics/vSwitch for this.  To create the port groups on vSwitch0, just open your vSphere Web Client, select your host, go to Manage, then the Networking tab, Virtual Switches, and click the symbol for “Add Host Networking” – when the window opens, select the bottom option for creating “Virtual Machine Port Group for a Standard Switch“:

Adding Port Group to Standard Switch

Adding Port Group to Standard Switch

On the next window you’ll simply select the existing standard vSwitch (vSwitch0 in this case):

Select the existing vSwitch

Select the existing vSwitch

Then, when prompted to create the new Port Group, just give it a name that makes sense (you’ll be selecting this “network” when you configure VMs you wish to associate) and give it the proper VLAN ID that you’ve already created:

Creating new Port Group with VLAN

Creating new Port Group with VLAN

Once you’ve named and set the VLAN ID for the new port group, click next and Finish and you’ll have your new port group waiting for you.  Sticking with my VLANs of 51 and 200, see the screenshot below:

Host Port Groups with VLANs

Host Port Groups with VLANs

Above you’ll see that I created a “Conway DR” and “HPS Network” port group each tagged for VLAN 51 and 200, respectively.

The next step is to tell pfSense what the heck to do with traffic that is tagged for these VLANs.   To do that, we’ll log into our pfSense GUI.  Once logged in, we’ll hover over Interfaces and select (assign):

pfSense interfaces

pfSense interfaces

If you refer back to the second diagram I posted in this entry you’ll note that the diagram shows pfSense with vmx0 and vmx1 interfaces.  Specifically, it shows that pfSense has vmx1 on the “LAN” port group.  I made it easy in pfSense by labeling vmx1 as “LAN” but you may not have done that – either way it’s important to know which interface corresponds to the port group that was set for the VLAN ID of “All (4095)“.  Once you’ve identified that, click the VLANs tab:

pfSense Interface VLANs

pfSense Interface VLANs

You can see above that I already have my two VLANs created and they are associated with the vmx1 interface.  In order to create that, you simply click the “+” symbol at the bottom of the window which will open a window where you specify the interface to associate the VLAN with (important!), the VLAN ID, and a description (note, VLAN 999 does not exist on my switch this is just a demonstration):

Create VLAN within pfSense

Create VLAN within pfSense

Once you click save, you’re done.  You’ll see the VLAN is created but we’re not done yet.  We have to go back to the Interface Assignments tab so that we can create a sub-interface from the newly created/associated VLAN:

Create New Sub-Interface

Create New Sub-Interface

As above, click the drop-down, select your new VLAN you created,  and hit the “+” symbol on the right.  The result will be a new interface with the VLAN associated with it, called Optx:

Sub-Interface Added

Sub-Interface Added

Once you see the new interface, simply click on it (it’s a URL), and enable the interface, set the name of the interface, and create a static IPv4 address/subnet for this interface to use:

New interface enabled

New interface enabled

That’s it!  Now, you have a port group associated with VLAN 999, a sub-interface in pfSense associated with VLAN 999, and when you hover over Firewall and then select Rules you will see the interface as a tab and you can create rules!  You can also setup a DHCP server for this new interface if you choose.

From here,  you’d simply create a new VM and when you go to configure the NIC you will see your newly created port group in the drop down – simply select the port group you want, IP the VM (or let DHCP do it), and you’re set.  Remember that the gateway for the VM will be the sub-interface IP we set in the screenshot above (10.0.99.1 in the example).

Going forward, you could create a new port group/sub-interface for each “client” or network you want to isolate (remembering to trunk the VLAN  from the switch all the way into the pfSense VM) and create rules as needed.  Or, you could create a “client network” and simply drop whatever VMs are not part of your personal/management network into that port group and IP them accordingly.  There’s potential for the VMs to see each other this way, but the access rules can be configured so that they won’t be able to see anything outside of their own VLAN.  If you were a large cloud host you could create these many “client networks” with 24-bit masks so as to keep broadcast domains small, for example.

The whole point of doing it this way is not only to make it easier to manage from a vSphere perspective, but so to also not run up against the NIC limitation at the VM level.  And, as mentioned before, adding/removing NICs from a FreeBSD VM will require the VM be rebooted for the changes to take effect.  The “sub-interface” method is much more production-friendly and most similarly matches the first diagram in this blog entry dealing with conventional physical network equipment.

I hope you found this entry useful – let me know if anything needs clarification or is incorrect!  One last thing I wanted to add – I did this configuration 100% remote.  Everything from identifying the ports on the switch to making them trunks, to adding sub-interfaces is all non-invasive (if you do it right).  If you think ahead, you can move everything around to a properly secured/zoned VLAN setup without disturbing any network connectivity already in place!

Author: Jon

Share This Post On

13 Comments

  1. Great article are you still using PF

    Post a Reply
    • Thanks! Yes, still using pfSense. It’s been through many upgrades but this site is still served behind one!

      Post a Reply
      • I am a huge PF fan however I am in the process of moving my lab network over to a Edge Router Lite. Still having seconds thoughts since PF just works.

        Post a Reply
        • I’ve researched the ERL when making recommendations to friends and colleagues. The problem I have with it is that it’s really just a router. It can do some but not all of what pfSense does. Because I run my pfSense as a VM on VMware hosts with 10 Gbps interfaces, I get very fast routing/firewalling that I can’t afford to recreate on hardware.

          Post a Reply
  2. Excellent write-up. I’m considering a similar setup but perhaps eliminating vSwitch1 and moving vmnic06 to vSwitch0, essentially creating a 3-port trunk. I would then define a new VLAN for WAN traffic, say WLAN VLAN ID:300, and add a corresponding port group to vSwitch0 in ESXi. The pfSense VM would next be added to this port group to receive WAN traffic on vmx0 (i.e., with ESXi untagging WAN traffic as it enters the VM through this interface and tagging traffic as it leaves). The LAN (VLAN ID:All) port group would also be connected to the pfSense VM as vmx1 and then sub-interfaced in pfSense just as you have done.

    I suppose this is really just a trade-off between physically separating WAN traffic on a dedicated NIC (i.e., more security but with potentially a single physical point of failure) versus combining WAN traffic on the trunk (i.e., theoretically less security but with better physical redundancy by pooling the NICs).

    Thoughts?

    Post a Reply
    • Hey Don – the only problem with a 3-port uplinked vSwitch is that it is expected that any portgroup (and thus VLAN) defined on the vSwitch is available to all uplinks unless you override the teaming/failover policy for the portgroup. That said, there’s nothing keeping you from putting VLAN 300 on the uplinks as part of a trunk, and landing your WAN on an access port on the physical switch in that same VLAN. Not sure if you get that, but basically, you don’t have to have a dedicated NIC for uplink to WAN, you can just trunk it as you mentioned. Personally, it’s all 802.1q and secure – there haven’t been instance of VLAN hopping outside of very specific situations so I think you’d be fine either way. Personally, I prefer to not mess with the teaming/failover overrides if possible! Good thinking!

      Post a Reply
  3. Nice write up. I came across your article while trying to figure out how to set my stuff up. Would appreciate any feedback as I am stuck ATM. I have 2 dell r610 running some internal VMs for my house and also running some external web services. I am trying to put together a more extensive lab inside this environment. I currently have 6 esxi 6 nested hosts that will serve 3 separate vCenters so I can test NSX. my physical network is a flat network with a 3560g cisco switch. I know I can set vlans in it but would like to avoid it if at all possible. what I want to do is setup PFsense as a VM internal router to my nested esxi hosts. I have created it with 5 nics, 1 is a san nic, 1 is a mgmt nic, seperate network from my internal home network, 1 will be a vmotion network and the last 2 will be used for the NSX network across all hosts that will manage the VXLANs. The issue I am having is when I go in and set the mgmt lan in pfsense to vlan 10 and allow traffic, then set my mgmt nic on my esxi host to vlan 10 and set the port group on the physical esxi host to vlan 10 and then set my windows server nic to vlan 10 none of them can communicate. DO I really need to set the port on the physical cisco 3560g to vlan 10 for all of that to work and can the pfsense control all the vlan tagging only for nested environment. Thanks for any help anyone can add

    Post a Reply
    • Bugatti, any luck … i’m also on the same track to achieve it!

      Post a Reply
    • So the problem you’re having is that your switch needs to support this! The way it works is that your switch is still used even though pfSense is a VM with virtual interfaces. If you set your switch ports to trunk, all all vlans, native vlan 1, this will work. Also remember, you can’t double tag. So when you’re creating a port group setup for VLAN 10 then everything in that port groups is being tagged for VLAN 10. So don’t tag within the Windows guest. If everything is VLAN 10 then the switch port can be tagged for VLAN 10 and everything else untagged. A diagram would be helpful – feel free to post a link. Sorry it took me a while to reply. Your best bet to keep things flexible is to trunk.

      Post a Reply
  4. I’m thinking about a similar setup but with the particularity that each VLAN will host the exact same VMs and IP settings.
    The goal is to have an identical development environment for our developers that we could clone when a new person starts. We will then implement PAT on the pfsense to give access to each developer to the VMs. Developer 1 will connect using port 3391, dev2 port 3392 etc…
    I haven’t tried yet but do you foresee some issues in having that sort of setup where the IP settings are always the same but we isolate the traffic in using VLANs?

    Post a Reply
  5. Great read! I’m doing a similar setup!

    Post a Reply

Trackbacks/Pingbacks

  1. In Other BSDs for 2015/11/28 – DragonFly BSD Digest - […] Multi-tenant/VLANs behind a virtualized pfSense firewall in ESXi.  (via) […]

Submit a Comment

Your email address will not be published. Required fields are marked *

Share This
%d bloggers like this: