Hi all – just thought I’d share a recent pick up. I’ve been eyeing up a new switch for the “core” of my lab. I am currently using a Dell PowerConnect 5548 because it has 2 SFP+ ports, however, I have a workstation that’s 10GbE along with two ESXi hosts and so I have a “switchless” 10GbE network between hosts with my primary ESXi host cabled to the 5548 over SFP+ along with my workstation. So, in order to get proper 10GbE connectivity between the 3 devices, I needed something with more SFP+ ports.
I had been looking at the Arista 7124S (and still am) but the auction on eBay ended when I was traveling and so I missed out. Also, I still have 1GbE devices (APs, other desktops in the house, etc.) and since the Arista 7124S is all SFP+, I’d need to pick up SFP+ to copper transceivers. I would need about 8-10 transceivers and they’re around $16-20 each – you can see how that isn’t economical.
While chatting on one of my favorite Discord groups (for /r/homelab) a couple users mentioned the Nexus 3k line being affordable secondhand. I found a Nexus 3k C3048TP-1GE and made an offer and the seller accepted. A week later I had the unit in hand!
You’ll notice that there is a box from fs.com on top – that’s because I went ahead and ordered matching SFP+ transceivers specifically to work with “Cisco” products. I have a bunch of Finisar modules but I have heard that Nexus do not like to play well with non-Cisco stuff. The Meraki line doesn’t seem to care, but I want to avoid a headache when I go to swap this switch in. I also went ahead and ordered appropriate length OM3 patch cables. FS.com was a pleasure to deal with and even reach out to me to tell me that my order didn’t process properly and they packaged everything really well:
Update 5/16/18 – FS.com has really come through. I am finally getting around to deploying the Nexus 3k in the lab and had issues getting the SFP+ links to come up. FS.com reached out to their technical engineers and had me to a loop-back test, etc. confirming their SFP+ modules were working properly which then pointed to the “other side” – my ESXi hosts. Turns out the hosts had Finisar SFP+ modules in the Mellanox ConnectX2 cards that were 1310nm (Single Mode?) modules while the Nexus 3k had FS.com 850nm Multi-Mode. Swapping the Finisar modules in the Mellanox cards fixed the issue and thanks to FS.com’s help we got this figured out!
The good, the bad, the ugly
We run the Nexus-series switches at work (the 9K) which work really well. We use them with the FEX modules in Dell M1000E Blade Chassis along with rackmount TOR 9K/FEX when needed. So, I am somewhat familiar with the tribulations of Cisco Nexus. Like most people, when I get a “new” piece of hardware, the first thing I want to do is update the firmware and BIOS and make sure everything is up to date and current. Well, this gets a little hairy with the Nexus units.
The Good: It’s a Cisco. It runs NX-OS. The Cisco Nexus 3k is really flexible and obviously supports all of the advanced switching (and some routing) features that any data center top-of-rack switch would need. The hardware is generally very reliable and obviously Cisco (and the Nexus line) are heavily used throughout the industry. This specific switch features 48 1GbE copper ports plus 4 SFP+ 10GbE ports. I believe the end result is 52 usable ports unlike some other switches where the “uplink” 10GbE ports eat into the 1GbE ports should you use them. It’s not offensively loud, but I am going to see what I can do to make it even quieter.
The Bad: It’s a Cisco. It runs NX-OS. This means to get firmware you need support and Cisco support doesn’t come cheap. You may be able to find Nexus firmware if you Google for various search phrases as a ton of schools and businesses have public indexed websites with repositories of all sorts of Cisco firmware. Cisco has decent support forums but a lot of forum posts either don’t end with a solution or end with “I opened a TAC incident and Cisco was able to resolve the issue.” But, it’s a Cisco, so you can surely find how-to’s, books, guides, videos, etc. It runs NX-OS, which is the Nexus OS… it’s a fair amount different than the OS that runs on my Cisco Catalyst devices (IOS). It’s similar enough, but fairly different, too.
The Ugly: Cisco upgrade paths can be convoluted. There are certain situations where IOS or ASA firmware needs to follow a very specific upgrade path in order for a configuration to be preserved. The Nexus 3K (and other models, check!) is worse than that – it is plagued with what Cisco has deemed a “Severity: 1 Catastrophic” issue. The short of it is that the switch can become rendered useless if you try and upgrade to NX-OS 6.0(2)U6(7) from anything earlier than 6.0(2)U6(2a). You might say, “OK, big deal” – the problem is the release notes for various firmware upgrades may or may not have the same versions referenced because Cisco releases versions doesn’t go back and update the notes to keep them relevant. For instance, the bug I linked above applies to any version before NX-OS 6.0(2)U6(2). However, the only downloadable version is 6.0(2)U6(2a)… there is no “(2)”. There’s another instance in the NX-OS Guidelines that say “Upgrade to NX-OS 6.0(2)U6(3) before upgrading to… and that will destroy your switch if you do not pay attention. Obviously in a production situation you would put a TAC ticket in and get clarification but alas we have no support on eBay gear.
Do not be confused – failing to upgrade the switch in the exact path needed will result in a switch that is literally unusable. It’s an RMA situation – there is no recovery method. You can read people complain about it here and here, for instance.
To further complicate matters, the likelihood of you receiving a N3K-C3048TP-1GE that’s already running NX-OS 6.x is not very high because most data centers don’t upgrade firmware unless facing a specific bug that is fixed in a later version (which might make you decide that you don’t need to upgrade the firmware, either). If you read the Upgrade Guidelines for N3K to 6.x, you’ll see that it states:
For the N3K-C3048TP-1GE-SUP platform, if you are using software versions older than Cisco NX-OS Release 5.0(3)U5(1), upgrade to Cisco NX-OS Release 5.0(3)U5(1) first, then upgrade to Cisco NX-OS Release 6.0(2)U6(2a), and finally upgrade to 6.0(2)U6(7) or a latest release.
Yet, the Upgrade Guidelines for N3K to 7.x, you’ll find:
Note If you have a release prior to Release 7.0(3)I2(1), upgrade to Cisco Nexus 3000 Release 6.0.2.U6(3) first. Use the install all nxos bootflash:<image_name> bios command in Release 6.0.2.U6(3) to upgrade the BIOS version to version 3.x.x. Next, fast reload from Release 6.0.2.U6(3) to Release 7.0(3)I2(1) and later releases. On fast reload to Release 7.0(3)I2(1), the new BIOS is effective.
BE CAREFUL: The Upgrade Guidelines for N3K to 7.x quoted above just said “upgrade to 6.0(2)U6(3)… but if you didn’t read the earlier part of this article, you wouldn’t know it, but that will brick your switch! You have to go to NX-OS 6.0(2)U6(2a) first, then 6.0(2)U6(3) or later. This is what I mean by the release notes not really meshing together well!
Updating this thing
Once you decide that you want to risk rendering your newly acquired Nexus 3K useless you’ll have to pony up and get started. You will also need a Cisco Console Cable like this in order to reliably do the upgrade – you will want to be able to see that the switch is loading the bootloader and booting – you can’t do that over SSH alone. For NX-OS 5.x and 6.x, there is a kickstart image and a system image. In NX-OS 7.x they changed to a single image (nxos). The upgrade path goes like this:
- kickstart: version 5.0(3)U5(1) system: version 5.0(3)U5(1)
- kickstart: version 6.0(2)U6(2a) system: version 6.0(2)U6(2a)
- kickstart: version 6.0(2)U6(7) system: version 6.0(2)U6(7)
- NXOS: version 7.0(3)I4(7)
Note: As of this post, NX-OS 7.0(3)I4(7) is the Cisco “Suggested Released” for the N3K 3048.
I won’t go into all of the details of how to flash the device because it’s in the upgrade guidelines, but the basic idea is to get the kickstart and system image files over to the bootflash on the N3K. I used SCP but do understand that you cannot SCP a file to the N3K, you have to pull from the SCP server while on the N3K. The command for that might look like:
KCloud-Nex3k# copy scp://email@example.com/cisco/n3000-uk9-kickstart.6.0.2.U6.2a.bin bootflash:n3000-uk9-kickstart.6.0.2.U6.2a.bin
Obviously substitute the correct image files as needed.
The actual installation process looks like this:
KCloud-Nex3k# install all kickstart n3000-uk9-kickstart.6.0.2.U6.2a.bin system n3000-uk18.104.22.168.U6.2a.bin Installer is forced disruptive Compatibility check is done: Module bootable Impact Install-type Reason ------ -------- -------------- ------------ ------ 1 yes disruptive reset ISSU not supported Images will be upgraded according to following table: Module Image Running-Version New-Version Upg-Required ------ ---------- ---------------------- ---------------------- ------------ 1 system 5.0(3)U5(1) 6.0(2)U6(2a) yes 1 kickstart 5.0(3)U5(1) 6.0(2)U6(2a) yes 1 bios v1.2.0(08/25/2011) v1.2.0(08/25/2011) no 1 power-seq v4.4 v4.4 no ISSU is not supported on this platform!! Switch will be reloaded for disruptive upgrade. Do you want to continue with the installation (y/n)? [n]
You can see that in the output above, I upgraded from NXOS 5.0(3)U5(1) to 6.0(2)U6(2a). When I received my switch it was running 5.0(3)U3(2b):
Images will be upgraded according to following table: Module Image Running-Version New-Version Upg-Required ------ ---------- ---------------------- ---------------------- ------------ 1 system 5.0(3)U3(2b) 5.0(3)U5(1) yes 1 kickstart 5.0(3)U3(2b) 5.0(3)U5(1) yes 1 bios v1.2.0(08/25/2011) v1.2.0(08/25/2011) no 1 power-seq v4.4 v4.4 no
And so on.
Today, as it sits, my N3K 3048 is sitting like this:
KCloud-Nex3k# show ver Software BIOS: version 4.0.0 NXOS: version 7.0(3)I4(7) BIOS compile time: 12/05/2016 NXOS image file is: bootflash:///nxos.7.0.3.I4.7.bin NXOS compile time: 6/28/2017 14:00:00 [06/28/2017 21:53:29] Hardware cisco Nexus 3048 Chassis Intel(R) Celeron(R) CPU P4505 @ 1.87GHz with 3903332 kB of memory. Processor Board ID FOC17153Y5J Device name: KCloud-Nex3k bootflash: 1596672 kB usb1: 0 kB (expansion flash) Kernel uptime is 0 day(s), 0 hour(s), 15 minute(s), 14 second(s) Last reset Reason: Unknown System version: 7.0(3)I4(7) Service: plugin Core Plugin, Ethernet Plugin Active Package(s): KCloud-Nex3k#
And that’s about it! I have not pulled the Dell PowerConnect 5448 out yet because I am going to be moving my lab from the second story of my house to the basement shortly and don’t want to have too many moving pieces at once.
I’ve ordered port-side-exhaust PSUs and am going to see if I can simply flip the fans around or find more details about the port-side-exhaust fan module (without spending $179 on eBay!) so that I can mount this Nexus 3K in the rear of my rack with proper airflow (pro tip: burgundy colored PSU and fan module mean the exhaust is at the PSU side of the switch, blue colored PSU and fan module means the exhaust is out the switch port side).
Please do not take this article as an absolute “you must follow these instructions” to upgrade your switch. It’s a guide. Please read the release notes. I am not responsible if you brick your switch!
Thanks for reading hope you found this one useful!