Prelude (if you just want the “how-to” scroll down)
Boy oh boy. As my lab grows and grows I was receiving “Fully-automated DRS-enabled cluster has CPU contention” messages from vROPS every night and occasionally during the day. Of course for a lab it doesn’t matter much, but I do have some workloads that I care about behind NSX-T which can be touchy with contention and can yield weird results. I am running a Dell R620 and a Supermicro 24-bay with a X9DRi-LN4F+ motherboard. I was using dual E5-2680’s in each system which provided 8 cores and 16 threads per socket. A quick browse on eBay showed that I could pick up E5-2697v2’s (12 cores and 24 threads per socket) for not a ton of money and stretch the life on this cluster. I would just cut over to Dell R630-era hardware but I would need to buy new RAM as I am currently using 384GB of PC3L-12800R in each of my two hosts. Matching that RAM capacity in a DDR4 system would cost about $1,200 – $1,600+ in RAM alone. A bummer, because Intel E5-2697v4 CPUs are only a few bucks more than v2’s but come with 16 cores and 32 threads per socket while using a couple less watts… but it would require significant investment to jump. I decided to stick with the E5-2697v2 idea.
I convinced myself this will be an easy upgrade and acquired four E5-2697v2’s – two for the R620 and two for the X9DRi system. I did worry slightly about the fact that these are 135W processors and in a past life I ran into an issue using Dell M620 (blade) systems whereas there was a 135W version of the heatsink, but as it turns out the heatsinks I had in my R620 were the same that were paired with other 135W configurations and my Supermicro system is utilizing Noctua NH-U9DX i4 coolers which are more than capable. Great. Simple upgrade.
I threw the first pair of E5-2697v2’s into the R620 which was already running the latest iDRAC/LCC/BIOS so nothing to worry about there. The R620 booted fine right into ESXi and recognized the new resources. When placing the E5-2697v2’s in the Supermicro system it turned on but would not POST. I did make sure that the IPMI/BMC firmware and BIOS were the latest. There was no output on the monitor/IPMI at all. No beeps, either. Weird. I spent time pulling PCIe cards (each system has three BCM57810s NICs). I even considered the possibility that the PSU installed was not matching the CPU configuration or something odd and put in my big noisy 1200W PSUs to no avail. It wasn’t until I started doing some research involving keywords like Supermicro Ivy Bridge that I realized I may have an issue.
Apparently the X9DRi-LN4F+ boards had a couple revisions, mainly the 1.10 and 1.20a. However, Supermicro says 1.10 will support Ivy Bridge CPUs and even shows that revision board in diagrams and manuals and such. It seems, though, that there are revisions within the 1.10 build that have a specific ECO (Engineering Change Order?) applied to support the newer CPUs. If you luck out and got a “late build” 1.10 it’ll work, though maybe not, but all 1.20a will work. Well mine has to be a 1.20a, as I have another identical system (36-bay SC847 with an X9DRi with v2’s…) running fine. Right?
🙁 Sigh… I have a version 1.10 board. I did the quick math and buying another X9DRi-LN4F+ v1.20a was about $115 shipped while buying an empty R620 with heatsinks, HBA, 10 GbE daughter card, PSU, drive blanks, etc., was going to be about $275+. Easy call. I ordered a new v1.20a board, it arrived, and I swapped all of my components over. Fired it up and it did the usual “the BIOS was just cleared” Supermicro stuff where it power cycles twice or three times, etc. In fact, it even recognized my ESXi 7.0 installation on USB and booted right up seeing all of my iSCSI storage and RAM, etc. It was, however, on an old version of BIOS so I wanted to update that as well as change the system time and power settings only to realize… OH NO! A BIOS password was set! I tried the standard ADMIN, PASSWORD, etc. And this brings us to the fun of trying to get around the BIOS password. Not only that, but I couldn’t discover the IPMI IP address on my network and it wasn’t set for DHCP, either. This will be fun. Here it is all put back together, at least:
The main event
Having read a number of articles on STH (www.servethehome.com) I noticed a lot of people had mixed experience with the standard fixes. I am going to list things that did not work before touching on the path I took to fix this. So, here are things, in order, of what I tried.
What didn’t work:
- Removing the CMOS battery for several (up to 60) minutes
- Removing the CMOS battery and shorting contact pads of JBT1
- Booting off of a Linux live CD with the
nvrammodule loaded and issuing
dd if=/dev/zero of=/dev/nvram bs=1
Supermicro documentation even suggests:
JBT1 is used to clear CMOS. Instead of pins, this “jumper” consists of contact pads
to prevent accidental clearing of CMOS. To clear CMOS, use a metal object such
as a small screwdriver to touch both pads at the same time to short the connection.
Always remove the AC power cord from the system before clearing CMOS.
After spending a lot of time with each solution I started to think about how I could get around this in another way. On my v.1.10 board as well as whatever is in my SC847, I was able to update the BIOS from within the IPMI interface (just like Dell, HPE, Cisco, etc.) and I distinctly remembered there were some “preserve settings” options during that. However, I couldn’t get to the IPMI yet as I could not find it on the network and couldn’t enter the BIOS to change the IP configuration. You can use a tool local to the machine called
ipmicfg in order to do basic IPMI setup, but you need to get the machine up and booted to either DOS or a Linux environment. I know I can make a bootable FreeDOS image really quick with Rufus, so I did the following:
- Download Rufus and create a bootable FreeDOS USB thumb drive
- Download Supermicro’s IPMICFG tool and extract the
DOSfolder to the root of the bootable FreeDOS USB thumb drive I created
- Boot off the FreeDOS USB thumb drive and run
IPMICFG.EXE -dhcp on. There are a few ways to configure this for static vs. dynamic IP but I know I have DHCP enabled in the VLAN that the IPMI management port was connected to so DHCP was the easiest. I also did
IPMICFG.EXE -fdwhich will reset everything to the factory defaults. After this, restart the BMC/IPMI (unplug the server or run
IPMICFG.EXE -hfor the proper commands to reset without rebooting). You can either check your switch or firewall for a new IP lease or just do
IPMICFG.EXE -min order to view the current IP.
https://ip-address-from-above/and login with username: ADMIN and password: ADMIN then go to Maintenance and then BIOS Update.
- You will need to enter a license file into the IPMI web-UI for access to do Out-of-Band BIOS updates – you can apply for a trial key through Supermicro. You may also get lucky like me and acquire a board that already has it enabled. Do some searching on the internet for more information.
- Once you have the ability to upload a BIOS file, download the latest BIOS file from Supermicro’s X9DRi-LN4F+ site. Note: Also ensure that you have the latest/appropriate BMC/IPMI firmware installed first or you may experience some weirdness between BMC and other system communication.
- Finally, after you’ve uploaded the BIOS you will have the opportunity to review the current and new version. More importantly, you’ll have the option to preserve some options or not. You want to not preserve anything because this is where the password configuration exists. I opted to not preserve ME Region, NVRAM, and SMBIOS:
At this point the upgrade will run through and with luck it’ll inform you that it is successful and give you a prompt as to whether or not you want to reset the system – obviously, you do. After this reset, the system will power on and off a few times doing some initialization. Once complete, you’ll be able to Press Del for Setup and… if your experience is anything like mine, it’ll not prompt with a password any longer!
One thing I also tried was applying the same version of BIOS (my board came with 3.2, I applied 3.4, and re-applied 3.4 to test if it’d take) to my system to see if it’ll let you clear the NVRAM/etc. while applying the same, latest BIOS in case someone upgraded and then shipped it with a password. It worked for me all the same.
That’s it! It’s a little cumbersome of course – I wish the CMOS reset with or without jumping JBT1 worked, but it didn’t. Maybe you’ll get lucky and it’ll work for you. It’s a drag that the IPMI for Supermicro doesn’t have a CTRL+E or CTRL+R or whatever like Dell has for iDRAC so you can configure the IPMI outside of the BIOS but at least it’s recoverable with the FreeDOS method of IPMICFG.EXE.
Hope you all find this useful! If your board has a custom BIOS for an OEM or other system-specific BIOS this may not work. At that point I am not sure what you’d do to be honest. For most users, though, this should work with retail/standard Supermicro boards. Enjoy and goodluck!