Posted By Jon on Feb 4, 2017 | 3 comments

Hey all – sorry it’s been a while since I’ve updated but I assure you I’ve been working on lots of cool stuff.  I’ve been leaning hard on my latest ESXi host revision which has provided my primary lab with:

☑ Bunches of compute
☑ Gobs of RAM
☑ 10 jigabits of network tubes
☐  Enough drive bays to hold tons of disks

Hrmph.

I backed myself into a corner that I saw coming for a while.  I create more and more data and watched my free space fall and fall!  I often lab out entire customer environments for migration testing!  And, soon thereafter, run out of disk space.  Ok, maybe “running out” is premature, but my free space is getting to a point where I need to be cautious about what I keep and who likes that?

6.7TB free after much, much pruning!  That should be good for a bit but won’t work for me in the long run.  The problem is, my Rosewill RSV-L4000 box can only really hold 8 disks without getting reckless.  And, even if you jam another four spindles in there, you will need an expander or another HBA to cable to.  That’s no fun.  So, as usual, that just means it’s time to upgrade!  I originally kept the Rosewill chassis during the last hardware build because I wanted to keep things affordable.  Now, I only have a few reasonable requirements:

1. A replacement chassis must be 4U+ because I have Noctua coolers I want to reuse (because they’re amazing, expensive, and brand new)
2. Quiet – not whisper quiet, but if my wife can hear the server over a blender I’ll have issues
3. Support for 20 – 24+ 3.5″ 4TB+ disks
4. Cheap (of course…right?)

So what’s that all come down to?  Two options:

Norco 4220/4224

OR

Supermicro SC846

The difference between these two units comes down to one simple issue for me – the Norco unit doesn’t have a real backplane.  It has several individual backplanes that either need to be cables with individual SATA cables or multiple SFF8087 mini SAS cables.  That, and the backplanes are not extremely compatible and people have had issues with them failing.  Now, to be honest, I’ve never been a huge Supermicro fan.  I know they make many of the big name chassis out there and badge them otherwise, but I was still skeptical.

Because I value reliability over anything else, I decided to check out the Supermicro scene and did a ton of reading.  Turns out, I could pick up a SC846 24-bay chassis for $135 off eBay.  I ordered and it arrived – at first I was super excited that I may of mistakenly received a NetApp FAS8xxx!

SC846 in NetApp clothing

Once removed from its awesome packaging from the seller, I was able to check out the construction of the Supermicro SC846 chassis.

SC846 Initial Unbox

Results?  I’m skeptical no more.  Not only is the Supermicro SC846 about as solid/robust as any Dell/NetApp/EMC box I’ve touched, it also has immense adaptability and support for non-standard configurations.  I am doing something a little oddball afterall – I am putting the Intel S2600CP2J from this post into the Supermicro box.  That means I need 12″ x 13″ motherboard support, dual +12V CPU PSU connectors, and since I’ll be using all of my existing network adapters and HBAs, I need those to work too!  Cliff notes:  everything works (though there were a few hurdles, which is what this post is about after all).

Making it all work

Firstly, I bought the Supermicro SC846 from eBay for $135.  Yes, you read that correctly – $135.  However, I knew up front that I’d need to replace the backplane.  Why?  Well, the older generation SC846 (the <$700 ones you see on eBay), though physically the same as later models, shipped with SAS1 backplanes.  Because I am using 6.0 Gbps disks (and SSDs) and have disks >2TB, I needed a SAS2 backplane:

The image above shows that the SAS1 and SAS2 backplane look almost identical.  The top one, however, is the SAS2 unit which is needed in order to provide 6.0 Gbps throughput as well as support for 2TB and larger disks.  I am using 4TB disks along with SSDs, so obviously this is the only choice.  Unfortunately, the SAS2 backplane is not super easy to find.  You can buy Supermicro boxes with the SAS2 backplane already installed but expect the price to be >$500 or so for the SC846 with the correct backplane.  That said, I did find one for sale for $200 and swapped it in – kind of a pain but I got it done.

All I needed to do next is swap the mainboard w/ CPUs and RAM already installed from my Rosewill box into the SC846 and I’d be set:

I did run into a few decent snags.  All of them are because I am installing an Intel S2600CP2J into a Supermicro box.  Supermicro surely accounts for some of my issues with their own boards.  Firstly, the +12V CPU power connectors on the SC846 are JUST long enough to mate up to the Intel S2600CP2J.  I mean, JUST long enough.  After that, if you thought the +12V CPU connectors were short… the ATX 24-pin PSU motherboard connector is even shorter.  In fact, I had to straighten the cabling out to even make this work.  I was concerned I’d have to “roll back” my plans for the evening and move everything back into the Rosewill chassis while I ordered a 24-pin extension to make this work.  While I was able to get the ATX 24-pin to reach, I ordered an extension because it was way, way too tight:

ATX 24-pin Extension

Next up is the front panel control – Supermicro uses a ~16-pin connector for this and has a nice cable supplied with it.  However, Intel doesn’t use that – they use an SSI 24-pin front panel header which is actually more standard-issue.  Fortunately, Supermicro makes a break out cable for this!  The part number for that is CBL-0084L:

Supermicro FP Adapter

In order to use the CBL-0084L adapter you just plug the original 16-pin ribbon (hopefully supplied with your SC846 chassis) into the power board and then into the male/male union which then breaks out into conventional pin header plugs:

Supermicro FP Header

You will need to consult your motherboard manual for the pin-out on the board, but you get the idea.

I also needed to buy longer SFF-8087 cables because the ones that came with the chassis were just too short for me to use.  Finally, I did manage to get a PWS-920P-SQ power supply for the chassis which is much much quieter than the originals.

One thing worth mentioning is that I purchased the Intel RMM (remote management module) with hardware key and dedicated NIC port for my Intel S2600CP2J board since I am remote of this machine often.  This guy has a ribbon cable that goes to the motherboard, a small hardware key that sits on the board, and then a PCI slot adapter you can use – it’s very nice, although it uses Java for KVM support but so do all of the other manufacturers:

Intel Remote Module

So, after all the swapping of parts, here is everything that was removed and no longer needed from the original Rosewill build combined with what came out of the SC846 as shipped:

In the image above, you can see I no longer need the traditional ATX EVGA Power Supply.  The blue SFF-8087 cables are no longer needed as they are just a bit short for my LSI 9260-8i card (which has perpendicular connectors).  You’re also looking at the plastic duct used for Supermicro motherboards w/ low profile HSF’s, the original SAS1 backplane, and the two standard hot-swap PSU’s that came with my SC846 (that are just way, way too loud).

Inside the SC846 is pretty plain jane.  You’ll notice right away a pair of 80mm fans at the back of the chassis with their power wires draped outside of the case.  I am currently using these two fans as exhaust fans because the stock Supermicro units combined with my Intel S2600CP2J are just way too loud.

In case you don’t know, the fan situation with the Intel S2600CP2J is a bit of a PITA.  You need to edit the FRU/SDR files to control different chassis fans at different RPM profiles.  So, until I get around to tweaking that, the 80mm retrofits are quiet enough but move adequate air to keep things in check.  One super convenient thing about going to a chassis with a proper backplane is that there are not longer oodles of power cables and individual SATA cables going to each disk.  This greatly improves the internal cabling situation within the chassis:

Inside the SC846

Inside the SC846

One small modification I have made is in regards to the LSI 9260-8i controller I am using.  I have found, especially lately with my newly added SSD tier for storage, the controller can get warmer than I prefer.  I have not experienced any temperature alerts while under load/operation, but I do know that these cards are originally shipped/used in many 1U/2U/3U/4U chassis with higher velocity airflow inside.  Because I have basically no air moving in the area of the controller, I decided to fit a 40mm Noctua fan directly to the heatsink on the controller.  Again, no alerts and no reference temperatures means I don’t have anything to compare to, but at the same time I think it’s worth the cheap investment to do this:

Noctua fan on LSI-9260-8i

As soon as I get the fan profile for the Intel S2600CP2J board to work well with the SC846 fans, I’ll follow up!  For now, they sit on my shelf, sad, waiting to move cool breezes across components:

SC846 Fans

All said and done, this is probably the cheapest way I could have gone about getting 24-bays of capacity while also having a reliable, hot-swap, redundant solution.  While the Supermicro SC846 may not be the most attractive beast in the kingdom, it’s well made and does what it needs to.  I will have to update everyone with the contents of my primary lab rack now that this powerhouse is installed:

SC846 looking majestic

That’s all for now everyone!  Apologies for not updating since November, I have been busy on all fronts!  I will update very soon with the new storage improvements within this very server.  As always, thanks for reading!