Aaron Delp’s BladeVault.info

I ran across this one a few weeks ago and I wanted to share it. If you are using the BNT (Nortel) switch modules in the IBM BladeCenters and booting iSCSI to various OS’s, you might run into problems using the default values on the Nortel switch. We were seeing our LUN’s intermittently not attach at boot and/or crash loading up. This happened with Windows, Linux, and VMWare. After some searching I came up with this link to turn OFF autonegotiation. This setting used to be off but in the latest BNT firmwares, the default value was changed to ON. If you switch it back to off, you are able to boot everything fine.

Sorry, I’m being a little cynical with my quotes. This is based on the last time they announced an “Intelligent” product and it turned out not to be functionally equivalent to the old product .

I saw this article over at xAnalysis and after some digging I came up with the new CPM Product Announcement . The big difference I can see is the new product no longer requires “octopus” cables. The announcement states the product will have 14 external network connections to match one:one the internal blade connections. I would love to see a picture on this sucker, I can’t believe they managed to get that many ports on a module. Only time will tell if it is functionally equivalent…

UPDATE: Simon was nice enough to leave a comment with a link to the product picture.  That is an impressive use of real estate!

This has been out a few weeks but it just came to my attention. IBM recently announced a Retain tip that the IBM BladeCenter Optical Passthru Module needs to be replaced because it will fail after 511 days of activity. This ECA (Engineering Change Announcement) is considered MANDATORY. Please see this link for more information and how to submit a form to have IBM replace the module:

IBM Retain Tip on faulty OPM

In my day to day activities I depend a lot on the hardware vendor forums for the products I support.  I wanted to take a second to share all the links I have acquired.  Some will be obvious to everyone, but a few of them are pretty obscure.  Enjoy!

• Q-Logic Adapter Forums
• BNT (Nortel) IBM BladeCenter switch Forum
• VMWare Communities
• NetApp Communties
• HP BladeSystem Forum
• HP Insight Manager Forum
• HP Proliant Server Forum
• HP iLO Forum
• IBM Blade Forum
• IBM System x Server Forum
• IBM Director Forum

I’ve been following all the press regarding IBM’s new Multi-Node 3950 M2. The machine looks great and the ability to scale from one 4 socket box to multiple boxes (2 at this time) is simply awesome. You simply can’t match that kind of raw horse power.

But then I got to thinking… At what point is it too much? Let’s just talk about ESX for a second, I know the box has other applications but I want to focus on ESX. A single 3950 M2 offers a maximum of 16 cores (4 socket x 4 cores) plus up to 128GB of memory (256 GB when the 8GB DIMMS are released). Double that in a two node configuration and you get 32 cores and 512 GB.

ESX 3.5 currently supports 32 cores (64 core is experimental) and 256GB max. I could see an extreme situation where 32 cores with 256 GB (using the cheaper 4GB DIMMs) might be feasible if your workload is CPU bound. Yes, I know if you compare the underlying chipsets the IBM X4 chipset screams compared to anything else. It will blow the doors off most blades (no matter who makes them) for pumping raw data through the pipes. The problem at the end of the day is money. It is getting increasingly more difficult to justify the high end servers on a price vs performance comparison. The 3950 M2 performs better, but at what cost? Also, with VMWare HA and DRS features, the scale “out” (using more, smaller boxes) has become more appealing than the scale “up” (using less, larger boxes) for distributing workloads across machines while maintaining overhead for a machine failure.

Increasingly in Information Technology, it is becoming a “Wal-mart” world. Often times, good enough will do. What do you think?

(Thanks to Scott Lowe and Matt Portnoy for keeping me honest on the max values for ESX!)

This article over at the Server Virtualization Blog got me thinking… Are Blades the next “Pizza Box” servers for ESX? By that I mean are Blades reaching the mainstream to the point that they are becoming a commodity? In my role as a pre-sales Engineer, I speak to many customers and you can predict what many of them want to talk about before we walk in the door, Blades and ESX.

Yes, we still move pizza boxes and many customers (usually the price sensitive ones) still love them. Now that the blade market has matured, we are seeing less and less of the pizza box attitude.

Let’s take that a step further, what are they buying? Some decide to go with the smaller form factor blades (IBM HS21 and HP BL460c) but a surprising number are going for the larger HP BL680c. The BL680c Blade is a four socket Intel Blade with a maximum of 128 GB of memory and plenty of expansion ports, especially using the quad port Ethernet expansion (remember to use the right model quad port card for ESX!!) For me, the small blade vs large blade decision always ends in the “it depends” answer.

You’ll notice I didn’t throw out an IBM 4 socket model. Not to throw IBM to far under the bus on this one but LS41 product just isn’t appealing to most customers right now. That is a discussion for another day.

What are you seeing? What are your thoughts?

Just a heads up for everyone, the new 31.4 GB Solid State “Disk” (43W7618) for the IBM Blade Servers is supported on the entire product line. The previous generations (16GB single and dual platter) were only supported in the HS21 XM.

Also, this model is a single platter, not a dual platter (think RAID 1 within the disk case) and as far as I know a dual platter isn’t in the works for now.

IBM also has a Power Calculator, available here.  The IBM tool is a little rough around the edges, but it gets the job done.

The IBM tool is a download vs. an on-line tool.  The configuration of a chassis is very easy to do and the report is very straight forward.  I would like to see a user populated utilization level to customize the report but you can make the calculations yourself with the information provided.  Lastly, IBM allows you to add the PDU’s you need, a nice touch.
IBM only supports exporting the data to an Xls format.  All in all, a bit of an ugly baby but it provides the necessary information.

This is an update to this article on the IBM HS21 BIOS 1.07B.

IBM has confirmed to me that this is a problem that should be addressed in another BIOS patch. The only OS that seems to be affected is Red Hat. According to IBM support, it turns out that the BIOS looks at the stepping levels of the chips and if the second CPU level is different than the fist CPU level, it will “mask” them to the OS so this problem will be avoided.

For some reason, there is a bug in the code as of 1.07B that this doesn’t happen. It has been confirmed and should be corrected shortly.

This is actually old news and is one of those you won’t believe it until you try it kind of tips. It occurred to me to post this tip when I was creating the bootable floppies to flash the HBA and TOE cards.

Believe it or not, IBM Blade Centers (and most IBM servers these days) will not boot to an NEC USB Floppy. In order to boot to a floppy, it has to be a TEAC floppy disk drive. For whatever reason, the NEC drive doesn’t pick up as a boot device and the system will by-pass it.

If you are using IBM Blades in a Boot from SAN configuration and you want to flash to the latest BIOS and firmwares, how do you do it before you install the OS?

The answer is good old DOS. The 4GB HBA and iSCSI cards are both made by Q-logic and if you look hard enough, you will see that they provide DOS utilities to flash both of these cards to the latest version, if you have a DOS boot disk. Remember the DOS Boot Disk? Do you still have some in your desk or in a storage cabinet somewhere?

If you’re like me, it took a little bit to come up with a floppy drive and some blank media to work with. I didn’t want to go through this again so I created the disks, made IMG files of them, and stored them on the site for safe keeping. I created a custom page over here if you are interested in downloading them. I also included the commands to invoke the flash for both disks.

I had a chance to play with the IBM Blade Center Intelligent Pass-thru Module (IPM) on Friday. The device looks pretty slick, as long as you are using it in the right environment. We were told by IBM that the OPM (old 2GB max Optical Pass-thru Module) was going End of Life in January. So, being the good little Partner we are, we moved our customer to the IPM.

It turns out the IPM is a whole different beast. The OPM was a device that simply passed the optical signal from the expansion card on the blade to an external optical network or FC switch. This relationship was a one to one ratio.

The new IPM is basically a “dumb” Q-Logic FC switch that doesn’t participate in the FC fabric. It takes the 14 internal connections and pipes them through 6 external connections. The internal to external port mapping is configurable in the switch. The switch can also be upgraded (and downgraded back down) at any time to either a 10-port or 20-port FC switch.

All in all, it is a pretty slick set up but this approach does have down sides that the OPM didn’t have. First, the OPM had a one to one ratio (limited to 2GB speed) and second, the OPM can be used for Ethernet. The IPM has the advantage of 4GB speed and user configurable mapping of internal to external ports.

Our customer was using the OPM to pass Ethernet to Cisco switches so this wasn’t a good solution for them!! We made a few calls and it turns out we weren’t the only ones. IBM is now planning to continue production of the OPM for the near future.

Link to the IBM Redpaper on the IPM