Do We Need a Multi-node IBM 3950 M2 for VMWare?
Posted by: Aaron Delp in IBM Blades, VMWareI’ve been following all the press regarding IBM’s new Multi-Node 3950 M2. The machine looks great and the ability to scale from one 4 socket box to multiple boxes (2 at this time) is simply awesome. You simply can’t match that kind of raw horse power.
But then I got to thinking… At what point is it too much? Let’s just talk about ESX for a second, I know the box has other applications but I want to focus on ESX. A single 3950 M2 offers a maximum of 16 cores (4 socket x 4 cores) plus up to 128GB of memory (256 GB when the 8GB DIMMS are released). Double that in a two node configuration and you get 32 cores and 512 GB.
ESX 3.5 currently supports 32 cores (64 core is experimental) and 256GB max. I could see an extreme situation where 32 cores with 256 GB (using the cheaper 4GB DIMMs) might be feasible if your workload is CPU bound. Yes, I know if you compare the underlying chipsets the IBM X4 chipset screams compared to anything else. It will blow the doors off most blades (no matter who makes them) for pumping raw data through the pipes. The problem at the end of the day is money. It is getting increasingly more difficult to justify the high end servers on a price vs performance comparison. The 3950 M2 performs better, but at what cost? Also, with VMWare HA and DRS features, the scale “out” (using more, smaller boxes) has become more appealing than the scale “up” (using less, larger boxes) for distributing workloads across machines while maintaining overhead for a machine failure.
Increasingly in Information Technology, it is becoming a “Wal-mart” world. Often times, good enough will do. What do you think?
(Thanks to Scott Lowe and Matt Portnoy for keeping me honest on the max values for ESX!)
Entries (RSS)
We have a bunch of the 3950’s predecessor, the 460, as well as its predecessor, the 445, all of which can be daisy-chained like the 3950. Originally it seemed like a great idea, but it’s a maintenance nightmare. Let’s say you have four of them chained together: when it’s firmware upgrade time, you can’t just upgrade one bios. You have to break them apart, boot them each into a standalone OS (whether it’s a boot CD or local disk), and then flash each one’s firmware independently. They all have to be on the same versions of firmware (bios/bmc/diags/ServeRAID/etc), but there’s no easy way to actually get & keep them there.
Then there’s a NUMA mess with memory setup: you have to make sure that the processes are tied to the right CPUs and memory. You don’t want a process executing on 3950 #1 while the memory it’s accessing sits on 3950 #2, because the memory calls go between the box interconnects. (And of course, you want all of the 3950s to have the exact same amount of memory to make this whole process easier.)
It’s a horrible solution for VM, but it’s a great solution for Windows-based apps that you think will need a lot more CPU power soon. You can start with a single 3950, and then scale it up with more 3950s if demand requires. It was a neat solution for SAP BW - add capacity with a single reboot. Just buy another 3950 (okay, well there’s that), daisy-chain them, get them on the same bios/firmware, and then reboot into a bigger partition. You could even change the partitioning over the web via the RSA. Pretty nifty.
If you ever want to talk about experiences with these, let me know and I’d be happy to.
Good Morning Brent - Thanks for the comment!
I dealt with the x445 and the x460, as well as the x3950. You are right on about flashing the nodes, it was a pain! On the 3950 you could flash mutil-nodes in place, but only from the DOS Boot disk, not from within the OS (using IBM Director to push everything down).
I haven’t had to deal with the NUMA in a LONG time so I can’t comment on that one. All I can say about the boxes is that everyone I know that had has a love/hate or a hate/hate relationship with them. The hate usually revolved around the maintenance of the box, just like your experience.
BTW - I really like your site, especially the article on IBM Director Firmware versions, I’m going to be using that one!
Thanks again!
Why, thank you sir! Have a great weekend, and keep up the blogging.
It’s an amazing server indeed, would love to play with it some time. But I can’t agree more, rather scale out and have more redundancy for probably a lower tco.
Brent - Look for a link to some of your articles in the near future, good stuff!
Duncan - It’s an honor to have you posting here
We just put a 2-node 3950 on order today for a customer (for VMWare!) so I will get some hands on soon enough! If you have anything you would like to know about the box, let me know and I’ll make a point to check it out. I’ll take lots of pictures and notes!
Those copper colored cables are too sensitive and hard to cram into the cable management trays. I too have had my own experience with the mismatched firmware problem between a pair of ESX hosts each in a 4 node scalable partition. The customer would have been so much better off with 8 ESX hosts from a virtual infrastructure and administration perspective. It’s amazing to me that the strategy of the biggest server with the highest vm to host ratio is still attractive.
Rich - Thanks for taking the time to comment! It appears I’ve opened some old wounds with this post, sorry about that!
I can see the 3950 M2 having a place in organizations that want the fastest VMWare box on the market and they are OK with a bunch of eggs in one basket. In my opinion, the majority of the time the customer will be more comfortable with blades and scaling out with VMWare HA and DRS.
I will also add that even though I can see the 3950 M2 two-node having a place in the market, the ability to scale past two-nodes just seems like a waste.
Back on xAnalisys.com a sizing guide can be found that explain why you sometimes need a 4-socket or larger server.
Just look at the numbers and study the headroom information…. because on a 2-socket systemen you can only have a max load of only 60% !!!! Read it @ http://www.xanalisys.com/Downloads/Download-document/Virtualisation-Guide-2007-/-System-x-Sizing-Guide-2007.html
Thank you very much! I will take a look over the next day or so and I’m sure I will have some questions. Thanks again!
Compare the 3.0.x system guide to the 3.5 system guide:
3.0.x http://www.vmware.com/pdf/vi3_systems_guide.pdf
3.5 http://www.vmware.com/pdf/vi35_systems_guide.pdf
Only 3.5 Update 1 support multi-node 3950 M2. So if you have an earlier ESX version, single node only. Also worth checking the other footnotes for the server type, as there are version, CPU and patch requirements.
x3950 M2 is page 15 of the 3.0.x guide, page 20 of the 3.5 guide, again both have footnotes to read.
Hope this helps,
Mike