Comments on: Do We Need a Multi-node IBM 3950 M2 for VMWare?

By: Mike

Mike — Mon, 23 Jun 2008 15:11:39 +0000

Compare the 3.0.x system guide to the 3.5 system guide:

3.0.x http://www.vmware.com/pdf/vi3_systems_guide.pdf
3.5 http://www.vmware.com/pdf/vi35_systems_guide.pdf

Only 3.5 Update 1 support multi-node 3950 M2. So if you have an earlier ESX version, single node only. Also worth checking the other footnotes for the server type, as there are version, CPU and patch requirements.

x3950 M2 is page 15 of the 3.0.x guide, page 20 of the 3.5 guide, again both have footnotes to read.

Hope this helps,
Mike

By: Aaron Delp

Aaron Delp — Thu, 05 Jun 2008 19:02:34 +0000

Thank you very much! I will take a look over the next day or so and I’m sure I will have some questions. Thanks again!

By: Cornel

Cornel — Thu, 05 Jun 2008 18:49:25 +0000

Back on xAnalisys.com a sizing guide can be found that explain why you sometimes need a 4-socket or larger server.
Just look at the numbers and study the headroom information…. because on a 2-socket systemen you can only have a max load of only 60% !!!! Read it @ http://www.xanalisys.com/Downloads/Download-document/Virtualisation-Guide-2007-/-System-x-Sizing-Guide-2007.html

By: Aaron Delp

Aaron Delp — Sat, 08 Mar 2008 03:19:33 +0000

Rich - Thanks for taking the time to comment! It appears I’ve opened some old wounds with this post, sorry about that!

I can see the 3950 M2 having a place in organizations that want the fastest VMWare box on the market and they are OK with a bunch of eggs in one basket. In my opinion, the majority of the time the customer will be more comfortable with blades and scaling out with VMWare HA and DRS.

I will also add that even though I can see the 3950 M2 two-node having a place in the market, the ability to scale past two-nodes just seems like a waste.

By: Rich

Rich — Sat, 08 Mar 2008 00:50:51 +0000

Those copper colored cables are too sensitive and hard to cram into the cable management trays. I too have had my own experience with the mismatched firmware problem between a pair of ESX hosts each in a 4 node scalable partition. The customer would have been so much better off with 8 ESX hosts from a virtual infrastructure and administration perspective. It’s amazing to me that the strategy of the biggest server with the highest vm to host ratio is still attractive.

By: Aaron Delp

Aaron Delp — Fri, 07 Mar 2008 20:07:46 +0000

Brent - Look for a link to some of your articles in the near future, good stuff!

Duncan - It’s an honor to have you posting here We just put a 2-node 3950 on order today for a customer (for VMWare!) so I will get some hands on soon enough! If you have anything you would like to know about the box, let me know and I’ll make a point to check it out. I’ll take lots of pictures and notes!

By: Duncan

Duncan — Fri, 07 Mar 2008 18:48:24 +0000

It’s an amazing server indeed, would love to play with it some time. But I can’t agree more, rather scale out and have more redundancy for probably a lower tco.

By: Brent Ozar

Brent Ozar — Fri, 07 Mar 2008 17:59:46 +0000

Why, thank you sir! Have a great weekend, and keep up the blogging.

By: Aaron Delp

Aaron Delp — Fri, 07 Mar 2008 13:07:12 +0000

Good Morning Brent - Thanks for the comment!

I dealt with the x445 and the x460, as well as the x3950. You are right on about flashing the nodes, it was a pain! On the 3950 you could flash mutil-nodes in place, but only from the DOS Boot disk, not from within the OS (using IBM Director to push everything down).

I haven’t had to deal with the NUMA in a LONG time so I can’t comment on that one. All I can say about the boxes is that everyone I know that had has a love/hate or a hate/hate relationship with them. The hate usually revolved around the maintenance of the box, just like your experience.

BTW - I really like your site, especially the article on IBM Director Firmware versions, I’m going to be using that one!

Thanks again!

By: Brent Ozar

Brent Ozar — Fri, 07 Mar 2008 12:24:04 +0000

We have a bunch of the 3950’s predecessor, the 460, as well as its predecessor, the 445, all of which can be daisy-chained like the 3950. Originally it seemed like a great idea, but it’s a maintenance nightmare. Let’s say you have four of them chained together: when it’s firmware upgrade time, you can’t just upgrade one bios. You have to break them apart, boot them each into a standalone OS (whether it’s a boot CD or local disk), and then flash each one’s firmware independently. They all have to be on the same versions of firmware (bios/bmc/diags/ServeRAID/etc), but there’s no easy way to actually get & keep them there.

Then there’s a NUMA mess with memory setup: you have to make sure that the processes are tied to the right CPUs and memory. You don’t want a process executing on 3950 #1 while the memory it’s accessing sits on 3950 #2, because the memory calls go between the box interconnects. (And of course, you want all of the 3950s to have the exact same amount of memory to make this whole process easier.)

It’s a horrible solution for VM, but it’s a great solution for Windows-based apps that you think will need a lot more CPU power soon. You can start with a single 3950, and then scale it up with more 3950s if demand requires. It was a neat solution for SAP BW - add capacity with a single reboot. Just buy another 3950 (okay, well there’s that), daisy-chain them, get them on the same bios/firmware, and then reboot into a bigger partition. You could even change the partitioning over the web via the RSA. Pretty nifty.

If you ever want to talk about experiences with these, let me know and I’d be happy to.