Aaron Delp’s BladeVault

I was able to attend a session on Citrix XenServer today.  It wasn’t as technical as I would have liked but I did take away a good bit of information.  Like my Hyper-V write up , here are my notes in no particular order.

UPDATE:  I have been contacted by multiple people at Citrix regarding this post and misconceptions on my part.  First, thank you to them for contacting me!  Be gentle, I’m learning the product as quickly as I can.  I have updated the post below to reflect the latest information.  If you have any more feedback, please leave a comment or shoot me an e-mail.  Thank you!

• 6 physical NICS in a server
• XenServer supports Live Migration (think VMotion)
• XenCenter is the management GUI.  It provides the management GUI similiar to Virtual Center but is a client product that can be loaded on any server, client, etc
• Requires a 64 bit proc to run.  Supports up to 32 cores and 128GB
• Sizing of XenServer -> XenServer runs on first core (plan on it using the whole core), Xen will take an average of 580MB - 880MB (128MB at core and 200-742MB for ring 0) for XenServer,16GB for OS, one NIC for “service console”
• A “farm” of servers are called a resource pool.  This pool can have up to 16 hosts.  One server is the master and the rest are members (think PDC/BDC model in the pre-Active Directory days).  The Master holds the configuration for the entire resource pool and has the only writable copy.  Member servers are updated every minute on configuration changes
• The pool holds the configuration of all servers.  Networking must be similar but not exactly the same (although it is recommended).  Network adapters must be in the same order, on the same network, and have the same speed.
• If removing a host from a pool and VM’s are on the local storage, they will be DESTROYED!  Best practice when inserting a machine into a pool is to remove all local virtual machine storage
• If Master goes down, the members will try to reconnect for 1 minute and then go into “Emergency mode”.  New VM’s can not be created but exisiting VM’s will continue to run
• When in Emerency Mode, the members will retry every 3-5 minutes to reestablish communications
• If the Primary will be down for an extended time, a member server can be promoted to the new Master server
• NIC bonding is supported but it is Active/Passive (remember 4 maximum NICS)
• NFS Client is supported - it supports Fast Cloning and thin provisioning
• XenServer will take advantage of hardware features on a NetApp filer using iSCSI for connectivity
• XenServer has a GUI interface to apply hotfixes (think ESX Update Manager Plugin for Virtual Center)

I have a problem, I admit it.  I am an RSS Feed Addict.  Because of this I check a BUNCH of RSS feeds a few times day.  Many of these are Virtualization and Server related.  I took the time to update my blogroll on the side of the page to reflect the relevant feeds I have discovered over the last few months.  The list has probably doubled in size and I’m always on the look out for more.  If you get a chance, please check it out, you might find a few gems.  Also, if you have one to add, please leave me a comment.

Oh, in case you aren’t getting this information directly from my site or my feed, you can go to www.bladevault.info or blog.aarondelp.com and the blogroll is on the right hand side of the page.  Thank you!

This post is aimed at the ESX community.  Consider this a cautionary tale in regards to the ESX vs. Hyper-V war that is on the horizon (I’ve always wanted to use the term cautionary tale in a post, now I have!)  I’m gonna pull out my crystal ball for a second and take a crack a predicting the future.

The sleeping virtualization giant of Microsoft is now awake and there is more than one way to gain market share.  We have many past examples of products that have been good enough (the right features at the right price) that have gone on to reduce the market leaders to niche products.  Will Microsoft crush VMWare?  Probably not.  Will they become a competitor in this market that we will HAVE to pay attention too?  Yes they will.

Remember one thing…  ESX didn’t always have VMotion, DRS, HA, NFS and Virtual Center.  We were creating flagship users and doing everything through the MUI (or the command line) and guess what, it was good enough.  It was the right features at the right price.

In many ways, the features we all know and love spoil us.  If you have been doing virtualization for years, it is easy to lose sight of the beginner’s view to virtualization.  This beginner’s view is the Microsoft market.

Many people say VMWare will always rule the virtualization world.  VMWare has a head start and some technologies that simply can’t be touched by the competition today.  In addition, they are always adding products and features to turn virtualization from a commodity based stand alone product into an advanced eco-system with many unique features that both virtualization competiors and physical servers just can’t reproduce.  VMWare has done a fantastic job levering emerging technology into usable products.  They will hold the technology lead for years to come.  But, if you think for a second that Microsoft can’t out spend VMWare in R&D to play catch up, you’re fooling yourself.

Some may ask where the Citrix Xen product is at in all of this.  Honestly, I don’t know.  I haven’t been exposed to the product very much.  They seem to be caught in the middle.  They can’t compete with VMWare on enterprise level virtualization technologies and they can’t compete on the low end with Microsoft’s price point.  What sets them apart in the market?  Time will tell…

What are your thoughts?

I just attended a great session on Microsoft Hyper-V.  Before I go into my notes, let me give you my background to better frame this post.  I used to be an MCSE (NT4!) but I really haven’t touched Microsoft products in a number of years.  My focus has been on hardware architecture and eventually this has led me into the virtual architecture as it has gained acceptance into the market place.  If some of my highlights seem a little different than most posts out there, it is because I am making the ESX to Hyper-V mapping in my head.  I know ESX, I don’t know Windows Server 2008 (or 2003 for that matter).

With that out of the way, here are my points of interest in no particular order.

• Hyper-V is paravirtualized - paravirtualized means the virtual machine is “aware” (Microsoft uses the term enlightened) that it is virtualized.  If the machine isn’t enlightened, it will run in emulation mode.  Emulation mode requires a lot of context switching between user mode and kernel mode.  This will understandably slow down performance.
• The Hyper-V “Service Console” is referred to as the Management Partition.  This is a Windows VM with privileges into the kernel that other VMs do not have.  This (at least on the surface) is similar to ESX’s Service Console.
• It is recommended to run Hyper-V on Windows Core (stripped down version with no GUI).  The core version will consume less resources, require less patches, etc.
• Server 2008 has “roles” that determine the functions on the server.  Hyper-V is recommended to be the only role on the server for production
• Hyper-V does not share memory pages
• Hyper-V has quick migration instead of VMotion.  Instead of a live migration, the machine is suspend and resumed on another host.  The amount of memory will have a direct impact on the amount of time required because the memory contents will written to the disk and then read from the disk on the new host.
• Hyper-V relies on Microsoft Clustering Services right now to provide multiple host functionality for SAN connected virtual machines.  This means that Enterprise Edition is the minimum required OS level for the host to perform Quick Migrations
• It is recommended that each LUN contain only one VM.  Space needed will be disk space required + virtual RAM assigned to the machine (for quick migrations) + room for snapshots of the virtual machine
• Live Backups of a VM are supported through VSS if the guest OS is VSS aware
• Virtual Hard Disk files are .vhd files instead .vmdk files for ESX
• Raw Device Mapping (RDM) in ESX is called Pass Through Disks in Hyper-V

At the end of the session we briefly covered the Microsoft Enterprise Management Product (think Virtual Center).  It is part of Microsoft System Center and is called SCVMM (System Center Virtual Machine Manager).  Here are some points for this product.

• Since Distributed Resource Scheduling doesn’t exist today for Hyper-V, they support the idea of Intelligent Placement of a VM onto the farm.  This data is configurable but the SCVMM basically tracks performance of the hosts over a recent time period in an attempt to recommend the best placement of the new virtual machine on a host.
• The entire product is driven by Windows Power Shell and is completely customizable, exportable, etc.
• Upcoming version of the product will support ESX and well as Hyper-V.  In order to support ESX, an existing Virtual Center will be required for SCVMM to interface.  (Think single pane of glass for management).  I have my doubts on this one but I’m curious.
• Self Service Portal - End Users will be able to provision their own machines.  Again, I’d have to see this one.

I really enjoyed the presentation and Hyper-V at least has my interest.  I’ll be trying to do some hands in the near future.  Stay tuned…

I sat through a great session the other day about ESX 3i.  Here are some notes in no particular order.  This is more of a brain dump than anything else.  Nothing new here, just new to me.

• Installation -> has a time out feature, if no response is received, it will autoconfigure and continue.  i.e if no root password is entered, it will continue with blank
• Installation -> Along the same lines, if no IP address is given, it will attempt DHCP.  If it can not DHCP it will assign itself a 169.254.XXX.XXX address
• Hardware Agents and other vendor software will run as a CIM (Common Information Model) plug-in - This is how an IBM Director / HP Insight Manager Agent will be installed
• For embedded Hypervisor, 2 copies of the OS will reside on the flash.  One will be the primary boot, the other will serve as a backup in case the primary can’t boot
• Upgrades will be an entire new flash image, not a patch to the flash
• Before an upgrade is peerformed, the primary flash is copied to the backup flash
• Remote CLI commands have changed.  i.e. esxcfg-* command will now be vicfg-*
• ESX 3i installations will be seen as a Level 0 Enhanced Agent in Director, this will allow basic hardware alerts from the machine
• the installable 3i can not be copied to a flash and "converted" into embedded 3i.  The only way to get embedded 3i is to order the machine with 3i installed

This is related to my post yesterday on HP servers on the limitation of two 10G cards per server.

IBM has some similar restrictions for the 3850 M2 and 3950 M2.  As stated in the IBM Red Paper on the product , the box contains two PCI bridges (Reference page 33 of 42 in the Red Paper, last footnote).  IBM recommends that a limit of two "high speed" cards be used per PCI bridge.  Bridge #1 is slots 1-4 and Bridge #2 is slots 5-8.  A high speed card is defined as any card that can push 8GB or greater bandwidth.  There is also an IBM Retain Tip (H192284) that states the same information.

I have spoken to a few people in IBM about this and here are a few more recommendations.  In addition to the obvious 8GB+ cards (10G Ethernet, dual 4GB HBA, 8GB HBA), some other cards are also considered high speed.  The quad 1GB NIC and the RAID card for that machine can also be considered high speed.

HP recently announced an interesting product, an iSCSI HBA based on the Q-Logic 4062 chipset.  There is a link to the product specs at the bottom of this post.  The product is interesting for a number of reasons.  First and and foremost, HP needed this card to fill a hole in their product portfolio.  IBM has been shipping a Q-Logic iSCSI HBA on the blades for years.  Why do you care about this product?  Because the Q-Logic 40XX chipset is currently the only chipset supported by VMWare ESX Server.  If you want iSCSI hardware TOE and HBA, this is the only game in town.

Now, here is where the HP product shines…  The IBM card is an older form factor and you can not use the new expansion cards with it.  With the IBM Blades you need to make a decision, iSCSI HW functionality or additional expansion ports.  This has hindered the IBM prodcut somewhat from a VMWare prospective where multiple I/O paths are the name of game.

The HP card does not have this limitation.  You can use it along with any of other I/O cards with no restrictions.  Pretty slick…

Link to the Quick Specs

We have many customers that consider iSCSI vs. NFS when implementing ESX to a NetApp or N-Series array.  The pros and cons of iSCSI vs. NFS have been covered in depth many times in the past.  My favorites are here and here .  The story for NFS is very compelling vs. iSCSI but one big thing to consider is that VMWare Site Recovery Manager (SRM) isn’t supprted on NFS right now.  Yes, I’m sure it will be supported eventually.  But, if you have short range plans to implement SRM, make sure you take this fact into consideration.

I will be posting notes from the non-NDA sessions I attend here at the IBM Technical Conference but I wanted to port this one quickly.

The IBM 3850 can be purchased with embedded ESX 3i (or whatever they are calling it these days).  As of TODAY (this will proabaly change in the near future) this machine can not be grown into a multi-node x3950.  This is due to a driver limitation.

ESX 3.5 is NOT affected.  As a matter of fact, with ESX 3.5 Update 2, you can now do up to a four node 3950.

Here is something interesting I came across earlier this year while trying to add a bunch of 10G cards into a DL580 server.  It turns out that HP will only support a maximum of two 10G adapters in most systems.  I have included the link below for more information from HP.  As adapter cards increase in speed, we will have to consider bandwidth limitations such as this going forward.

By the way, IBM has similar problems.  I’m still tracking down some last second details on that one and I will post an IBM specific version shortly.

Link to HP Technical Notice

This is obvious to the experienced Linux admins out there but I wanted to pass this along.  Let’s say you have a remote ESX server with SSH access to root disabled (the default).  You now realize you need access and you have no other way to get into the box (no remote KVM, warm body on-site, etc.).  You can connect to the box and gain root access in the following way:

1. Connect to the web interface of the ESX server using the VI Client with root credentials.
2. Create a user level account on the machine.
3. SSH into the box using the newly created user account.
4. At the SSH prompt, type su and press enter.
5. You will be prompted for the root password.

After this, you will have remote access at root level to the box.  For Linux challenged out there, su stands for Super User and grants you access at the root level to the box.

Well, time to dust off the site (again!).  I have recovered from another relaspse into World of Warcraft (don’t play WoW kids, it will suck your life away) and I’m ready to start posting again.  I’m at the IBM Technical Conference this week so I thought it would be a good chance to get things going again.  More to follow shortly.