Wednesday, December 5, 2012

vC Ops and Hyperic - Dynamic Duo of Enterprise Management

In this post I will explain how to configure vC Ops 5.6 and Hyperic 5.0 integration. A few requirements and assumptions need to be stated up front.

- You own vC Ops 5.6 Advanced or Enterprise and have access to the Custom UI.
- You have installed and configured Hyperic 5.0 (I highly recommend the new Hyperic vApp - easy to install and configure plus it scales nicely).

Why would you want to do this? Very briefly, Hyperic adds a capability to combine metrics analysis from the guest OS (as well as physical systems) with the analytics already being derived for the vSphere environment. Basically, there's only so much that vC Ops can "see" via the vCenter API and it's important to understand what's going on in the guest OS on a VM to have a more complete picture of performance and health.

Onward and upward, time to get this working.

First, you will need to obtain the Hyperic adapter for vC Ops from VMware. Once you've downloaded the adapter, connect to the Admin UI of your vC Ops UI VM and log in as the admin user. Under the "Update" tab you will find the option to browse for the adapter file (should be a *.pak file). Press the "Update" button and accept the EULA. The install will proceed and you will have nice series of tasks in the Update Status window.

Once this is done, you will now log into the Custom UI of your vC Ops UI VM where you will click on "Support" link in the upper right section of the UI. In the Support page, select the "Info" tab to get to Adapters Info.
 You will need to allow vC Ops to initialize (or "Describe" as it is called) this adapter so you can use it. Click on the small gear icon in the "Adapters Info" section to start the describe process. Just select "Yes" on the confirmation diaglog. You can refresh the "Info" tab after a minute to make sure that your adapter has been added to the list - here you see I've already described a Hyperic adapter.


 Next you'll need to configure an adapter instance of the Hyperic adapter to point to your Hyperic server.  From the "Environment" pull down menu, select Configuration > Adapter Instances. 


This will bring up the Adapter Instances manager where you can add/edit/delete adapter instances. You will select the green plus icon in the manager to add the Hyperic instance.


This is a simple dialog and you can see the fields I have completed. The "Adapter Instance Name" can be whatever you wish, I recommend you make it descriptive as you would in naming a VM or server. If you are using the Hyperic vApp, note that you will use -
  • DB Type = PostgreSql
  • Port = 5432 (NOT the default 9432)
  • DB Name = HQ
The final item "Credential" you may now add by clicking, the "Add" link (** thank you Captain Obvious)

Again here is an example completed. The "Instance name" is a descriptive name you provide - it doesn't have to match the adapter instance, I just like to do that for simplicity. The user name is hqadmin for DB access if you are using the vApp. Once completed, click OK to return to the instance configuration.


At this point you can confirm you settings by clicking the "Test" button - if all is well, you'll get a pop up like the one above. If not, you'll get a pop up that is really not helpful at all but never fear, just validate the settings and credential and try again. You may also want to make sure you can ping the Hyperic DB server from the vC Ops Analytics VM. That's the extent of troubleshooting I'll provide here, but feel free to leave questions in the comments section - or even better, head over the VMware's Community and post your question in the vC Ops or Hyperic community (I regularly monitor the vC Ops community).


Now that your adapter instance is configured and working, head over the the Environment Overview page and after about 5 - 10 minutes you will have some new resources. Open and select Adapter Instances - Hyperic Adapter Instances - (your instance name) to see.

Notice I now have good stuff like various platforms (Windows, Linux) and some services.

That's all it takes. I'll have a future post on creating relationships between vCenter and Hyperic resources as well as some custom dashboards featuring Hyperic metrics.

*this post was created with a trial version of ScreenSteps

Friday, July 27, 2012

Protecting vC Ops Data with VADP and SRM

If you use vC Ops to monitor and report on your environment, you are aware of how important the historical analysis and metrics collected are to getting value out of the product. 

Yesterday, VMware published three KB articles outlining ways to protect your vC Ops data.

KB 2031899 - Managing a VMware vCenter Operations Manager 5.x deployment where the vApp is broken

KB 2031890 - vCenter Operations Manager 5.0.x: Using VMware Data Recovery to Back Up and Restore a vApp deployment

KB 2031891 - vCenter Operations Manager 5.0.x: Using Site Recovery Manager to Protect a vApp Deployment

 A couple of things to call out from these KBs...

- While VDR may not be a viable option for your environment, note that it's the VADP capabilities that allow you to perform an snapshot based backup of your vC Ops vApp that is critical.  So, if you are using an enterprise backup solution which supports VADP you're in good shape.  Ideally, your backup solution will include support for vApp aware backups - but even if not you can still take advantage of VADP-based backups to protect your vC Ops data.

- Embrace the broken vApp!  As you can see from KB 2031899, the vApp is not necessary for running vC Ops as long as you are aware that things like startup/shutdown sequence as well as IP address management are going to require a little more attention.  I've had some customers ask about moving or re-IP of vC Ops and this KB should help.  This is also key information if you are using a non-vApp aware backup solution and you have to restore the two VMs - you need not worry about recreating the vApp at that point.

- Speaking of vApps - while SRM isn't going to support protection of vC Ops as a vApp, you can still protect the two VMs individually as described in KB 2031891.

Hopefully this helps customers who are concerned about losing months of collected data and analysis!

Wednesday, July 25, 2012

Monitoring vC Ops Upgrades

I was working with a customer this morning updating their vC Ops vApp from 5.0.0 to 5.0.2 and while everything went well it was a little frustrating because there is no feedback on the status of the update other than a "progress" animation.

To provide more detail on the status of the update, we opened a putty session on the UI and found that the /var/log/vmware/updatecoordinator.log was being updated as the upgrade progressed.

By running tail -f updatecoordinator.log we were able to get a better understanding on the progress and status of the upgrade as in the image below.

certainly a lot more interesting to watch than a progress bar!

Friday, July 6, 2012

vCenter Operations Manager - What's new in 5.0.2?

VMware released a minor update for vCenter Operations Manager yesterday evening.  The 5.0.2 release contains some bug fixes as well as enhancements for those using the custom UI in the vApp or the standalone version of the product (i.e. Enterprise or Enterprise Plus customers).  There's also a critical security fix for the vApp as well as a fix for support bundles.

Some items of interest -

A new Text widget has been created to allow personalization of the custom dashboards.  You can display plain, unformatted text or HTML from a URL or file.  Below I show an HTML configured Text widget to provide up to date weather forecasts on my dashboard.  KB 2030076 has details on configuring the Text widget.

Mashup charts widget now has three buttons to filter alerts (only active, only inactive or all alerts).  Also, there's more detail on the pop-ups when hovering over an alert.

By the way, this applies to the vSphere UI as well - check out the Operations > Events tab...

You can now configure hint text for KPI alerts.  See KB 2021380 for configuring.

The Alert widget now shows all children of the selected resource, where previously only first level decedents were shown.

You can now configure complex alert rules that evaluate multiple conditions on related resources. The example included with the update (vmware-vcops/user/conf/analytics/multi-level-alert-rules.xml) shows how to format the rule.  Basically, this allows you to create alerts based on a combination of hard thresholds (HT) and dynamic thresholds (DT).

Other enhancements include -

  • Weighted metric groups 
  • Skip resources for health rollup
  • Prioritize threshold checking over Dynamic Threshold calculation

If you are already running the 5.0 or 5.0.1 vApp, you only need to download the VMware-vcops-5.0.2-770497.pak file and then update your vApp using the Admin UI (using the Update tab).  Be sure to shut down your vApp and take a snapshot before applying the update!

For the standalone version, you will need either the .bin or .exe file depending on your OS and then install using the "reinstall" option.  See the release notes for more detail.

Tuesday, May 22, 2012

vC Ops - Basic vApp Troubleshooting

vCenter Operations Manager 5 has really generated a lot of interest and many people are deploying a trial in their own environments to give it a test run.  I highly recommend this, there's nothing like using vC Ops with your own data to see what it can do and how it can help you.

However, I am seeing some common issues that have caused some trials to come to a halt.  For the most part, these are easily addressed and prevented.  I've put together a basic troubleshooting list to help make your trial or proof of concept run more smoothly.  Comments welcome!

Update to 5.0.2. Once again, several bug fixes have been introduced and this release includes the fixes from 5.0.1 noted below.

Update to 5.0.1. This version introduces an automatic fix for the "broken vApp" where a repair is done automatically when it is detected that the VMs can no longer communicate.  If you update to 5.0.1 from 5.0 you'll be prompted for reconnection information the first time you log into the Admin UI after a connectivity issue.  For 5.0.1 fresh installs you'll get prompted the first time you log into the Admin UI after deployment.

  • Upgrading to 5.0.1 fixes several stability and performance issues
  • 5.0.1 removes the need to run a manual repair when IP addresses change on the VMs

Shutting down/starting up.  The vApp is designed to start and stop the VMs in proper sequence.  It is highly recommended to shut down the ENTIRE vApp even if you only want to modify the UI VM (for example).

  • Always use the vApp to start the VMs by selecting the vApp in vCenter and then clicking the "green arrow" start button as you would with a VM.
  • Always use the "Power Off" button to shut down the vApp - even though it seems like that would cut power to the VMs it actually does a safe shutdown.
  • Failure to do so will (eventually) cause DB corruption.

IP Pools setup.  Basically, the IP Pool is used by the vApp to set up a VPN tunnel between the two VMs.  Here are some things that often get overlooked or mis-configured:

  • Do not enable the IP Pool, I've seen this cause issues with IP address assignment for the VMs in the vApp.
  • You only need configure the subnet, mask and gateway on the IPv4 tab - an IP pool range is not required (and you won't be able to edit unless the IP pool is enabled).
  • Use the DNS tab on the IP Pool properties to configure your DNS servers.  
  • In general do not use the "blue console" to configure networking, even though there's an option to do that.  Always use the IP Pool and vApp properties to manage networking.
  • Always use static IP.  Even if your DHCP environment is well run, a change in IP creates havoc. 
  • Always change IP addresses by shutting down the vApp and editing the vApp properties.
  • Verify network connectivity between the first and second VM.  From the console of the UI VM, ping "secondvm-internal" and "secondvm-external" to check that all is well.  If not, very likely you need to run the repair (see the section on vcops-admin command below).  If a ping works, simply shut down and restart the vApp to get things working again.
Disk space.  The sizing options when you deploy the vApp are pretty helpful, and many customers will select small because it goes as high as 1500 VMs.  This is an estimate and I have seen customers with as few as 1000 VMs run out of space within 90-180 days.  This can cause a number of issues depending on which VM has used all the space available on the /data partition.  I have seen this manifest itself in a few different ways so I always check disk space now as a matter of routine.

  • Check the /data partition on both the UI and Analytics VM using df -h 
  • Add additional storage if needed by first properly shutting down the vApp (Power Off - trust me, that's the right way) and add another VMDK to the VM(s).  Start the vApp and the additional disk space will be added to the /data partition automatically by the VM.
The vcops-admin command can be your friend.  Run from the UI VM (as the admin user) this is a set of tools to address most common issues.  Use vcops-admin help for a list of commands and options.

  • The repair command fixes a broken vApp.  That is, if your VMs are not communicating due to an IP address change (because you used DHCP for example) this command will get them talking again.
  • You can use vcops-admin to restart the services on the UI VM.  Sometimes this corrects issues with web pages not being accessible or other odd behavior - after you've checked storage and networking.
  • Sometimes the admin password is forgotten.  It happens.  Use the password command.
  • You can create a diagnostic bundle for VMware support from vcops-admin if you are not able to access the admin UI via web browser.
Missing scroll bars. It has been noted by some customers that certain combinations of Windows OS and IE 8 will cause the scroll bars to go missing in the the main view panel of the vSphere UI.  If you experience this, try using IE 9, Firefox or Chrome (I actually find Chrome works best).
I will likely be updating this post with additional information in the future, so please check back frequently.

*Updated 5/22 - 5.0.1 upgrade
*Updated 7/25 - 5.0.2 upgrade and missing scroll bars

Thursday, May 3, 2012

vC Ops Upgrade - Back Up Your Heat Maps

I was looking over the release notes - AFTER I upgraded my lab, of course - and found that custom heat maps are not preserved when upgrading from vC Ops 5.0 to 5.0.1  so be sure to back them up!

Fortunately, I took snapshots of my vApp before the upgrade.  I highly recommend this, even if you aren't concerned about the heat maps.  Just be sure to do a "Power Off" of the vApp before taking snapshots of the UI and Analytics VMs.

Anyway, for backing up the heat maps see these instructions from the release notes for 5.0.1 build 677931 -

Custom Analysis heat maps are not preserved after upgrade from vCenter Operations Manager 5.0 to 5.0.1
During the upgrade, vCenter Operations Manager 5.0.1 overwrites the configuration files that contain the heat map settings. As a result, the custom heat maps that you created in vCenter Operations Manager 5.0, are no longer found on the Analysis tab.
Workaround: Create a backup copy of your heat map configuration files before you initiate the upgrade.
  1. Log in to the UI VM as root user.
  2. Navigate to folder /usr/lib/vmware-vcops/tomcat/webapps/vcops-vsphere/WEB-INF/classes/, and create a backup copy of the heatmapConfigs.xml file to a folder outside the installation path.
  3. Navigate to folder /usr/lib/vmware-vcops/tomcat-enterprise/webapps/vcops-vsphere/WEB-INF/classes/, and create a backup copy of the heatmapConfigs.xml file to a folder outside the installation path.
  4. After the upgrade, use the backup copies of the heatmapConfigs.xml files to update the new heatmapConfigs.xml files with your custom settings.

Monday, April 30, 2012

vC Ops Custom Dashboard Widget Interaction Reference

The past few posts have explored some capabilities of the Custom UI in vC Ops, including custom dashboards.  Please keep in mind that you don't need to use custom dashboards or the Custom UI to get a lot of benefit from vC Ops.  A lot of consideration went into the design and functionality of the vSphere UI and it is a very useful interface.  Before jumping into custom dashboards, take time to get familiar with the vSphere UI as it's very likely going to provide the vSphere admin with everything they need.

If you've worked with custom dashboards you may know that widgets can interact with each other.  That is to say, widgets can provide certain resource types to other widgets to display specific information on your dashboard and make it interactive.

The following matrices may be of use to you as a reference in creating widget interactions.  The first shows each widget as a provider and then describes the resource types the widget provides and which widgets can receive them.  The second matrix enumerates the resource types by receiving widget and then provider.

Thanks to the folks in Technical Marketing at VMware for allowing me to share this information.

Select "read more" below to see the matrices.

Thursday, March 29, 2012

Modify Timeout in vC Ops vSphere UI

If you're using vC Ops 5 you may have noticed there's a timeout for the vSphere UI login.  Kind of a pain if you're wanting to display the "skittles" view at your NOC or on an extra monitor at your workstation.  While there's no way to configure this timeout in the UI, you can change the timeout or set it to unlimited by modification of an XML file on the UI VM.

On the UI VM the file /usr/lib/vmware-vcops/tomcat/webapps/vcops-vsphere/WEB-INF/web.xml contains the setting in the section (conveniently near the top of the file).  By default it is 30 minutes.  Setting it to -1 will make it unlimited.  From the web.xml file -

Session timeout is measured in minutes. Use '-1' to have infinite session.

Simply modify the file using vi or editor of your choice.  You'll need to restart your vC Ops service:

vcops-admin restart

Sunday, March 18, 2012

Demo, Trial, POC, Pilot, Project - What's the difference?

Frequently I'll be asked to help a customer with a Proof of Concept or POC.  Almost always, the requester (be it a sales executive or a customer) actually wants something else and we spend some time trying to figure out what that is.  I'll define each of the terms from the title of this post which I hope will be helpful.  These terms may seem self-explanatory but I have seen a lot of time wasted when two parties do not have the same understanding.

Demo - a demonstration, or demo, for short, is a brief overview of the product's features and capabilities followed by a live presentation of the product going through various tasks that provide the customer with a practical understanding of how the product works.  A demo typically is the first time the customer has seen the product in action.  Demos usually take no more than 1-2 hours and can be customized to focus on features important to the customer.  Demos may also be recorded so that they can be viewed at the convenience of the customer but this does not provide an opportunity for questions or customization of the presentation.

Trial - A trial, sometimes called a live trial or evaluation, is an opportunity for the customer to implement the product (with or without assistance) in their own environment.  The objective is to give the customer the ability to have hands-on experience with the product and to validate the features and capabilities pitched by the manufacturer in the demo.  Trials typically have a duration of 30-60 days and in the case of hardware may involve an agreement to return the evaluation equipment or to purchase at the end of the trial.

Proof of Concept - Often referred to as a POC, this is a more formalized version of the trial and involves a more targeted and guided implementation of the product in the customer's environment.  The goal is to provide a high level of understanding of how the product may function for a specific use case and give the manufacturer an opportunity to more closely align the product's capabilities with the customer's needs.  As it is a more formal engagement, the customer and manufacturer should agree on the specific objectives and outcomes expected as well as the responsibilities of each party (i.e. the customer should devote resources such as staff and infrastructure and the manufacturer provide technical support and training).  A POC can vary in length, depending on the objectives agreed to at the outset but great care should be taken by the customer and the manufacturer to conclude the POC as agreed

Pilot - As the POC builds on the trial, likewise the pilot provides more focus and structure.  A pilot usually, but not always, is proceeded by some level of commitment by the customer and manufacturer through the purchase of a limited amount of the product suitable for testing against s sample of the proposed solution set.  The understanding is that the customer has made a decision to purchase the product pending the successful outcome of the pilot.  Unlike the previous stages (demo, trial and POC) the pilot is typically not a competitive engagement.  The pilot can also be used to measure the real benefits of implementing the product (versus those proposed by the manufacturer or perceived by the customer).  Data collected during the pilot can be used in developing and executing the implementation project.

Project - Typically done after a purchase has been made, the project is often a collaboration of the customer, manufacturer and any partners involved to fully implement the product in the customer's environment.  Projects have some of the same characteristics and methodologies employed in the POC and pilot engagements - desired outcome, agreement on responsibilities, defined start and end dates.  However, the project adds one more key element; SUCCESSFUL turnover of the project to an operational team.  Unlike the previous engagements, the project costs incurred by the manufacturer are typically paid by the customer as part of the overall solution purchase.

I welcome your comments on these definitions.  As you can see, these tend to build on each other and could be considered stages in making a purchasing decision.  Also, they can be combined (for example a demo and trial could be one engagement).  If nothing else it is always good to make sure that you and the other party agree on what is meant by each term at the outset.

Tuesday, March 13, 2012

Creating Custom Map Dashboards with vC Ops 5

Unless you were a user of the previous 1.x version of vCenter Operations Enterprise edition,  you've probably never seen the full dashboard which was boiled down to vSphere specific elements and standardized with the now familiar "3 badges" for health, workload and capacity.

As you're probably aware, with newly available vCenter Operations Management Suite 5.0 the vC Ops dashboard has been enhanced with a number of new badges and additional graphs and charts.  Lot's a good info and still very intuitive and easy to use.

Good news is that the custom UI is also available in vC Ops 5.0, giving you the ability to create dashboards for app owners, executive management team and the like.  You can even create a dashboard using Google Maps to show various physical locations with status indicators for various infrastructure resources like the one below.

You can access the custom UI by hitting


and logging in with your vC Ops admin credentials.  Feel free to look around, but to activate and use the GEO location feature to enable the maps on your dashboard, you will first need to get a Google Maps API key. To do this, you'll need a Google account so you can log into the Google APIs Console -

Once you log in, provided you've never used Google APIs console before, you will be presented with the image below - click the "Create project...." button and give your project a name.
From the list of available APIs you will want to enable the Google Maps API v2 shown in the image below.

You will have to read and agree to the Google EULA.  Please read and understand the Google EULA.  I am simply guiding you in how to set up this feature and any questions about entitlements for use should be directed to Google.  Note that you have a "courtesy limit" of 25,000 queries per day.  This should be ample for a lab or demo environment, but if you're going to use this for your production monitoring it would be a good idea to get pricing for activity above that limit.  From the Google APIs console, you can monitor the amount of queries against your API, if you're interested.

Once that has been done, you can now get your key from your project's API console by selecting the API Access link.  You only need the Simple API Access key for vC Ops.

Back in the vC Ops custom UI, select the Admin pull down menu item and choose Global Settings.  As shown below, you will enter your Google API key here - be sure to select Google as the map provider.  My key was obscured for this image, you will want to protect your key so that someone else doesn't use it and eat up all your queries!

Now you will need to add values for the GEO location resource tag.  This resource tag is created by default but is not populated with information.  Basically, this is where you will add map locations that you want to link to various resources (e.g. datacenters).

1. From the Environment pull down menu item choose "Environment Overview."
2. From the "List" tab view, click on the wrench icon to bring up the "Manage Resource Tags" pop-up.
3. Select "GEO Location" tag name and click the icon indicated below to add a "Tag Value"
4. Enter the name of your location
5. Click on the wrench icon to manage the location - this will pop up a Google map an allow you to search for and select a location.  Notice I've selected LSU's world famous Tiger Stadium, where Alabama will suffer a humiliating loss on November 3rd, 2012. :)

BE SURE to click on the blue hyperlink of the location you wish to activate that as your GEO location before clicking the "Save" button.

At this point, you just need to add some resources to your data center location.  While in the Environment Overview, simply drag any resources you want to associate onto the GEO Location tag you created.  In my case, I added several types of objects as you can see below.

You can now check your GEO Location on a map by switching to the Geographical tab.  As seen in the image below I have zoomed in for more detail on my location at Tiger Stadium.  Notice the boxes with the associated health scores for each resource associated with this location.

Finally, you can create your dashboard containing a map with your GEO Location tags in it.  From the Home page, click on the plus sign to the right of the last dashboard tab.

This is the easiest part, just scroll down through the template sets on the left side of the screen to find the template containing the GEO widget - you can't miss it, it's the only one with a map! :)

Drag the template to the right side and drop it onto the placeholder there.  You can change the name of the template if you wish.  Save the template and you're done!

Of course, you can play around with the dashboard and use another widget if you don't want the heatmap.  For example, I edited the dashboard to a single column so the map would take up the whole screen - much better if you have a lot of locations to view and are presenting this on a large screen at the NOC.

Reference the VMware vCenter Operations Manager Enterprise Administration Guide for vCenter Operations Manager Enterprise 5.0 for more details on configuring Resources.

Thursday, February 2, 2012

Find a VM without vCenter

This probably doesn't happen too often, but it did to a friend of mine - vCenter was down and he needed to find the ESX host where the supporting SQL server was registered so he could start it and troubleshoot if needed.

Putting aside why this happened and how it could have been prevented, we just needed to fix it quickly.  So, I put on my PowerCLI hat and pounded out the following....

$findvm = Read-Host -Prompt "Missing VM Name" 
Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -Confirm:$false 
$esxservers = get-content "c:\esxiplist.txt" 
Connect-VIServer -Server $esxservers 
$vmlist = (Get-VM | select -expandproperty name) 
foreach ($vm in $vmlist) 
    if ($vm -eq $findvm) 
        { Write-Host "$findvm was located on $esxserver" exit } 
Write-Host "$findvm was not located on any host"

The file esxiplist.txt contained the IP addresses for each host (one per line).  This could be made a lot prettier but for our purposes worked well enough to locate the SQL server plus another handful of critical VMs which were also down.  This script assumes that your login credentials are the same for each ESX host.

Monday, January 16, 2012

vSphere 5 Resource Pools Do Not Restrict Simultaneous vMotion

Something I encounter frequently in the field is the use of Resource Pools as nothing more than organizational containers.  This is not a good idea as this can have many implications on your VM utilization and performance.  As a best practice, you should use the folders feature in the "VMs and Templates" inventory view for organizing your VMs into logical groups.  Administrators typically do this because they default to the "Hosts and Clusters" inventory view.

Frank Dennemen, among others, has a really good explanation on the impact of using Resource Pools as organizational containers.  Frank goes on to explain that simultaneous vMotions are restricted under certain Resource Pool configurations, which is also documented in KB 1026102 but I noted that the KB is related to vSphere 4.x only.

Indeed, testing this out in a vSphere 4 lab I find that if I have a VM migrating from one server to another AND changing Resource Pools in the same cluster my vMotion migrations are serial (note one active and the other waiting).

However, if I perform another vMotion without changing Resource Pools, they are allowed to progress simultaneously.

So, does this limitation still apply with vSphere 5?  It would appear not, but I haven't found any official statement to that effect.  I did conduct an experiment with vSphere 5 in the lab and found that I was able to vMotion two VMs simultaneously under the conditions stated in the links above.

I created a three node cluster and three Resource Pools.  I then placed one VM each in two of the resource pools and on two of the ESXi hosts.

From there, I kicked off a vMotion of both VMs, selecting the third host AND designating the third Resource Pool as the target.  As you can see, the vMotion requests are both actively running and they eventually completed.

Other iterations mixing moves between two different Resource Pools as well as to a host external to the cluster all allowed simultaneous vMotions.

I still do not recommend using Resource Pools as folders in vSphere 5.  But at least you can now employ Resource Pools without concern over this particular limitation.

If you do use Resource Pools, you should leave unreserved CPU for overhead (10% for clusters and 30% for hosts).  See the vSphere 5 vMotion Performance Best Practices Guide for more info.

Friday, January 13, 2012

vCSA Network Error After Copy

I had been running my home lab's vCSA under Workstation while I played around with vShield 5 on my small, two server, ESXi cluster.  Mainly because I needed the resources free for all the vShield appliances and also because I didn't want to take the chance of isolating my vCenter while I was testing vShield out.

Anyway, when I got done with vShield and cleaned up the environment I decided to go ahead and move the vCSA back to the vSphere cluster.  I did this using a capability of Workstation 8 that allows you to upload a copy of a Workstation VM to a remote ESXi host.  All fun and games until I powered on my vCSA. 

Ruh-Roh Raggy! NO NETWORKING DETECTED!?  IP address is!

What I didn't realize is that the MAC address would change as a result of the copy.  I should have known - you know that little message that pops up when you manually copy or move VM files around and then power on said guests?

Well, what's happening is vSphere trying to prevent you from shooting yourself in the foot by having duplicate MAC addresses on your network.

In the case of my uploaded vCSA, I got no such message.  The MAC address of the vNIC was changed.  As a result, the vCSA registered the new MAC address as a new ethernet interface - eth1.  The problem is, vCSA expects to use eth0.

To fix this, I had two choices - I'll explain both, but start with the one I did first which is much easier but less "clean" and I'll tell you why I think so in my summary.

Fix "A" involves recording the MAC address associated with eth0 and then selecting manual addressing that to the vNIC of your vCSA.  Note the VM has to be powered off to make this change (the image below shows a powered on VM, thus the selection is unavailable but you get the point).

The VM will now boot up and the OS of the vCSA will be happy to see eth0 again and you're in business.  Oh, by the way I knew what MAC address I needed by looking at the properties of the vNIC on the source VM.

For fix "B" you'll need to login to the vCSA OS (which is SLES 11 for those interested) from the VM console and delete the /etc/udev/rules.d/70-persistent-net.rules file.

Now, this sounds pretty drastic but the file is actually auto generated by the Linux kernel device manager (udev).  What's happened to my vCSA upon changing vNIC MAC addresses is that the udev added another entry to the rules file as eth1.  As you can see from the output of the rules file below, eth0 is still there with the old MAC address.

So, I'll make a copy of the rules file (because I'm anal that way) and then delete it...

cp /etc/udev/rules.d/70-persistent-net.rules /etc/udev/rules.d/70-persistent-net.rule.backup
rm /etc/udev/rules.d/70-persistent-net.rules

Restart the guest OS and I'm back in business.

As I mentioned, fix "B" is the better way to go in my opinion.  Simply because you don't want to run the risk of having two nodes on your network with the same layer 2 address.  Plus, I just don't like manually setting things like MAC addresses unless there's a really good reason.

This fix would actually apply to any SLES virtual appliance that you may happen to copy.  Frankly, this is probably a rare problem but you never know - if it happened to me, it could happen to someone else!

Tuesday, January 10, 2012

Horizon Mobile - demos, links and feeds

Since the announcement at VMworld 2011, I have had some interest from my customer base around Horizon Mobile and set up a couple of calls with one of the Product Marketing Managers for Horizon Mobile.  She shared some great publicly available information as well as some feeds you may want to follow to stay up to date.

Also, at CES this week  LG demonstrated a 4G Andriod smart phone featuring VMware's Horizon Mobile technology.  It created a lot of interest.  Some CES coverage of the LG/VMware device here:

Increasingly enterprises want to enable a highly mobile workforce and at the same time offer a solution that is cost effective and combines the best of user choice and IT controls.  Horizon Mobile will help businesses create a BYOD/BYOM (Bring Your Own Device/Mobile) program that works for everyone.