Month: March 2014

Vcenter Update manager error: "The host returns esxupdate error code:99"

/ johannstander / Leave a comment

Just out of blue started receiving the following error after trying to scan my ESXi hosts with baseline group:

“The host returns esxup error code:99. An unhandled exception was encountered. Check the update manager log files and esxupdate log files for more details”

well the /var/log/esxupdate.log on ESXi host showed the following error, but not much else:
Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file

Resolution:

Tracked down the KB: 2043170 which allows for the update manager database to be rebuild without having to reinstall update manager:  Please be aware that you will loose all your settings, bundles, drives, extensions, etc! 

  1. Backup the update manager database
  2. Stop update manager service
  3. open command prompt on server where update manager is installed
  4. Go to update manager installation folder
  5. run the following command:
    1. “vciInstallUtils.exe -O dbcreate -C . -L .”  (Both .  should be used) 
  6. After completion restart the update manager service
  7. Login to vcenter server 
  8. Within update manager, within configuration tab perform the “download now” task.
  9. Re-import any offline bundles, async drivers, third party extensions, and ESXi upgrade bundles. 
Link:

Powercli – Create alarm actions triggers for existing alarm definitions

/ johannstander / Leave a comment

Here is a script i created to add the following actions for triggers on alarms:
– Send a notification email from specified alarm state and email address
– Send a notification trap from specified alarm state and email address

I recommend just testing with a single Alarm definition in CSV file to make sure it works correctly for you. Only run this once otherwise you will create duplicate triggers.

Connect-VIServer
$alarmlist = Import-CSV C:\vcenter-alarm.csv

foreach ($item in $alarmlist) {

$alarmname = $item.alarmname
$alarmstartstatus = $item.alarmstartstatus
$alarmendstatus = $item.alarmendstatus
$alarmemail = $item.alarmemail

Get-AlarmDefinition -Name $alarmname | New-AlarmAction -Email -To $alarmemail | New-AlarmActionTrigger -StartStatus $alarmstartstatus -EndStatus $alarmendstatus
Get-AlarmDefinition -Name $alarmname | New-AlarmAction -Snmp | New-AlarmActionTrigger -StartStatus $alarmstartstatus -EndStatus $alarmendstatus
}

CSV file has the following columns: (no quotes necessary for text)

alarmname alarmstartstatus alarmendstatus alarmemail

Disclaimer:
Please use this script at your own risk and test it out in your test lab first before using it in production.

How to retrieve a VM log file through vSphere Web client

/ johannstander / Leave a comment

In the web client there is an easier way instead or using outside methods for instance:
SSH into host and upload log files to ftp site
GUI interface apps like WinSCP.

These normally have issues with copying of files due to file being locked while VM is powered on.

Firstly need to note the folder file name for VM:

  1. Select VM
  2. Select Summary tab
  3. On VM hardware window and select “settings”
  4. Expand the Hard disks and note the Disk File name provided.

To retrieve the log files following these instructions:

  1. Select VM
  2. Select Related Objects tab
  3. Select Datastores
  4. Right click on databastore and select “Browse files”
  5. Double click the folder associated to VM which was retrieved in beginning.
  6. Highlight each vmware*.log file individually
  7. Right click and select “Download from datastore”
  8. Select local folder location
  9. done!
If your server is unresponsive or crashed you can also export all the log files.  However beware this option will suspend or crash your VM so please use with caution!(KB 1007802)
  1. Select VM
  2. Right click VM and select “all vCenter Actions”
  3. Select “export system logs”
  4. Select system logs to export
  5. Select finish
  6. done

vSphere web client login error: "Client is not authenticated to VMware Inventory Service"

/ johannstander / Leave a comment

This week received the following error when trying to login to vSphere web client.
“Client is not authenticated to VMware Inventory Service”

Troubleshooting:

Was able to login with local user “administrator@vsphere.local”.
Problem seems to be only related to domain user authentication.

Looked at all services on vcenter server, which was running.
Service health on vSphere web client also all green without any errors.

Solution:

Finally looked at VMware KB articles and found the following:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2059637

Group names with permissions seems to be case sensitive.
Our domain admin has change the security groups cases and we were not notifiied which cause to the authentication to fail.

Resolved by removing and re-adding the active directory security groups back under permissions for the objects it was registered on.

I would recommend notifying your Domain administrators regarding this issue.

Commvault – Backup Copy of snapshot for Exchange 2010

/ johannstander / Leave a comment
Been running into an interesting scenario with a backup process which I hope will help someone else.
I have an exchange backup of DAG with intellisnap which creates a snapshot of each of the exchange database volume on Netapp.  The snapshot is then mounted and copied via SAN transport on the backend fiber connection to the storage library on commvault agent.
This has a storage policy to perform a backup copy of the latest snap via SAN transport weekly on Thursdays.
With commvault I find that sometimes the snapshots are not copied to the commvault storage library, without any type of error.
In viewing the jobs for the primary copy of snaphost i just shows a status of “to be copied”, but no failure.
This however causes problems since if you do not pay attention you can sit with weeks of uncopied backup copies of the snapshots, and if you snapshots are set for retention of a few days then you loose all those copies and have to mark them bad! (not good!)
Have not find a way to resolve this, even with the help of Commvault support, but here are a few tips to make sure you are at least inform in more detail regarding this process:
  • A good indication that the backup copy is not taking place is when the snapshots on netapp volume is not being deleted.
  • Also make sure you setup a very detailed report for the day after the snapshot copy was to take place and make sure the backup copy detailed information is also included in this report.

If however you do run into this problem, and your snapshots have already been rotated on the volumes on storage you would have to perform the following tasks to continue the backup copy process.
(I recommend contacting Commvault support if you are not familiar with your storage, snapshots and intellisnap, the steps below are provided for use at your own risk, you will loose backup data with these steps)
  • Verify you oldest snapshot date which are still on volumes for databases
  • Select storage policy
  • Right click and primary copy and select view -> jobs
  • this will list all the backup copies and show the ones with status “to be copied”.
  • Select all jobs up to before the date you have for snapshots on Netapp volume.
  • Right click and select “Mark job bad” (i know not my favorite step)
  • Right click on the storage policy again and select All Tasks -> Run Backup copy.
  • This will start the backup copy of your latest snapshot which resides on backup storage volume
Another solution is to manually add snapshots taken during the week to be added for backup copies.

  • Right click on the storage policy again and select View -> Jobs
  • Right click on the snap you want to create backup copy off.
  • Select “Pick for backup copy”
  • As a note, you cannot create a backup copy of snapshot which is older than your latest backup copy. so make sure you backup your oldest snapshot first . 
Anybody else running into the same problem would love to hear from you.

Dell DRAC and Internet Explorer 11

/ johannstander / Leave a comment
I don’t think i have ever had more problems that i have had with my IE browser and Dell DRAC interface 🙂
Seems that there is always something broken and can either never get logged in, or show me the turning circle of death or just a blank white screen.
I am still a big IE browser user, yes yes I know, mainly due to compatibility but I think it is getting worse with newer versions and the whole implementation of their compatibility view is just crazy. 
Already downgraded from IE 11 back to IE 10, hence why i am writing this email since for the life of me I could not get the DELL DRACs to work. Alas I am back on IE 11 and have this working..
Problems:
If you cannot login and you know you typing the correct username and password:
Delete your internet browser history, trust me for some reason this works.
If you see blank white screen:
Add the site address to trusted site and check your activeX settings in IE
Add site address to the compatibility view list in IE
If you see the turning circle of death as I call it:
Make sure that server address as added to your Java control panel.
Open control panel
Select security tab
Set security level to appropriate level
Add the site address at bottom to exception site list to allow running after security prompts.
IF you still get stuck my final recommendation would be the following:
  • Upgrade your DRAC to latest firmware version
  • In IE 11 press F12 to bring up the debug bar.  
    • Scroll down to Networks button and select
    • Click on the “Always refresh from server” button (3rd from left).