Month: January 2015

Cisco UCS error: will_boot_fault sensor failure asserted

/ johannstander / Leave a comment
After replacing a faulty UCS blade the following error presented itself after inserting the new blade into chassis:  will_boot_fault sensor failure asserted
Troubleshooting:
Ran the following –
Tried to run the board controller activation
# scope server
# scope boardcontroller
# show version
Showed version 13 (2.2.3d)
# show image
Did not display version 13 but latest was version 8 (2.2.1d)
# activate firmware .0 force
# commit-buffer
Received an error message that commit cannot proceed.
Solution:
The problem turned out to be the new blade had a newer version of firmware installed that what I had loaded in UCS manager. 
To fix this upload the latest firmware version ( in our case 2.2.3d).
Verify the new version is available by running show image under scope for chassis/server/boardcontroller.

  • Run the same process as listed above

vCenter Server – how to move your database to new server

/ johannstander / Leave a comment

This solution is for vCenter Server hosted on a Windows OS. I am just providing the short and sweet version since VMware KB 7960893 provides very detailed information on this process.

Solution:

  • Create new database on new server
  • Setup users on new database instance and provide security access with same permissions.
  • Shutdown all vcenter services
  • Shutdown all vcenter update manager services
  • Backup databases
  • Restore databases on new database server
  • On vcenter server, copy all the scripts for SQL Agent rollup jobs that need to be recreated in new database server instance
    • KB 1004382
  • Update DSN entries for vcenter server and venter update manager to reflect new database server location.
  • Start vCenter services
  
Note: In some instances, it is necessary to update the vcdb.properties file with the new SQL server if the SQL database name is referenced here. If this is not updated, the performance overview charts and the Storage Monitoring Service (SMS) may be affected after migrating the vCenter Server database. For more information, see VMware vCenter Storage Monitoring Service fails with the error: Service initialization failed (2016472). The vcdb.properties file is located at:
  • Windows 2008 – C:\ProgramData\VMware\VMware VirtualCenter
  • Other Windows versions – C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\
Links:



vCenter Log Insight – move NFS archive location

/ johannstander / Leave a comment
Had to recently change the vCenter log insight archive to a new NFS location.   Here is how you change this 
Solution:

SSH to appliance.
Stop the service

  • /etc/init.d/loginsight stop 
Copy the archive files over the new NFS location.

Change log insight configuration file to new NFS location

  • /storage/core/loginsight/config/ 
  • Vi loginsight-config.xml#
Start the service 
  • /etc/init.d/loginsight start

vCOPS – Recreate missing vSphere group dashboard

/ johannstander / Leave a comment
If you are familiar with the custom-ui of vCops you will find that since 5.7 you are providing with some default custom dashboards under the vsphere group.
  
We recently installed VNX analytics adapter which seems to have removed the vsphere group default dashboards. 
To fix this perform the following tasks:
SSH into UI VM
Browse to
cd /usr/lib/vmware-vcops/user/conf/dashboards/
Run the script  “import-dashboards.sh”
This will only share the dashboards with admin user
Edit the sh file at bottom where states:
    # import imported templates as dashboards for admin user
The xml files listed here are the custom dashboards.
These can also be copied out (for instance with WinSCP) and imported into a  manually created new dashboard.
Add lines for different users but firstly create duplicate of file for backup purposes
Cp import-dashboard.sh import-dashboard.sh.default
Vi /import-dashboards.sh
/usr/lib/vmware-vcops/user/conf/dashboards # ls -l
-rw-r–r– 1 admin admin  61794 Aug 26 04:56 DashboardTemplate-Alerts.xml
-rw-r–r– 1 admin admin 117798 Aug 26 04:56 DashboardTemplate-ClusterUtilization.xml
-rw-r–r– 1 admin admin  90979 Aug 26 04:56 DashboardTemplate-DatastorePerformance.xml
-rw-r–r– 1 admin admin  74401 Aug 26 04:56 DashboardTemplate-DatastoreSpace.xml
-rw-r–r– 1 admin admin 170642 Aug 26 04:56 DashboardTemplate-Heatmaps.xml
-rw-r–r– 1 admin admin  77214 Aug 26 04:56 DashboardTemplate-HostMemory.xml
-rw-r–r– 1 admin admin 117370 Aug 26 04:56 DashboardTemplate-HostUtilization.xml
-rw-r–r– 1 admin admin  76295 Aug 26 04:56 DashboardTemplate-Troubleshooting.xml
-rw-r–r– 1 admin admin 128042 Aug 26 04:56 DashboardTemplate-VMPerformance.xml
-rw-r–r– 1 admin admin 117626 Aug 26 04:56 DashboardTemplate-VMUtilization.xml
  # import imported templates as dashboards for all administrator users
    bash  ./dbcli.sh dashboard import group:Administrators $TD_1 –default true –set 1
    bash  ./dbcli.sh dashboard import group:Administrators $TD_2
    bash  ./dbcli.sh dashboard import group:Administrators $TD_3
    bash  ./dbcli.sh dashboard import group:Administrators $TD_4
    bash  ./dbcli.sh dashboard import group:Administrators $TD_5
    bash  ./dbcli.sh dashboard import group:Administrators $TD_6
    bash  ./dbcli.sh dashboard import group:Administrators $TD_7
    bash  ./dbcli.sh dashboard import group:Administrators $TD_8
    bash  ./dbcli.sh dashboard import group:Administrators $TD_9

    bash  ./dbcli.sh dashboard import group:Administrators $TD_10
From vCOPS 5.7.3  release notes:
  • All and group options for dashboard commands
    For the dashboard import, dashboard delete, and dashboard reorder commands, you can use one of the following options instead of specifying a user name:
    * --all
    * --group:<group_name[,]
    The --all option applies the command to all user accounts. The --group:<group_names[,] option applies the command to all user accounts that belong to the specified user groups. For example, the command dashboard import group:Administrators,Operators "templates\MyDashboard.bin" imports the dashboard MyDashboard.bin to all user accounts that belong to the Administrators and Operators user groups.
    The dashboard import command also has a --set option that enables you to set the dashboard order. For example, the command dashboard import MyUser2 "dashboards/MyDashboard.bin" --set 1 imports the dashboard MyDashboard.bin and makes it the first dashboard in the dashboard list for the user MyUser2.

vCenter Server log file locations by services

/ johannstander / Leave a comment

Location of vCenter Server log files (KB 1021804)


vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003:    %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs\
vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008:    C:\ProgramData\VMware\VMware VirtualCenter\Logs\
vCenter Server Appliance 5.x:    /var/log/vmware/vpx/
vCenter Server Appliance 5.x    UI: /var/log/vmware/vami

vpxd.log: 

The main vCenter Server logs, consisting of all vSphere Client and WebServices connections, internal tasks and events, and communication with the vCenter Server Agent (vpxa) on managed ESX/ESXi hosts.
vpxd-profiler.log, profiler.log and scoreboard.log: 
Profiled metrics for operations performed in vCenter Server. Used by the VPX Operational Dashboard (VOD) accessible at https://VCHostnameOrIPAddress/vod/index.html.
vpxd-alert.log: 
Non-fatal information logged about the vpxd process.
cim-diag.log and vws.log: 
Common Information Model monitoring information, including communication between vCenter Server and managed hosts’ CIM interface.
drmdump\: 
Actions proposed and taken by VMware Distributed Resource Scheduler (DRS), grouped by the DRS-enabled cluster managed by vCenter Server. These logs are compressed.
ls.log: 
Health reports for the Licensing Services extension, connectivity logs to vCenter Server.
vimtool.log: 
Dump of string used during the installation of vCenter Server with hashed information for DNS, username and output for JDBC creation.
stats.log:
 Provides information about the historical performance data collection from the ESXi/ESX hosts
sms.log: 
Health reports for the Storage Monitoring Service extension, connectivity logs to vCenter Server, the vCenter Server database and the xDB for vCenter Inventory Service.
eam.log: Health reports for the ESX Agent Monitor extension, connectivity logs to vCenter Server.
catalina..log and localhost..log: 
Connectivity information and status of the VMware Webmanagement Services.
jointool.log: 
Health status of the VMwareVCMSDS service and individual ADAM database objects, internal tasks and events, and replication logs between linked-mode vCenter Servers.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021804


Location of vSphere Profile-Driven Storage log files (KB 2056646)


vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003:    %ALLUSERSPROFILE%\Application Data\VMware\Infrastructure\Profile-Driven Storage\Logs
vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008:    C:\ProgramData\VMware\Infrastructure\Profile-Driven Storage\Logs
vCenter Server 5.x Linux Virtual Appliance: /var/log/vmware/vpx/sps

sps.log:

The main Profile-Driven Storage logs, consisting of all vCenter Server and Management Webservices connections, internal tasks and events, and information about the storage profile integrity.
vim-sps-install.log: 
This file consists of information about the installation of Profile-Driven Storage including computer name, operating system revision, the date of installation and the number of revisions that have been installed or upgraded on the system.
wrapper.log:   
This file provides information about the state of the Java runtime environment.


For vSphere 5.0, all the logs for the vSphere Web Client service are located at:
   Windows: C:\Program Files\VMware\Infrastructure\vSphere Web Client\DMServer\serviceability\
   Linux: /usr/lib/vmware-vsphere-client/server/serviceability/
For vSphere 5.1, all the logs for the vSphere Web Client service are located at:
   Windows: C:\ProgramData\VMware\vSphere Web Client\serviceability\
   Linux: /var/log/vmware/vsphere-client/serviceability/
For vSphere 5.5, all the logs for the vSphere Web Client service are located at:
   Windows: C:\ProgramData\VMware\vSphere Web Client\serviceability\
   Linux: /var/log/vmware/vsphere-client/

logs\vsphere_client_virgo.log:

The main vSphere Web Client log, consisting of all communication between underlying Flex endpoint connections, internal tasks and events
logs\access\localhost_access_log..txt:
Individual HTTP requests from the Web browser
logs\virgo-serverlog.log:
Information about plug-in bundle communication
eventlogs/eventlog.log
Time stamps of startup and shutdown of the vSphere Web Client service

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004090

Location of vCenter Single Sign-On log files for vCenter Server 5.1 and 5.5 (KB 2033430)


vSphere 5.1 using vCenter Single Sign-On 5.1:
   vCenter Server 5.1.x on Windows Server 2003 – C:\Program Files\VMware\Infrastructure\SSOServer\logs\
vCenter Server 5.1.x on Windows Server 2008/2012 – C:\Program Files\VMware\Infrastructure\SSOServer\logs\
vCenter Server Virtual Appliance 5.1.x – /var/log/vmware/sso/
vSphere 5.5 using vCenter Single Sign-On 5.5:
   vCenter Server 5.5.x on 2003 – %ALLUSERSPROFILE%\Application Data\CIS\
vCenter Server 5.5.x and 2008, 2012 – C:\ProgramData\VMware\CIS\
vCenter Server Virtual Appliance 5.5.x – /var/log/vmware/sso/

vminst.log

 SSO installer log
vim-sso-msi.log 
MSI installer verbose logs for SSO installation
vim_ssoreg.log 
SSO Service log
exported_sso.properties 
Endpoint information about each of the SSO Solution Users and identity sources extracted from previous vCenter Single Sign On 5.1.0 instance
vim-openssl-msi.log 
MSI installer verbose log for OpenSSL installation
vim-python-msi.log 
MSI installer verbose log for Python installation
vim-kfw-msi.log 
MSI installer verbose log for MIT Kerberos installation
SSO logs grouped by component and purpose include:
logs\vmdird\vdcpromo.log
Promotion and demotion operation information for the SSO instance when joined or removed from a linked configurations
logs\vmdird\vdcsetupIdu.log
 VMware Directory Service setup post-installation log containing information about the localhost name
logs\vmdird\vmdir.log 
Health reports for the VMware Directory Service (VMDir) service and the VMDir database
logs\vmkdcd\vmkdcd.log 
Key Distribution Center (kdc) run-time log, reports ports conflicts preventing the service from starting
logs\vmware-sso\vmware-sts-idmd.log 
VMware Identity Management service run-time logs, time-stamped records of user attempts when accessing SSO for administrative purposes
logs\vmware-sso\vmware-sts.ldmd-perf.log 
VMware Identity Management service performance counter logs
logs\vmware-sso\VMwareIdentityMgmtService. date.log 
Commons Daemon log once the Identity Management Service has started
runtime\VMwareSTS\logs\ssoAdminserver.log 
SSO admin server
runtime\VMwareSTS\logs\lookupServer.log 
Lookup server log
runtime\VMwareSTS\logs\catalina. date.log 
Standard Tomcat logs
runtime\VMwareSTS\logs\vmware-identity-sts.log 
VMware Security Token Service service run-time logs, time-stamped records of issuing, validating, and renewal of security tokens.
runtime\VMwareSTS\logs\vmware-identity-sts.log 
VMware Security Token Service performance counter logs.

vCenter Server – verify your heap memory size configuration!

/ johannstander / Leave a comment

Lately been getting a lot of service status failures for the following vCenter Server services:

VMware vCenter Storage Monitoring service.
VMware vSphere Profile-driven Storage service

The monitoring service was the most frequent and non-impacting but now and then the profile-drive storage failed which caused large impact since users could not deploy or start vApps in vCloud director.

Turns out when we installed vCenter server for the first time we sized it for a small environment and current this is not the case anymore so with the heap sizes set to small for all services we started running into problems.

Troubleshooting:

The only way to resolve this temporarily was to restart the vCenter server.  Restart just the specific service did not always resolve the problem for us.

Looked in dslog files and found errors as follows:
C:\ProgramData\VMware\Infrastructure\Profile-Driven Storage\Logs\wrapper.log

Caused by: java.lang.OutOfMemoryError: Java heap space

Solution:

Update the vCenter Server heap memory sizes for specified services to reflect the current size of your environment.

This table provides the JVM heap size recommendations:

vCenter Server 5.5:

vCenter Server Inventory VMware VirtualCenter Management Webservices (tc Server) Inventory Service Profile-Driven Storage Service
Small inventory (1-100 hosts or 1-1000 virtual machines) 512 MB 3 GB 1 GB
Medium inventory (100-400 hosts or 1000-4000 virtual machines) 512 MB 6 GB 2 GB
Large inventory (More than 400 hosts or 4000 virtual machines) 1024 MB 12 GB 4 GB


Note:After adjusting the JVM heap size in the wrapper.conf file, VMware recommends restarting the VirtualCenter Management Web services. For more information, see Stopping, starting, or restarting vCenter services (1003895).

Links:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2021302

vCenter Operations Manager (vCops) errors and not collecting stats (/data out of space)

/ johannstander / Leave a comment

Recently I logged into vCenter Operations Manager (vCOPS) custom UI and found that all objects stopped collecting stats.

Troubleshooting: 

Logged into vCenter server and tried health status tab but received connection error.
In Custom UI select Environment tab -> Configuration -> Adapter instances.   Edit a vcenter instance and selected test. Received the following error

 “Test was not successful.  Connection refused to host: Secondvm-internal;nested exception is: java.net.ConnectionException: Connection refused”

I ran the same test on our VNX and Brocade adapter instances and received the exact same error.

In vCops Administration page under registration tab i selected the update bottom on vCenter Server registration and re-entered the password.  After a long while i received an exception error.
I went to status tab and saw that some of the vCenter Operations Manager Statuses was not running.
I tried to start the service but did not work and status did not change nor did i get an error message.

I SSH into the Analytics VM and tried to manually restart the service “vcops-admin restart”.  showed that the database was not coming online on which i though about looking at space.

Ran “df -h” and found that the /data drive was out of space.

Solution:

Shut down vCops vapp container from vCenter server.
Edit the VM and add new disk.
Power on the vapp container.

vCops will automatically extend the /data drive with the newly added disk.

Links:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2016022