When you resize the wrong hard disk on a vm

I’m running WSUS on a 2012R2 server and the drive where the updates are tends to grow a lot. On this VM, I increased the provisioned size of the c:\ drive and not the  drive where the updates reside. I didn’t notice I’d touched the wrong disk until I’d rescanned the disks on the VM and saw what I’d done.

260gbdrive

 

As panic set in, I changed the provisioned size back to 60GB (I know, I know) and well, you know the rest. This article saved me today. I didn’t want to restore the VM unless I really had no choice, this did the trick.

wrongdisk

Power off the VM and identify the host & datastore that the VM resided on, I enabled SSH and browsed to the VM location.

Cat out the descriptor file  (vmname.vmdk) that outlines the data layout on the virtual disk. In the Extend Description section, you’ll see the extent(s) info. Next to RW (if it’s read/write) a number. That number represents the size of the vmdk in sectors, with the additional 200GB included.

extentsize

To change the sector size to reflect the value I’m shrinking the drive back down to, 60GB, calculate it:

60GB = 60 * 1024 * 1024 * 1024 / 512 = 125829120 sectors

Edit the descriptor file and change the sectors to the new number, save and exit.

newsectorsize.png

Come out of the directory where the VM resides and VMotion  it to another datastore. Review the settings of the vm and take note of the new and improved disk size.

newsize.png

Power on the VM and review the size in Disk Management.

diskmgnt

Now, grow the right disk and try not to ever do that again.

 

Advertisements

Deprecated VMFS volumes found on the host Warning on ESXi after adding a datastore with Dell VSM

I’m running ESXi version 6.1 U1 and Dell Virtual Storage Manager  (VSM)version 4.5.2.974. I added a datastore to  the cluster using VSM and on 2 of my hosts I got the following alert:

vsmversion

deprecatedvmfs

Enables SSH on the host and check out the logs.

In /var/log/hostd.log , I found the error:

 

 warning hostd[xXxXxX] [Originator@6876 sub=Hostsvc.DatastoreSystem opID=123456-789-abc-def user=vpxuser] UpdateConfigIssues: Deprecated VMFS filesystems detected. These volumes should be upgraded to the latest version.

When I created the datastore, I did select VMFS 5, so I wasn’t sure why this error appeared.

selectedvmfs5

According to vmware KB2109735, this is a known issue in version 6.0 and there is no resolution. Just restart the management services and the message goes away.

 

#  /etc/init.d/hostd restart
#  /etc/init.d/vpxa restart

Then the message goes away.

 

*note*

This only happens to me when I create a datastore with VSM. If I present a disk from my array, set up the iSCSI connections and rescan, I don’t get this message. Your mileage may vary.

VDP backup fails due to error consolidating disks on VM

We are running VDP 5.8 on vSphere 5.5. One of the appliances failed to backup a VM last night. Checking the VM in the client, there is a message to consolidate the disks. I attempt to consolidate by right-clicking on the VM > Snapshot > Consolidate and it failed with this error:
locked

I checked the vmware.log file for the VM and saw this error:
2015-09-15T03:00:29.146Z| vcpu-0| I120: DISK: Failed to open disk for consolidate '/vmfs/volumes/5490a09a-eb6682ee-affb-bc305bef5520/mobxxxxxx/mobxxxxxx-000002.vmdk' : Failed to lock the file (16392) 8162

I check the settings on the VDP appliance and notice that the VMDK that belongs to the vm is attached.

extra-vdp-disk

Shut down the VDP appliance and remove the disk. Making sure to only remove from virtual machine and NOT remove from virtual machine and delete files from disk.

remove-extra-vdp-disk

Run the consolidation and if it succeeds, re-run the VDP backup on out of date sources to remediate your failed backup.

BSOD on a Windows VM bugcheck code 24

A VM that is running Windows 2008 R2 has experienced a BSOD. The application owner notified me and sent me some useful information.

Problem signature:
Problem Event Name: BlueScreen
OS Version: 6.1.7601.2.1.0.400.8
Locale ID: 1033
Additional information about the problem:
BCCode: 24
BCP1: 00000000001904FB
BCP2: FFFFF88006C34E28
BCP3: FFFFF88006C34680
BCP4: FFFFF880014E81B2
OS Version: 6_1_7601
Service Pack: 1_0
Product: 400_3
Files that help describe the problem:
C:\Windows\Minidump50415-20202-01.dmp
C:\Users\servername\AppData\Local\Temp\WER-42663418-0.sysdata.xml
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt

 Good info here, but with all BSODs, you should look at the dump file for more definitive information.

I grabbed the dump file, and downloaded Windows SDK. During the install, I unchecked all the boxes except the Debugging Tools for Windows and installed.

Features to Install

Launch WinDGB (x64) as administrator. The first step is to configure the symbol file path. It can be any folder on your server, just create an empty one and go to File > Symbol File Path.

In the symbol search path, I entered the following and clicked OK.

SRV*c:\tmp\symbols*http://msdl.microsoft.com/download/symbols

Symbol Search Path

Save the workspace. File > Save Workspace

Opened the dump file. File > Open Crash Dump.

Open Dump File

Before you proceed, read the info under the bugcheck analysis. It tells you what probably caused the error.

NTFS.sys

Click the link in the Bugcheck Analysis section, !analyze -y to get more information.

Read through the info. Note the following:

  • Process Name
  • Bugcheck_str
  • Image Name
  • Area right above the debugging details

At the very top you’ll see a message and some arguments.

Analyze

At the kd> prompt at the bottom of the debugger, enter

.bugcheck

Bugcheck command

In the analysis you see the Arguments. Do a .cxr on the 3rd argument as it suggests.

arguments

.cxr fffff88004523680

CXR command

CXR output

Next,

Run the kb command for a more informative stack trace.

Run KB command

There are now a few pieces of information to hone in on.

  1. The bug check code
  2. The stack track text

Look up the bugcheck code here on the msdn bug check code reference page.

https://msdn.microsoft.com/en-us/library/windows/hardware/hh994433%28v=vs.85%29.aspx

Google the stack trace text and this vmware KB article # 2115997 comes up.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2115997

This appears to be a known issue with a few versions of vmware tools:

I looked in the vmware.log file for the entries mentioned and they matched.

vmware-logs

The image name on the dump file referenced vds.exe and this VM is running tools version 9.4.11. This issue only appeared since the tools upgrade when we upgraded to 5.5 Patch 4

To confirm what version of tools you’re running, go to the VM and in the notifications bar, right click on the icon and select About  VMware Tools

VMware Tools version

Time to downgrade tools.

Find the tools ISO that will have a version that is not listed in the KB, but one that is compatible with your version of vsphere.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1036810

https://packages.vmware.com/tools/versions

Search for your current version and find the one closest to it that is not affected with this issue.

Version of tools to downgrade to

You’ll have to uninstall the current version of tools before installing older version. It will require a reboot.

The hardware version will not be an issue with this versions of tools, but confirm that for your specific environment.

Coming down off my VMworld high

This is my 3rd year attending VMworld and each time I walk away with a renewed focus and energy for my virtual infrastructure and what I can do to better manage and improve it. As I’m writing this post, I’m scrolling through my tweets and notes from VMworld and what stands out is that for my organization, VMware has the tools an SMB needs to deploy, monitor, upgrade, backup, administer, BD/DR, and grow its virtual infrastructure with ease.

My schedule was filled with sessions that covered just the things I needed to hear and to use day-one when I’m back in the office. Using the schedule builder as soon as it was available gave me time to make smarter choices and decide where my time should be spent in a more efficient manner. The VMworld app was a life saver! I knew right where I was suppose to be with notification turned on. The social media section keep me abreast of all the #vmworld tweets easily.

Here are my 3 favorite sessions. The sessions that made me want to roll up my sleeves the minute I had a WiFi connection to VPN back into the office and ‘test’ something out.

Here are my top 3 all-around favs from VMworld 2014:

  • The Solutions Exchange Welcome Reception
    • I got to spend a lot of time looking at what vendors are bringing to the virtualization space. Software, products and services that are moving VMware forward as the leader.
    • The Games at the CDW booth were more fun than I expected to have on opening day. The trivia questions were thought provoking and the ‘game show host’ and his assistant really kept us laughing.
    • Free beer & wine.
  • Winning a scooter (razor type) from Ravello and terrorizing the Moscone Center.
  • Taking a selfie with my SRM mentor, Mike Laverick of VMware.
Me and Mike Laverick of VMware

Me and Mike Laverick of VMware

Here are a few things I would change for next year:

  • The volume of the music at the Monday General Session was deafening. I really didn’t understand why the AV people didn’t rectify that by the end of the keynote. I promised myself I wouldn’t listen to another GS in the Hall D. The hang space was just fine.
  • I was in just ONE session with a woman speaker and I think she was a vendor. ‘Nuff said.
  • I know they’re trying to accommodate those with dietary restrictions, but vegetarians do need protein too.

Now that I’m back in the office, I’m ready to, upgrade to vSphere 5.5, tweak SRM  and  test out the vSphere 6.0 beta. I can’t talk about it (NDA), but you can find out more about it here.

Looking forward to next year.

 

Updating Java to 1.7 Breaks Dell EqualLogic VSM

I loathe updating Java because you never know what it will break. This morning, it was the Dell EqualLogic VSM plug-in for vCenter.

Upon launching the plug-in, I got these errors:

Image

Image

I thought to disable the current version and enable the previous version, but there is a way to add this URL as an exception.

Launch the Java control panel (I’m running 2008 R2) and click on the security tab.

Click the edit Site List button:

Image

Add the URL in the above mentioned error, by clicking Add, then typing in the URL.I also added the IP to vcenter as well. Click OK.

Image

Image

One more warning about the HTTP location you’ve entered. Just click continue. Apply the changes and close the java applet. Be sure to close the vCenter client as well.

Launch the client and log into VSM as usual.

Update- info on Windows 7

In Windows 7, there is no exception list under the security tab. To add your site to an exception list, you can create an exception.sites file and add the URL.

win7-java-securitytab

Browse to this location: C:\Users\username\AppData\LocalLow\Sun\Java\Deployment\security

If  there is a file named exception.sites, add the URL to that file. One URL per line.

If not, you can create it and add your URL.

exception-site

 

The command has timed out as the remote server is taking too long

I’m running vSphere 5.1 and I can’t launch the vSphere Client. I am able to launch the web client just fine. Upon launching the fat client, I get an error:

vsphere client could not connect

We updated our domain from 2003 to 2008 R2 and replaced all old domain controllers  (DCs). I’d changed the primary and secondary server URL (there was no port number previously) during the cut over and all has been working well for the past 26 days and today, nothing.

I logged into the web client as admin@system-domain to confirm I was pointing to the new domain controllers and I was. So now I’m really stumped. KB article 203918 solved the riddle.

Web client login as admin

To view the identity sources:  Administration > Sign-On and Discovery > Configuration. Here you can view the identity sources. Click the pencil to edit and see more detail.

Edit Identity Sources

Here are my current settings:

Current Server URL

I added the port number to both the primary and secondary server URLs and tested the connection. Click OK to close the dialog box.

Updated URL with Port Number

test-success

I was then able to launch the fat client without error.

Don’t forget to log out of the web client as admin@system-domain.

It’s safe to assume when I installed 5.1, I didn’t enter the port number as suggested in the Configuring vCenter Single Sign on doc and it worked just fine. Upon upgrading, the port became a required setting.