Checking for VMware Snapshots

So besides the standard way of simply looking at vCenter to see snapshots, there are a couple of other ways to accomplish this. The first way is manual, and uses RVTools, which is a GREAT tool for gathering lots of information about your VMware environment and its virtual machines. I run it every Monday just to make sure thing look healthy. Veeam's monitoring software, VeeamONE will also alert you when a snapshot is active for longer than a specified time, which can be altered by editing the alarm. VeeamONE is free, but if you want more features you have to go full version which is not free, of course.

I recommend both of these free tools to anyone who wants a health checkup or wants to create some quick documentation of their VMware Environment. These tools helped me immensely when I inherited mine, so that I could hit the ground with good information and a list of things to fix first.

I should not leave out vCheck, which a VERY full featured script offerend by Virtu-Al.net that can give you a LOT of good info. As a matter of fact, I created the script at the end of this entry from one of the subscripts of the vCheck project; the one that checks for snapshots.

I created this script to run at 6AM and let me know if there are any snapshots running before the start of the business day. Reasons for these snapshots can be a lot of things, but any of them bear looking into more closely. I could have a Veeam Backup that is stuck, like happened to me last week. I, or another admin, might have left a snapshot running (which merits a flogging!).

Also, you'll notice that the first thing it does is retrieve your credentials, which I created beforehand using the technique outlined in this post.

Without further ado, here's the script:

#Gets the credentials to facilitate connection to the Vcenter Server
$password = Get-Content c:\PSCred\mycred.txt | ConvertTo-SecureString
$credential = New-Object System.Management.Automation.PsCredential "TaskSchedUser@Contoso.com",$password

#Gets some other miscellaneous variables for use in the email alert
$smtpServer = "mailServer.contoso.com"
$MailFrom = "helpdesk@contoso.com"
$MailTo = "IT@contoso.com"
$VISRV = "vCenterServer"

#Add the snapin for PowerCLI (The VMware Powershell module)
add-pssnapin Vmware.VimAutomation.Core

#Connect to your vCenter Server, using the credentials we created to authenticate
connect-viserver -server vCenterServer.contoso.com -credential $credential

#Function that finds the user that created the snapshot, so you can flog them
function Find-User ($username){
if ($username -ne $null)
{
$usr = (($username.split("\"))[1])
$root = [ADSI]""
$filter = ("(&(objectCategory=user)(samAccountName=$Usr))")
$ds = new-object system.DirectoryServices.DirectorySearcher($root,$filter)
$ds.PageSize = 1000
$ds.FindOne()
}
}

#Function that gets snapshot info
function Get-SnapshotTree{
param($tree, $target)

$found = $null
foreach($elem in $tree){
if($elem.Snapshot.Value -eq $target.Value){
$found = $elem
continue
}
}
if($found -eq $null -and $elem.ChildSnapshotList -ne $null){
$found = Get-SnapshotTree $elem.ChildSnapshotList $target
}

return $found
}

#Function that gets more detailed snapshot info
function Get-SnapshotExtra ($snap){
$guestName = $snap.VM # The name of the guest

$tasknumber = 999 # Windowsize of the Task collector

$taskMgr = Get-View TaskManager

# Create hash table. Each entry is a create snapshot task
$report = @{}

$filter = New-Object VMware.Vim.TaskFilterSpec
$filter.Time = New-Object VMware.Vim.TaskFilterSpecByTime
$filter.Time.beginTime = (($snap.Created).AddSeconds(-5))
$filter.Time.timeType = "startedTime"

$collectionImpl = Get-View ($taskMgr.CreateCollectorForTasks($filter))

$dummy = $collectionImpl.RewindCollector
$collection = $collectionImpl.ReadNextTasks($tasknumber)
while($collection -ne $null){
$collection | where {$_.DescriptionId -eq "VirtualMachine.createSnapshot" -and $_.State -eq "success" -and $_.EntityName -eq $guestName} | %{
$row = New-Object PsObject
$row | Add-Member -MemberType NoteProperty -Name User -Value $_.Reason.UserName
$vm = Get-View $_.Entity
$snapshot = Get-SnapshotTree $vm.Snapshot.RootSnapshotList $_.Result
$key = $_.EntityName + "&" + ($snapshot.CreateTime.ToString())
$report[$key] = $row
}
$collection = $collectionImpl.ReadNextTasks($tasknumber)
}
$collectionImpl.DestroyCollector()

# Get the guest's snapshots and add the user
$snapshotsExtra = $snap | % {
$key = $_.vm.Name + "&" + ($_.Created.ToString())
if($report.ContainsKey($key)){
$_ | Add-Member -MemberType NoteProperty -Name Creator -Value $report[$key].User
}
$_
}
$snapshotsExtra
}

#Function to send mail. I normally just use a one-liner, but I'm reusing code, and this was here.
Function SnapMail ($Mailto, $snapshot)
{
$msg = new-object Net.Mail.MailMessage
$smtp = new-object Net.Mail.SmtpClient($smtpServer)
$msg.From = $MailFrom
$msg.To.Add($Mailto)

$msg.Subject = "Snapshot Reminder"

$MailText = @"
There is a snapshot active on $($snapshot.VM) which was taken on $($snapshot.Created).

Name: $($snapshot.Name)

Description: $($snapshot.Description)
"@

$msg.Body = $MailText
$smtp.Send($msg)
}

#Cycles through any snapshots found, and send an email for each one
foreach ($snap in (Get-VM | Get-Snapshot)){
$SnapshotInfo = Get-SnapshotExtra $snap
SnapMail $mailto $SnapshotInfo
}

#Disconnect from the vCenter server
Disconnect-VIServer -Confirm:$false

Storing Credentials for Powershell Scripts

The next script I want to show you has a prerequisite of storing your credentials in a file for later use. I figured I'd better cover that first, so here goes.

These credentials are user speceific (duh) but they are also computer specific. Let's say I was to run a scheduled task that uses The 'ScheduledTaskUser' user, and that this task will run from 'TaskServer'.

In order to make this work, this process must be run on 'TaskServer'!

1. Create the securepassword string from the computer that's going to use it:
read-host -assecurestring | convertfrom-securestring | `
out-file c:\temp\ScheduledTask_Credentials.txt

2. When you press enter to execute the command above, you will be on a blank line. Enter the password and press enter.

3. The password is now saved on the computer as a secure string (not in plaintext) in the file that you specified.

4. In your script file, build the credential:

$password = get-content `
c:\temp\ScheduledTask_Credentials.txt | convertto-securestring

$credentials = new-object `
-typename System.Management.Automation.PSCredential `
-argumentlist "DomainName\ScheduledTaskUser",$password

5. Use the credential. You can pass $credentials anywhere you can use the -credential parameter, like so:

get-wmiobject win32_service `
-computer AnotherServer `
-credential $credentials

Aside:

You'll notice above that I used the backtick character ` to allow me to continue the script on the next line. I'm trying to do this more and more for script readability......

My 100th Post! -- Troubleshooting Change Block Tracking in Veeam

I never thought I'd make it to 100 posts, but here I am! My initial goal was  post per day, but I soon realized that was unrealistic. I have a family, a job, and a life after all. So I'm proud that I've created 100 nuggets of information that can help people. In the end, that's what IT is here to do: help people.

Change Block Tracking (CBT, and I don't mean Nuggets) is a VMware feature that Veeam uses extensively. It's what lets Veeam decide which block have changed since the last backup, so it can skip everything that HASN'T changed. Using it speeds up your backups a lot. The first time you back up a VM, Veeam has to go through every block of data, but thereafter it uses CBT to be selective.

Once in a while, one of my Veeam backups will report a warning, and I'll see error messages for a specific VM like these:

10/22/2013 11:27:26 PM :: Disk [VMFS-VOL3] VMNumber1/VMNumber1_1.vmdk size changed. Changed block tracking is disabled.

This was caused by expanding a drive on a Windows Server 2008 R2 server. You can do this while the VM is running, which is really handy, but you should reset the CBT on the VM before backups run. I'll outline that process here in a bit.

Another error you might see is this:
10/21/2013 11:16:08 PM :: Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/5122935d-291050d2-ee9c-ac162d75bf50/VMNumber1/VMNumber1.vmdkDetail: '', endpoint: ''

This was caused by using svMotion to migrate a VM to another datastore, but it can be caused by other things, like using vConverter on a VM. I've seen CBT break for no reason at all, but that's been few and far between.

To fix this issue, perform the following steps to reset CBT. It will require two shutdowns, so get a maintenance window if you need one.

Short version/process outline:
1. Shut down the VM
2. Disable CBT
3. Delete the -CTK files from the filesystem
4. Power on the VM
5. Shut Down the VM
6. Enable CBT
7. Power on the VM

The step-by-step version:
1. Shut down the VM
2. In vCenter, edit the settings of the VM
3. Click on the "Options" tab
4. Click on "General"
5. Click "Configuration Parameters"
6. Click on the "Name" column header twice to sort alphabetically
7. set ctkEnabled to FALSE
8. Scroll down and find each virtual hard drive listing. They look like this: scsix:x.ctkEnabled and set these to FALSE as well. YOU MUST DO THIS FOR ALL HARD DISKS. JUST LOOK FOR THE "CTK" IN THE NAME
9. Click OK as many times as it takes to get back out.
10. Ensure the VM is selected and that you are viewing its "Summary" tab
11. On the right-hand side is a box called "Resource", and within it there is storage listed.
12. Right-click on the listed storage (for example, VMFS-VOL17) and select "Browse Datastore"
13. Look for a folder with the same name as the VM
14. Go into that folder
15. For each virtual disk and snapshot disk there is a .ctkfile. For example:

vmname.vmdk
vmname-flat.vmdk
vmname-ctk.vmdk
vmname-000001.vmdk
vmname-000001-delta.vmdk
vmname-000001-ctk.vmdk

Delete ALL of the -CTK files (make sure you are choosing wisely!)

16. Now, boot the VM, and wait until VMware Tools has loaded. The easiest way to accomplish this is to open the VM's console, then use the menu (VM->Power->Shut Down Guest) to shut the VM down. If that option is greyed out, then VMware Tools is not running. Just try again in a few seconds.
17. Shut it down
18. Now, we will turn CBT back on
19. In vCenter, edit the settings of the VM
20. Click on the "Options" tab
21. Click on "General"
22. Click "Configuration Parameters"
23. Click on the "Name" column header twice to sort alphabetically
24. set ctkEnabled to TRUE
25. Scroll down and find each virtual hard drive listing. They look like this: scsix:x.ctkEnabled and set these to TRUE as well. YOU MUST DO THIS FOR ALL HARD DISKS. JUST LOOK FOR THE "CTK" IN THE NAME
26. Click OK as many times as it takes to get back out.
27. Power the VM on, and you're done.

VMware vCenter with a SQL Express back-end can get full!

If you have a smaller VMware infrastructure, you are probably running vCenter with its database on a SQL Express instance. This works fine, but eventually that database will fill up if you don't adjust the retention setting in vCenter, because SQL Express is limited to a maximum database size of 4GB.

This happened recently to a friend of mine, but in my haste to create a monitor in PRTG to monitor the size I neglected to get a screenshot of what that moment looks like. For that, dear reader, I apologize.

What I CAN tell you is how to find out how big yours is, and how to keep it from growing too large. The size of the database can be found by looking in C:\Program Files\Microsoft SQL Server\MSSQL10_50.VIM_SQLEXP\MSSQL\DATA\VIM_VCDB.mdf. The path may differ based on whether you have a 32 or 64-bit SQL installation, and what you named the file, but that's where mine is. That MDF file is your database (the LDF file with the same base name is your Transaction Log file, just FYI). So, you don't want this file to get near 4GB. I have a sensor in PRTG (my monitoring software) that keeps track of the file size, so I can look at trends or whatnot. It would also be fairly easy to throw something together using powershell to alert me if the file grows over a certain threshold.

What I would also do, however, is go into vCenter, click Home-->vCenter Server Settings and adjust the Database Retention Policy to something smaller. I believe the boxes are unchecked by default, so there is no retention policy, meaning that it saves everything forever. See this picture:

Your needs might be different - I have VeeamONE monitoring my stuff, so I elected for 30 days.