My 100th Post! -- Troubleshooting Change Block Tracking in Veeam

I never thought I'd make it to 100 posts, but here I am! My initial goal was  post per day, but I soon realized that was unrealistic. I have a family, a job, and a life after all. So I'm proud that I've created 100 nuggets of information that can help people. In the end, that's what IT is here to do: help people.

Change Block Tracking (CBT, and I don't mean Nuggets) is a VMware feature that Veeam uses extensively. It's what lets Veeam decide which block have changed since the last backup, so it can skip everything that HASN'T changed. Using it speeds up your backups a lot. The first time you back up a VM, Veeam has to go through every block of data, but thereafter it uses CBT to be selective.

Once in a while, one of my Veeam backups will report a warning, and I'll see error messages for a specific VM like these:

10/22/2013 11:27:26 PM :: Disk [VMFS-VOL3] VMNumber1/VMNumber1_1.vmdk size changed. Changed block tracking is disabled.

This was caused by expanding a drive on a Windows Server 2008 R2 server. You can do this while the VM is running, which is really handy, but you should reset the CBT on the VM before backups run. I'll outline that process here in a bit.

Another error you might see is this:
10/21/2013 11:16:08 PM :: Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/5122935d-291050d2-ee9c-ac162d75bf50/VMNumber1/VMNumber1.vmdkDetail: '', endpoint: ''

This was caused by using svMotion to migrate a VM to another datastore, but it can be caused by other things, like using vConverter on a VM. I've seen CBT break for no reason at all, but that's been few and far between.

To fix this issue, perform the following steps to reset CBT. It will require two shutdowns, so get a maintenance window if you need one.

Short version/process outline:
1. Shut down the VM
2. Disable CBT
3. Delete the -CTK files from the filesystem
4. Power on the VM
5. Shut Down the VM
6. Enable CBT
7. Power on the VM

The step-by-step version:
1. Shut down the VM
2. In vCenter, edit the settings of the VM
3. Click on the "Options" tab
4. Click on "General"
5. Click "Configuration Parameters"
6. Click on the "Name" column header twice to sort alphabetically
7. set ctkEnabled to FALSE
8. Scroll down and find each virtual hard drive listing. They look like this: scsix:x.ctkEnabled and set these to FALSE as well. YOU MUST DO THIS FOR ALL HARD DISKS. JUST LOOK FOR THE "CTK" IN THE NAME
9. Click OK as many times as it takes to get back out.
10. Ensure the VM is selected and that you are viewing its "Summary" tab
11. On the right-hand side is a box called "Resource", and within it there is storage listed.
12. Right-click on the listed storage (for example, VMFS-VOL17) and select "Browse Datastore"
13. Look for a folder with the same name as the VM
14. Go into that folder
15. For each virtual disk and snapshot disk there is a .ctkfile. For example:

vmname.vmdk
vmname-flat.vmdk
vmname-ctk.vmdk
vmname-000001.vmdk
vmname-000001-delta.vmdk
vmname-000001-ctk.vmdk

Delete ALL of the -CTK files (make sure you are choosing wisely!)

16. Now, boot the VM, and wait until VMware Tools has loaded. The easiest way to accomplish this is to open the VM's console, then use the menu (VM->Power->Shut Down Guest) to shut the VM down. If that option is greyed out, then VMware Tools is not running. Just try again in a few seconds.
17. Shut it down
18. Now, we will turn CBT back on
19. In vCenter, edit the settings of the VM
20. Click on the "Options" tab
21. Click on "General"
22. Click "Configuration Parameters"
23. Click on the "Name" column header twice to sort alphabetically
24. set ctkEnabled to TRUE
25. Scroll down and find each virtual hard drive listing. They look like this: scsix:x.ctkEnabled and set these to TRUE as well. YOU MUST DO THIS FOR ALL HARD DISKS. JUST LOOK FOR THE "CTK" IN THE NAME
26. Click OK as many times as it takes to get back out.
27. Power the VM on, and you're done.