The $500 Checkbox

It's been a long time since I've posted, but my desire to write has returned, and I've written down a number of projects to post about in the coming days and weeks.

Today's tale is a precautionary one, that hopefully can save you some grief (and money) down the road.

A couple of months ago, I was replacing my primary domain controller. By "primary", I mean that this was DC number 1. It's a physical box (as opposed to my other 4 virtual DCs), and it runs my DHCP scope, the domain's external NTP client, and holds all of the FSMO roles.

I've done this a few times and have a checklist, so I really wasn't that concerned. I had some issues when migrating DHCP to another 2008R2 DC. Old DHCP superscopes which had been removed somehow made their way back, and not all server options were transferred correctly. This wouldn't be apparent for a couple of days because, silly me, I trusted a Microsoft migration process. Next time I'll just rebuild the DHCP from scratch, thankyouverymuch.

My problem came when I was demoting DC1. I couldn't. I kept getting access denied messages, despite trying multiple logins and multiple group membership combinations (Domain Admins, Enterprise Admins, DNS Admins, Schema Admins, all of the above, some but not others). Furthermore, my DCDIAG's, which I run daily, had all been sparkling clean for some time. Finally, at the end of my rope and with my Google-Fu failing me, I decided to bite the bullet and call MS support (which I've found is really good, by the way). This was after hours, so it ended up costing about $500.

After poking around, support came to the conclusion that we should forcefully remove the problematic domain controller and do a manual cleanup of Active Directory. We still had problems altering the computer object, due to some funky permission issues. After changing them manually, this rectified the problem, but they, and I, still didn't know the actual root cause of the issue.

As often occurs, after stepping away from the problem the answer hit me like a lightning bolt. The next day while eating breakfast, I remembered:

A while ago I used Powershell to enable the "Protect object from accidental deletion" feature on every user, computer, and group object in my Active Directory. I did this to protect myself from doing something stupid, like pressing a sequence of keyboard shortcuts while my window focus was in the wrong place and screwing up AD.

I looked at my other domain controllers and they all had the "Protect object from accidental deletion" option enabled. You can see this if you open Active Directory Users and Computers, click the View menu and then "Advanced Features". Now open a computer or user and click on the "Object" tab.

Fast forward a couple of weeks later and it was time to rebuild another DC. I had the same issue with demoting the domain controller. I went into AD and unchecked this box, then let it sit for 15 minutes to let AD replicate. When I tried again, I had no issues!

Get Internet Explorer Versions from a List of Computers

If you haven't been following the tech news, last week Microsoft announced that they were revicing which versions of Internet Explorer that they would support on different Operating Systems. You can find the original article with a list of what's what here. Having just completed testing for IE10, I have a lot of work to do this next year, as the new policy goes into effect on 1/12/2016. I don't have many internal websites to worry about, but the Finance departments reliance on stupid banks that seem incapable of being current is making things difficult. I'm hoping that Internet Explorer Enterprise mode will help me out, but I haven't had the time to really delve into it yet.

I track my client computers' software inventories in Spiceworks, and it was easy enough to get a report from Spiceworks showing who had what, but I don't track my servers in Spiceworks. For that, I needed a script to go through a list of my servers and tell me what version of IE they were running. I found a couple of script online, but they weren't to my liking. Some gave me inaccurate info, even. But they did point me to which registry tree I should look at, so I made my own, and here it is:

#=====================BEGIN SCRIPT==========================
$Computers = Get-content "C:\Temp\QueryInternetExplorerVersionsComputerList.txt"
$TempFile = "C:\Temp\IEVersion.csv"
$Delimiter = ","

Foreach ($Computer in $Computers){
$Version = (Invoke-Command -computername $Computer {$reg = Get-Item ('HKLM:\Software\Microsoft\Internet Explorer'); $reg.GetValue("svcVersion")})
$Computer + $Delimiter + $Version + $Delimiter | add-content $Tempfile

}

#=====================END SCRIPT==========================

1. So, what we're doing here is as follows:
2. Get the list of computers from a text file
3. Specify a TempFile for output
4. Specify a delimiter
5. For each computer in the list, pull the SrcVersion value from the registry
6. Combine the Computername and SrcVersion, along with use of the delimiters to make a CSV file, which I can then import into Excel.
7. .....
8. Profit!

Good luck and Godspeed, my fellow admins!

Windows Filtering Platform events in Security Log ...OR... Don't Screw with Advanced Audit Policy Configuration

I have 4 domain controllers, and the are set up to forward events in their Security Logs to a Kiwi Syslog Server (Solarwinds). I have a 30 day rotation, and I monitor the space usage in a dedicated pie chart on my heads-up display. This past week it started getting lower. After poking around, I noticed that the file sizes of each days' log files were getting bigger and bigger. I opened one up in the Kiwi log viewer (which is the most horrible part of this software, by the way), and noticed a TON of messages from "Windows Filtering Platform". These were basically noting success every time an event was sent to the syslog server. Well..... thanks I guess?

I rarely have to deal with auditing, so I consult the Google and find instructions for how to stop success auditing for Windows Filtering Platform Events. It seemed straight forward enough: Open you default domain controller group policy, and drill down into the Advanced Audit Policy Configuration, and there are two options there dedicated to it. I enable the auditing policy and check neither success or deny, so that the policy will be in effect and tell Windows auditing not to monitor either of them. I document my changes in the ticket I made for myself and close it out.

A few minutes later I get an email from our Netwrix AD auditing software telling me that auditing isn't configured correctly. I go into Active Directory Users and Computers, add a period to the end of a computer account's description, and run a new report. The change is noted, but there are big red warning letters telling me something with auditing isn't right. Huh. I decide to see what the overnight report says, make another benign change in ADUC, and head out for the day.

The next morning, the report comes in showing changes in AD, but it still has big red warning letters saying that auditing isn't configured correctly. I go to check my syslog file size, and it's 5KB (down from 4GB), and the only thing in there are messages over and over saying that the audit settings have been changed.

So, back to Google, and APPARENTLY, when you use any part of the Advanced Audit Policy Configuration, it supercedes ALL of the normal auditing settings. So, by simply turning off logging for the Windows Filtering Platform, I had negated all settings in the regular auditing settings. Super.

Now, I don't know if you've ever had to reverse a group policy setting, but it is not intuitive. Simply turning the setting off does not reverse what has been done. A group policy restore from backup cannot reverse what you have done; you actually need to reverse the setting. Likewise, clearing the checkboxes in the advanced auditing section would not restore auditing to the way it used to be. Sadly.

Here is what I had to do to reverse this calamity:

1. Put the GPO back to the way it was.
2. Get on a domain controller.
3. Find out what you Group Policy Object's policyID is. I used get-gpo -name "<name>".
4. Now, go to C:\Windows\SYSVOL\Domain\Policies\<PolicyID>\Machine\Microsoft\Windows NT
5. In this folder delete the Audit folder. 
6. Now get on a command shell, and type the following: auditpol /get /category:*
7. What you see is that nothing is being audited.
8. Run a gpupdate /force, then run the command again - SUCCESS!!!

While researching this I ran across the command that I should have used to get rid of those Windows Filtering Platform events, which is this:
auditpol /set /subcategory:"Filtering Platform Connection" /Success:disable

You can check the setting before and after you run that command with this command:
auditpol /get /subcategory:"Filtering Platform Connection"

Starting a Backup Exec job from Powershell (After Veeam jobs finish)

When my weekly Veeam jobs finish, I have a post-job script that triggers a Backup Exec job that writes the Veeam backup files to tape. Yes, Veeam has built-in tape writing abilities now, but I still have physical servers to care and feed, and therefore Backup Exec still handles my tapes. Having just upgraded from Backup Exec 2010 to Backup Exec 2014, I learned that bemcli.exe had been removed. This was the old, atiquated way of running commands from the CLI to control Backup Exec. Smartly, they have replaced that with a Powershell module.

At a high level, here are the commands that your script needs to start a Backup Exec job:
#This command loads the module
Import-Module "C:\program files\symantec\backup exec\modules\bemcli\bemcli"

#This command starts the job, with inline confirmation
Start-bejob -inputobject "YourBackupExecJobName" -confirm:$false

For your entertainment, I'll now paste in my entire post job script, where I have built in a couple of handy features. Namely, an email telling me that whether the tape job has started or not, and an if statement that only starts the tape job if there are no failed backup jobs. If there's a failure, this gives me an opportunity to fix it before starting the tape-writing job.

Further commentary within this Veeam post-job script:

#-------------------------BEGIN SCRIPT------------------------
#Add the Veeam snap-in and the Backup Exec module
Add-PSSnapin VeeamPSSnapin
Import-Module "C:\program files\symantec\backup exec\modules\bemcli\bemcli"

#Set the initial counter, used to count Veeam job failures, used later in the IF statement
$Result = 0

#Email variables
$To = "me@contoso.com"
$From = "helpdesk@contoso.com"
$SMTPServer = "mailserver.contoso.com"

#Get all of the weekly Veeam jobs. My weekly jobs all have a "Weekly" prefix
$Statuses = (get-vbrjob | where {$_.Name -like "Weekly*"})

#Go through each Veeam job and increment $Result if a job has an unsuccessful status
#I'm not counting warnings, which can be triggered when a VM has been resized and Change Block Tracking has been reset
Foreach ($Job in $Statuses){
If ($Job.GetLastResult() -notlike "Success" -and $Job.GetLastResult() -notlike "Warning"){
$Result++
} #End If
} #End Foreach

#If the $Result equals 0, all the jobs were successful. Start the tape copy and email me
If ($Result -eq 0){
        #The following line uses the old (Backup Exec 2010) method for starting the tape job
#start-process "C:\Program Files\Symantec\Backup Exec\bemcmd.exe" -ArgumentList '-o1 -j"MyTapeJob"'
        #This next line is the new way of calling the tape job in Backup Exec
start-bejob -inputobject "MyTapeJob" -confirm:$false
    $To = "me@contoso.com"
$Subject = "Veeam to Tape Job Started"
Send-Mailmessage -to $To -Subject $subject -From $From -body $body -smtpserver $SMTPServer
} #End If

#If the result does not equal 0, email me
If ($Result -gt 0){
$Body = "The Veeam to Tape job has NOT started due to errors"
$Subject = "VEEAM BACKUP ERRORS: Veeam to Tape Job NOT Started"
Send-Mailmessage -to $To -Subject $subject -From $From -body $body -smtpserver $SMTPServer
} #End If

#-------------------------END SCRIPT------------------------