Click an Ad

If you find this blog helpful, please support me by clicking an ad!

Monday, March 31, 2014

Automated Backup of a MySQL Database

I have a couple MySQL databases out here in the wilderness. I use Veeam Backup and Replication to back up my virtual machines, but Veeam doesn't have any way to quiet MySQL database activity down so that it can get a proper backup, that I'm aware of. So, I got my hands dirty and created the following process to automatically back up MySQL databases.


  • The first thing that needs to be done is that a user needs to be created within the MySQL instance that can be used for backups. I accomplish this on the server in MySQL Workbench. Under Server Administration, click on Security. Add a user account, give it a nice password, and add it to the BackupAdmin role.
  • The MySQL server is not a domain-joined computer, so I created a local Windows account (MySQLWinBackup), and assigned the account the "Log on as batch" right.
  • I created a local folder named C:\DB_Backup. This folder should be shared to "Everyone", and then the NTFS permissions modified so that the local Windows backup account has full permissions.
  • The following Powershell script file needs to be created. I store my scripts in C:\PS:

#--------------------- BEGIN SCRIPT -------------------------------

#Name of Server
$ServerName = "WebServer"

#The Path to the mysqldump.exe program. This is the program that backs up the MySQL database natively.
$PathToMySQLDump = "C:\MySQL\bin\mysqldump.exe"

#Credentials of the MySQL user account
$Username = "MySQLBackupAccount"
$Password = "MySQLBackupAccountPassword"

#Get today's date, to be used for naming purposes
$Date = (get-date).ToString('MM-dd-yyyy')

#Where to store the backup files
$LocalBackupPath = "C:\DB_Backup"

#Backup all of the Databases
cmd /c " `"$PathToMySQLDump`" --routines --events --user=$UserName --password=$Password --all-databases > $LocalBackupPath\$ServerName-AllDataBasesBackup-$Date.sql "

#--------------------- END SCRIPT -------------------------------

  • Now, I set that Powershell script to execute daily at 5:10PM in Task Scheduler.
  • I run the scheduled task and find out how long it's going to take. This information will be used later on.
  • Over on my backup server, I create another script. This script handles moving the backup files created earlier off-SAN and off-site:
#--------------------- BEGIN SCRIPT -------------------------------

#Name of Server
$ServerName = "WebServer"

#Path to the location where the MySQL server put the backups
$RemoteBackupPath = "C:\DB_Backup"

#Off-SAN copy location
$OffSANCopyLocation = "\\OnSiteNASDevice\MySQLBackup"

#Off-Site copy location
$OffSiteCopyLocation = "\\DR_NAS_Device\MySQLBackup"

#Body of the notification email
$Body = "."

#Map Y drive to the MySQL server's backup location:
net use Y: \\WebServer\DB_Backup /user:"MySQLWinBackup" MySQLWinBackupPassword

#Get a file count for success/failure test
$FileTest = (get-childitem Y:\ | measure).count

#Success/Failure Test - If the file count does not come back as expected, send an email, THEN EXIT THE SCRIPT
If ($FileTest -ne 1){
$Body = "The correct number of files is not present on the MySQL Server to be distributed off-site and off-SAN."
Send-Mailmessage -from "helpdesk@contoso.com" -to "itreporting@contoso.com" -subject "PS Report - MySQL Backup Status - FAILURE - WEBSERVER" -smtpserver contosomailserver -body $body
exit
}

#Copy the files from the MySQL Server to the off-site and off-SAN locations
copy-item -Path "Y:\*.*" -Destination "$OffSiteCopyLocation"
copy-item -Path "Y:\*.*" -Destination "$OffSANCopyLocation"

#Delete the backup files stored locally on the MySQL server
remove-item "Y:\*.*"

#For off-site location, get the creationtime and delete anything older than 7 days
$Files = (Get-childitem $OffSiteCopyLocation)
Foreach ($file in $files){
$FileAge = ((get-date) - ($file.creationtime)).totaldays
If ($FileAge -gt 7){
remove-item $File.FullName -Force
} #End If
} #End Foreach

#For off-SAN location, get the creationtime and delete anything older than 7 days
$Files = (Get-childitem $OffSANCopyLocation)
Foreach ($file in $files){
$FileAge = ((get-date) - ($file.creationtime)).totaldays
If ($FileAge -gt 7){
remove-item $File.FullName -Force
} #End If
} #End Foreach

#Send a success notification email
Send-Mailmessage -from "helpdesk@contoso.com" -to "itreporting@contoso.com" -subject "PS Report - MySQL Backup Status - SUCCESS - WEBSERVER" -smtpserver contosomailserver -body $body

#Unmap the MySQL Backup drive
net use Y: /delete

#--------------------- END SCRIPT -------------------------------

  • I save this script in the C:\PS folder on my backup server, then set a scheduled task to run the script at an appropriate time. How I determine what's appropriate? I look at the run time of the backup on the server itself, and then pick what's comfortable to me. In my case, the backup takes about 2 minutes to run, so I set this task to execute 10 minutes after the start of the MySQL backup script.

The resulting workflow is:
  1. MySQL Server backs up databases to its C drive.
  2. Backup server checks for the existence of the new backup file. If not ok, send failure email alert.
  3. Backup server copies that file to 2 other locations, then deleted the originals on the MySQL Server.
  4. Backup server deletes backup files older than 7 days at the 2 other locations.
  5. Backup Server sends success email alert.

Friday, March 28, 2014

Some PC App Recommendations

This past week I had to bite the bullet and reformat my work laptop. Piriform's CCleaner was a big help, as I was able to export a list of all installed apps. To do so, open up CCleaner and click on "Tools". At this point "Uninstall" should be highlighted, and on the bottom right-hand side, click on "Save to text file...".

Once I had my list, I installed all of the apps on my new laptop. I figured that I would note some helpful items here. I use Windows 7, so if you're running Windows 8, then your experience could be different.


  • Actual Multiple Monitors allows me to have the Windows 7 taskbar extend to my second screen. I can pin separate applications to this taskbar.
  • Advanced IP Scanner is my go-to address space scanner.
  • Autohotkey is something I would like to play with more, but my killer app for this so far is this gem, which allows me to use Ctrl+V to paste into the command prompt and Powershell.
  • Beyond Compare allows me to compare files, folders, and even registry entries. I've been using this for many years.
  • EMCO makes two utilities that I really like: MAC Address Scanner and Remote Console. Usually I use Powershell remoting, but this has come in handy when there is a configuration issue on the client side and I'm not able to use remote Powershell.
  • Image Resizer Powertoy Clone for Windows allows me to right click on picture files and quickly resize them.
  • NetSetMan allows me to quickly switch between different network adapter configurations, such as when I need a static IP address at a remote site, or when I want a static address on the iSCSI network.
  • QtTabBar gives me the ability to have tabs in Windows Explorer. This is a killer app......
  • Multiplicity allows me to have a kind of software KVM. If you've seen my setup, I have two screens above and two below. Each set of two is a different computer, and this lets me control all four screens with on keyboard and mouse.
  • Tabs for Excel was a big win with the accounting department when I discovered it and installed it for them. Now I can have multiple spreadsheets open in one Excel window.
  • VirtualCloneDrive runs in my system tray and allows me to mount ISO files.
  • Vistaswitcher (don't be scared off by "Vista" in the name), gives you a much better interface when using Alt-Tab to switch between open applications. Instead of just showing the name of the running apps, it shows you what's within the window.
  • Windows Grep will let me search through a bunch of text files for certain strings. I know I can do this through Powershell, and this is a total crutch, but sometimes things are just easier with a GUI.
  • WizMouse allows me to use my mouse's scroll wheel to scroll through something without changing my active window. All I have to do is put my mouse over the inactive app's window and scroll, and it works!




Monday, March 24, 2014

Pulling out lines that contain X from a text file

This is going to be a short one, and really is more for my reference, but hopefully it helps other people.

I had a long log file, about 490MB, and I was looking for entries that had certain content. Specifically, I wanted any line that had "5156" in it.

Get-Content c:\temp\LogFile.txt | where-object {$_ -match '5156'} | set-content c:\temp\output.txt

After the line above executes, only lines that contain 5156 are copied to the output file.

I'm going to be using this a lot in the future, I think.....

Friday, March 21, 2014

Use This Powershell Script to List Which VMs are Running on each VMware Host in Your Cluster

I don't use this that often, but sometimes I just want to know which VMs are running on each of my VMware ESXi hosts. I find it easier to fire off the script than to open up vCenter if I want to know which host a VM is running on, and what other VMs are running on that host.

#-------------------------------- BEGIN SCRIPT --------------------------------

#Get Credentials
$credential = get-credential stemalyc@kalamazoocity.org

#Add the VMware Snapin
add-pssnapin Vmware.VimAutomation.Core

#Connect to the vCenter Server
connect-viserver -server cityvc.kalamazoocity.org -credential $credential

#Get all the hosts
$VMHosts = (Get-VMHost | select Name | sort name)

#Get all the VMs
$VMs = (Get-VM | select name, vmhost)

#Find the number of hosts, which will be our counter maximum later on
$HostQty = (($VMhosts | measure).count)

#Initialize the counter to zero
$i = 0

#While the counter is less than or equal to the host counter
While ($i -le $HostQty){
#List the VMs on the host. I use $i here to reference the exact host in the $VMHosts array.
$Listing = ($VMs | where {$_.vmhost -like ($VMHosts[$i].name)} | sort name)
#Output the list to the screen with Format-Table (ft)
$Listing | ft
#Blank out the list variable so it can be reused
$Listing = $null
#Increment the counter
$i++
} #End While

#Create a pause at the end of the script
$Pause = Read-Host "Press Enter to Continue"

#-------------------------------- END SCRIPT --------------------------------

Thursday, March 20, 2014

The State of the Internet - Frankly I'm Worried

Ok, advocacy post, but I'm really getting worried about the state of the internet, so I'm dedicating a post to have people sign this online petition to the FCC asking them to block the proposed merger of Time Warner Cable and Comcast.

Three things have happened in the past few months have made me stop and really look around at the internet. I'm very concerned about its well-being.

1. The fact that Netflix is now paying Verizon to keep their traffic flowing. Verizon customers are already paying for this data to be delivered. This opens the door to Verizon extorting any data provider on the net to pay up, or something might happen to their packets along the way. What's next, Google? Cloud storage providers?

2. The fear that the Time Warner/Comcast merger might actually go through is very troublesome to me. Here we have ISP's who have been paid billions in government subsidies to improve speeds and extend network coverage. These ISPs are also in the content game and are trying to shut down competitors (See #1). It's NOT ok for a company to be providing the pipe AND the content. Also, the ISPs have been given monopoly status in most areas. They've been so brash as to sue municipalities to stop cities from building their own infrastructure. From the petition above, an approved merger would result in:

  • Comcast would control two-thirds of the nation's cable subscribers, five times any other cable company and 50 percent more than the largest satellite TV service. 
  • They would control nearly 40 percent of the U.S. broadband market. 
  • It would be trivial to leverage this power to force content providers to pay more to get their programming on, passing those price hikes on to customers.
  • And finally, it would combine two companies that already rank at the bottom of the American Consumer Satisfaction Index based on surveys with over 70,000 consumers. 

THIS IS TOO MUCH POWER.

3. The NSA. What the NSA is doing to the internet is shameful. This is supposed to be a bastion of free speech. What people don't really get is that all of this collected data is being stored somewhere. Try to think ahead 30 years; can you guarantee that there won't be some dictatorial regime in charge of the USA? Things that aren't against the law could be against the law then, and if someone wanted to know your political leanings, or whether you every did X, they need only access that data.

The citizens of the US need to pay attention to which politicians are not on the people's side in these matters and vote them out, but right now, we need to make them understand that we are watching them!

Wednesday, March 19, 2014

Some CellPhone App Recommendations

Recently, I got an HTC One, and after rooting it I finally got everything tweaked the way I wanted it to. I ran across a few great apps that I didn't know about before, so I though that I would share. :)

The first on the list is called List My Apps. I'm probably going to have to wipe my phone in the future, and this little gem allows me to export a list of all of my installed apps. Even better, I can export in HTML with links to the Google Play store so I can reinstall them easily.

I ran across an amazing anti-theft app called Cerberus. I really like being able to control my phone via the web.

Night Mode is a tool that turns your screen's brightness WAY down, so you can glance at the screen in, say, a dark movie theater without it being annoying to other people. Be warned, it's so dim that if you try using this when you aren't in a dark room you might have trouble getting the phone back to normal!

Swiftkey is another 3rd party keyboard that can look through your Gmail and Facebook items and learn to suggest words based on how you text. It's uncanny how good the prediction is.

SwipePad allows you to set a hotspot (or multiples) which opens a 'dialer' type interface to start a shortcut. For example, if I press in the upper right side of my phone, I can then drag my finger down and choose between a number of items - call my wife, start a program, etc. There are various extensions you can or need to buy for certain functionality, but this app has served me very well.

Timely is an absolutely beautiful stopwatch/alarm clock app.

I like to automate my phone, so I use Locale to change settings based on my location. When I'm at work, my phone goes to vibrate only. If I go to the grocery store, GroceryIQ starts up and my phone changes to not turn the screen off, etc. That really helps my battery life, because I keep my wifi off unless I'm somewhere I know I can connect, or I connect manually.

I FINALLY found a good program that I can use to stream things from my media server at home to my phone. I only use this on the network, but it's called Emit, and it works great so far.

I use FolderSync to sync up files on my local filesystem every night while I'm sleeping. I use KeePass to keep my passwords at home, and only ever edit the master list on my computer. At night, FolderSync downloads the the KeePass file from my PC and overwrites yesterday's copy on my phone. I always have a copy of my passwords should I need them. I also use it to copy pictures to my file server. I use the cloud too, but Google/DropBox/etc. doesn't care about my files, and if I don't take care of them they might get lost. I don't completely trust cloud storage providers yet. I use them, but I still do my own backups.


Tuesday, March 18, 2014

Automating an Audit of Service Accounts

The goal today is to show you how I am automating and documenting a list of which log on account my services are using.

To start, I have a master list of servers, which is simply a text file, out there on my scheduled task server. I made adding a server's name to that list part of my commissioning checklist (which you totally should have).

Where this has come in handy, besides answering the "How did I do this on the other server" question, is when you have to change the password for a service account. Now, you'll know where else you need to update the password and can plan appropriately.

As an aside, I like the idea of the Windows Server managed service accounts, but there are too many things that they don't work with for me to use them. I don't really need more exceptions to remember.

One thing to keep in mind is that the output is subjective. If you DON'T see something under a certain server's name, it may be good, or it may not be. A SQL box shouldn't be using LocalSystem accounts for its services, while something internal to the server, like the DHCP client off the top of my head, doesn't really need to be using a dedicated account.

Here's the script. Inline comments as per usual.

#----------------------BEGIN SCRIPT-------------------

#Create Temporary output file
$OutputFile = "c:\temp\ServiceAudit.txt"

#Get the list of servers from my master list
$Computers = (Get-Content "c:\Lists\ServiceAccountAudit-servers.txt")

#For each server in the list
Foreach ($Computer in $Computers){
#Append the server's name to the temporary output file
Add-Content $OutputFile $Computer

#Use WMI to get each services StartName (Log in account), EXCEPT for the common ones, select the name and account, append to file
Get-WmiObject Win32_Service -ComputerName $Computer | `
where {$_.StartName -notlike "LocalSystem" -and $_.StartName -notlike "NT Authority\LocalService" -and $_.StartName -notlike "NT Authority\NetworkService" `
-and $_.StartName -notlike "NT Authority\Network Service" -and $_.StartName -notlike "NT Authority\Local Service"} `
| select Name, StartName | Add-Content $OutputFile

#Add some space between server entries
Add-Content $OutputFile "`r`n"
} #End Foreach

#Declare email parameters
$To = "itreporting@contoso.com"
$From = "helpdesk@contoso.com"
$Subject = "PS Report - Services"
$Body = 'This is a list of services on all servers that are not running under "system" credentials'
$SMTPServer = "mailserver.contoso.com"

#I get the temp file as attachment simply because I copy it to our IT Sharepoint once in a while as documentation
$Attachment = $OutputFile
Send-Mailmessage -to $To -Subject $subject -From $From -body $body -smtpserver $SMTPServer -attachments $Attachment

#Delete the Temp File
remove-item $OutputFile -force

remove-item $Attachment -force

#-------------------END SCRIPT---------------------------



Friday, March 14, 2014

A Day in the Life.....

It's been a really crappy week. I had the flu and was out sick Monday through Thursday, and now I have to leave early on Friday to tend to the sick kids. :(

So, I didn't learn much this week, and REALLY haven't felt like writing. I guess I should be upfront and let everyone know: If you're checking this blog for daily posts you're going to be disappointed. I don't learn new things that are worth writing about every day, and some days I just don't have the time!

So today I'm going to bore you with my daily work routine. <grin>

The first thing I do before I sit down is walk through the server room and check for red or amber LEDs where there shouldn't be.

I get into my office and boot up my laptop, and see this.


The two screens on the top are running of a custom desktop I built to run VMware Workstation for testing. I've appropriated some additional hardware that I've found lying around, and now it has 3 SSD drives, 32GB of RAM, and a 3TB spindle drive. I can run quite a few things at once, which comes in handy when you need to spin up a domain controller, a SQL backend, a Sharepoint server, and a mail server all at once. Most of the time, though, it just sits there and is my heads-up display.

On the left side of the left monitor, I have Veeam ONE running, and next to that is Dell's SANHQ, which shows my Equallogic SAN activity. On the right side is a custom dashboard I built in PRTG. That thick green circle is my "donut". My co-workers joke about it all the time: "Don't make his donut turn red!" Those little graphs on the right side show me file system usage on my file servers. In the tables at the bottom I'm looking at top 10 highest bandwidth, least available diskspace, and slowest pings.

In just a minute, I can tell whether I'm having network, application, or storage issues, and I can see if someone dumped a bunch of stuff on my file server that probably shouldn't be there.

COFFEE!!!!! One cream, 3 sugars. Spare me your health lecture, or how non-manly my coffee is, because I have a ton of emails to read.

Usually, there's nothing in my inbox. I use the 0-inbox policy. Every email I get goes into a ticket or is cleared out when the conversation is finished. Right now, I have 3 emails because I'm conversing with some vendors and am awaiting replies.

I open the IT Reporting mailbox to about 70 emails. Most of these are simply reporting backup successes. Quite a few are from various Powershell scripts that run to report things like whether I have running VMware snapshots, or if there are any unassigned computers in WSUS, and I also get a DCDIAG report. I try to create scripts so that I get good info in the email's subject. Being able to glance at an email and tell whether I need to read further saves a lot of time.

Once I've cleaned out the inbox, I move on to the ticketing system. We use Spiceworks, and I feel like I have a pretty good system worked out to keep myself organized. Organization is key for sysadmins; it's too easy to get overwhelmed without a good system. I keep my tickets separated by priority: High for things that are very important, or that I plan on working on today. Medium for stuff to get to when the high priority stuff is done. Low for things that have a due date that is not today, or that I am waiting on someone else before I can proceed.

Due date, by the way, doesn't necessarily mean when the project has to be completed (of course, it can), but instead means that I need to do something with it on that day. For example, I had a project called "Roll out Syslog Server". The due date was only there because I needed to check on the size consumed by the logs after a week. If I had something else to do thereafter I would set a new due date or clear the due date and set the priority to medium (when I get to it), depending on what was needed.

I now clear out tickets for all of the things I just did above. I have a script that creates Spiceworks tickets for the tasks that I just went through (Checking backups, looking at my donut, yada yada yada). So right off the bat, I have 10 tickets done. YAY!

And now, I get to work on things and hopefully learn some stuff!

Friday, March 7, 2014

"Automating" Emailing a Daily Calendar

One of my colleagues needed something to email his daily schedule to his boss. I'm not really that well versed on pulling data out of Microsoft Office, but I figured I'd give it a go.

IMPORTANT: Outlook must be running in order to execute this! I know that's not ideal, but my plan is to have a shortcut to the script (as I wrote about Wednesday) so that the user can finalize their schedule for the day, then double-click to send the schedule.

#-----------------------BEGIN SCRIPT--------------------------

#Variables - Modify as needed
$TodaysDate = (Get-Date -format d)
$From = "user@contoso.com"
$To = "manager@contoso.com"
$SMTPServer = "smtpserver.contoso.com"
$EmailSubject = "User's Schedule for Today"

#Connect to Outlook session, and grab the calendar
Add-type -assembly "Microsoft.Office.Interop.Outlook" | out-null
$olFolders = "Microsoft.Office.Interop.Outlook.OlDefaultFolders" -as [type]
$outlook = new-object -comobject outlook.application
$namespace = $outlook.GetNameSpace("MAPI")
$folder = $namespace.getDefaultFolder($olFolders::olFolderCalendar)

#Initialize empty array for the appointments
$Appointments = @()

#Go through the appointments and grab any of them that occur today
#Then append those to the array
Foreach ($Item in $Folder.items){
If (($Item.start.ToShortDateString()) -eq $TodaysDate){
$Appointments += ($Item | select Organizer, ConversationTopic, Start, End)
} #End If
} #End Foreach

#Convert the array to HTML, then string for use in the email
$AppointmentsHTMLString = $Appointments | sort Start | ConvertTo-HTML | out-string

#Send the email
Send-MailMessage -From $From -To $To -Subject $EmailSubject -BodyAsHTML -Body $AppointmentsHTMLString -SMTPServer $SMTPServer

#-----------------------END SCRIPT--------------------------

Thursday, March 6, 2014

How to Automate Rebooting a Server

Let's suppose you have an application that acts up. Let's also suppose that instead of fixing the application, you are asked to reboot the server once per week, and that your persuasive arguments fail.

Here's how I automate the rebooting of one of these mythical beasts.

Create a batch file named RebootThisFilthyAppServer.bat, and then put the following in the batch file:

c:\windows\system32\shutdown.exe /r /c "Weekly Reboot Scheduled Task" /f /d p:04:01 /t 90

What the switches mean:
/r - reboot (/h would be a shut down. "halt" is easy to remember)
/c - The comment. In this case, it's "Weekly Reboot Scheduled Task"
/f - force
/d - provide the reason. If you do a shutdown /? at a command line, it lists the reasons. In this case, p:04:01 is "Application: Maintenance (Planned)"
/t - The timeout period before shutting down. This gives your apps a chance to close gracefully in response to the shutdown sequence, before they are forced by the /f

Now, make a scheduled task to run the batch file at the time that you want to rebooot, and you are good to go.

Wednesday, March 5, 2014

Creating a Shortcut to Run a Powershell Script

So I've got a bunch of useful script scattered throughout this blog, and sometimes I want to fire one off quickly. You know, without opening a Powershell prompt and typing a full pathname in.

BRING FORTH THE SHORTCUT!!!

Right-click on your desktop and make a new shortcut. For the location of the item, type in:
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -File <PathToScript>

Remember to enclose <PathToScript> in quotes if it has space in it. My advice to you would be to use CamelCase at all times so you don't have to worry about that, though.

And voila! You have a shortcut that you can click on to run a script!

If you need to see the output before the PowerShell window goes away, try adding a dummy read-host command to the end of the script you're calling, like so:

$Pause = Read-Host "Press Enter to Continue"

This way, when the script gets finished, it will hang there at a "Press Enter to Continue" prompt.

Tuesday, March 4, 2014

Alerting for Failed Scheduled Tasks

When I first started setting up my scheduled tasks and automating a lot of things, I had another script that ran every morning to let me know if there was anything that failed.

The downside to doing this was that I wasn't notified WHEN the failure occured, but hours later. A better way to go is to created a scheduled task triggered by a scheduled task failure event, and set the action for that trigger to be to send you an email.

To do this, make a new scheduled task, name it and set the user to run as, and set it to run even when the user is not logged on.

Next, go to the Triggers tab. You are going to make two triggers here.

Trigger One: Begin the task "On an event"
For the Log, drill down to Microsoft-Windows-TaskScheduler/Operational
For the Source, choose TaskScheduler
For the Event ID, choose 103

Trigger Two is going to all the same, except you're going to choose 203 for the event ID.

For the action, choose "Send an e-mail" and fill in your info.
FOR SOME STUPID REASON, beginning in Windows Server 2012 Microsoft has decided to get rid of the "Send an e-mail" action. The easy workaround is to have it run a program, and then call Powershell to run a script that sends the email. :)


Monday, March 3, 2014

What if your Monitoring Server Goes Down? Monitoring a website via Powershell.

Shortly after I got my monitoring server (PRTG) up and running, I started looking around for things to monitor.
"If X happens, will we be notified?"

My monitoring server was a VM. My VMware cluster runs on a SAN. Oops!

I moved PRTG to a physical box, with appropriate redundancies (RAID10, Power Supplies, and power source). NOW if the SAN or my VMware cluster fails I'll know about it.

WAIT! What if my email system is down? I verified that I had set PRTG to use it's built-in SMTP server. I ALSO set up a notification that goes to IT's personal email addresses in case the email server goes down, because you can't get the notification emails if they're not delivered to the mailbox on the server that's down.

I also set up a Powershell script to check that the http website of the monitoring server is up. I have this script scheduled on my "General Purpose IT Server" (it has all the admin software, and runs all of my scheduled tasks) to check it every 5 minutes:

#--------------------BEGIN SCRIPT---------------------------

#The URL of the main page you want to monitor
$URL = "http://MonitoringServer:8080/index.htm"

#The hostname of your monitoring server (used in the email later on)
$URL_Server = "MONITOR"

#Set up a web client connection
$webclient = New-Object Net.WebClient

#Download the text of the html code for the URL
$webtext = ($webclient.DownloadString($URL) | out-string)

#Copy that over to a temporary text file for later
$webtext | out-file c:\temp\PRTGWebsite.txt

#If you are making this for YOUR site, stop here and go look at the text file, then identify some unique text.
#Here, specify the search string. Put in the unique text you found.
$SearchString = 'Paessler AG - The Network Monitoring Company'

#Initialize the response variable
$Response = "FALSE"

#Search for the string in the text file. If it's there, change the response to true.
if (select-string -Path C:\Temp\PRTGWebsite.txt -Pattern $SearchString){
$Response = "TRUE"}

#If the response is NOT true, then the script sends email, because something's wrong with your site
If ($Response -notlike "TRUE"){
$body = "$URL on $URL_Server is down!"
Send-Mailmessage -from helpdesk@contoso.com -to ITSupport@contoso.com -subject "$URL on $URL_Server is down!" -smtpserver SMTPServer.contoso.com -body $body
} #end if

#Delete the temp file you used earlier
del c:\temp\PRTGWebsite.txt

#--------------------END SCRIPT---------------------------

We had this script go off after an upgrade because the HTML code had changed. We just needed to manually run the code (above) down to where it created the temporary text file, look for a new search string, and update the script.