Increase in Password Spraying Attacks

Per Microsoft, there is a marked increase in Password Spraying attacks. With this being security awareness month, take the opportunity to educate your users NOT to reuse passwords on different sites/applications. 

If you have the capabilities, I recommend watching your logs for login failures for different users from the same IP. Do some diligence in looking up WHOIS and geolocation data, then block IPs for unexplainable failures at your ingress points. 

I'm Back, with my Master Inventory Database methodology

I’m going to start writing again!

I was able to pivot my career from a Senior System Admin into a Security Engineer role for my organization a few years ago. I was very busy and kind of let this blog go while I sunk my teeth into building and managing organizational standards related to the CIS 20.

I learned a lot in my journey, and I have a lot of ideas for new articles that may help you in getting your organization up to snuff with regards to security.

One of my primary passions during this time was building a Master Inventory Database. I did this by pulling data from various sources within my environment, combining the data, and then asking questions of the data. I built a suite of Powershell commands and deployed these through profiles that could be used by various roles within the organization to view role-relevant data in a single pane of glass.

To retrieve this data, I mostly used tool-specific automated reporting (at midnight the tool exports its inventory to a CSV file, which I would then ingest), though I used Powershell where modules existed for the tool (WSUS/DNS/AD/DHCP), and I was in the process of branching out into querying APIs before I left the position.

DATA IS THE KEY. If you have the data, you can present it in a way that allows management to make good decisions for your organizations, tailored to risk appetite and security program maturity.

Controls 1 and 2 of the CIS 20 deal with inventory. If you start reading through the rest of the controls, you really can’t say that you conform to many of the other controls without having first identified every device and piece of software that exists within your environment. This Master Inventory Database project aimed to thoroughly satisfy those first two controls and pave the way for the implementation success on the rest of them.

1.       DNS

a.       Every system should have a DNS record, though logic should exist to weed out client systems that may be transient. The DNS record usually informs the other systems mentioned below on the name of a system.

b.       The DNS name should conform to a naming standard (note that this initiative may take years to accomplish – it’s much easaier to change system names on upgrade or replacement that renaming systems)

c.       Does a record exist?

d.       Does the system have a PTR record?

e.       Are there any aliases?

2.       Basic network queries:

a.       Is ping successful?

3.       Client Management System, which hopefully includes third party patch management:

a.       Form Factor/Bitlocker Status

                                                               i.      Protected Mobile System or not?

b.       RAM

                                                               i.      Standardize/Replacement Info Purposes

c.       CPU

                                                               i.      Standardize/Replacement Info Purposes

d.       Make

                                i.      Standardize/Replacement Info Purposes

e.       Model

                                                               i.      Standardize/Replacement Info Purposes

f.        HDD Space

                                                               i.      Ensure there’s space to start saving logs

g.       Last Logon User

                                                               i.      Who is using this system?

h.       Installed Software

                                                               i.      Have an idea of what’s running in your environment, so when you’re going through the days’ news you can identify things that may affect you. For example, seeing the headline “Google Chrome releases patch for 0-day vulnerability”.

                                                             ii.      Licensing

                                                           iii.      Are all systems running the software required by IT Operations/Security (Agents, Antivirus, etc)

4.       AD Computers

a.       Is the Bitlocker key successfully stored here?

b.       Last Logon Time

c.       Who is the “owner” of this system?

5.       AD Users

a.       Is there an employee number on file to verify identity for helpdesk calls?

b.       Is the user account a member of any special groups (Domain Admins, etc)?

c.       Is there a manager listed for out-of-the-ordinary requests, such as user requesting access to a share, or forgetting their employee ID during identity verification?

6.       Antivirus

a.       Has the system checked in recently?

7.       WSUS Status

a.       Is the system patched?

b.       Has it reported recently?

8.       Vulnerability Management Reports (I’ve worked with Rapid7 Insight IVM)

a.       System risk

b.       Open Ports (Can also get this through automating nmap scans)

c.       Configuration Standard scanning results (CIS/DISA/STIG)

d.       Software Installed

                                                               i.      Have an idea of what’s running in your environment, so when you’re going through the days’ news you can identify things that may affect you. For example, seeing the headline “Google Chrome releases patch for 0-day vulnerability”.

                                                             ii.      Licensing

                                                           iii.      Are all systems running the software required by IT Operations/Security (Agents, Antivirus, etc)?

9.       Wireless Network

a.       SSID Connection/VLAN

10.   DHCP    

a.       What’s the system’s IP address

b.       In what DHCP zone does the system pull from?

11.   Network Scanner (ManageEngine OpUtils, NetDisco, etc)

a.       What switch port is the system plugged into?

b.       VLAN

12.   Other things you could assess, but I didn’t get there:

a.       Are Backups present, up to date, and successful?

b.       Is the system being monitored for outages?

c.       Browser Plugins and/or Office Add-Ons Installed

d.       O365 user info/Conditional Access/Licensing/Sensitive email groups

e.       Local registry settings, cross referenced with group policy since all GPOs do is set registry settings

f.        System User Rights assignment

                                                               i.      Who’s a local admin?

                                                             ii.      Make sure that sensitive rights are appropriate (who can log in as batch user)

g.       System logging configuration

                                                               i.      Best configured via group policy

h.       Windows Firewall status

                                                               i.      Best configured via group policy

i.         System Shares and permission settings

                                                               i.      Tell me ‘Everyone’ doesn’t have access

                                                             ii.      Client systems shouldn’t have shares, typically

j.         Any other tooling you may have that contains information of value that has an API or reporting capability. For instance, you could pull Security Awareness Training records and phishing test results in to help identify your riskiest users and tailor future training accordingly.

 

Once you get a handle on even some of the above, you can start sorting systems into groups and creating standards (security or otherwise). You can build reports that you can hand to operations to resolve. For example:

a.       These mobile systems don’t have Bitlocker enabled.

b.       These systems don’t have antivirus installed.

c.       These systems don’t have a DNS name – what are they?

d.       These systems still have Adobe Flash installed!

e.       These systems have Office 2010 installed!

f.        These systems have open Telnet ports.

g.       These systems haven’t installed this months’ patches yet.

h.       These systems don’t have our web filter agent installed or proxy set correctly for protected web access.

i.         These systems haven’t been seen by <Insert tool here> for X Days (meaning their agents are broken, or they aren’t checking in, or maybe the computer isn’t being used). Such a rule exists to make sure you don’t have some device coming back online after 3 months and not have the appropriate patch levels. I didn’t have a network posturing capability….

j.         These users have high level access – do they need it?

k.       These systems have an agent that’s out of date.

 

Any exceptions to your standards should be tracked in a risk register and reviewed regularly with management.

Remember, it is up to management to assume (do nothing), remediate (fix), mitigate (lessen the risk, typically by putting such devices into their own VLAN), or transfer the risk (insurance, out-sourcing management).

I feel like most of the stress I’ve encountered in my career is due to this risk. I know the risk is there, and I’ve communicated this risk to management in a data-based fashion (keeping FUD to a minimum, which is a nebulous line).

Theoretically I should be off the hook, psychologically. HOWEVER, where this breaks down is that historically I’ve been the individual responsible for incident response.

So, the way this plays out in my brain a lot of the time is that I’ve given the powers that be all the data, and they’ve decided to accept the risk, BUT I’m the one waiting for the 3am phone call that we’ve been compromised, possibly because of the flaw I identified.

I’m still wrangling with this. In my security engineering role, I let the stress get to me. I took the stress out on my family and made some very bad life choices. Ultimately, I ended up quitting my job for my mental health.

I urge you to try to come to terms with the fact that you may not be able to control your security environment, and you need to steel yourself from the fallout that may land on you because of it.

AD Sites and Services: Show Services

Tonight I was working on something and deauthorized my DHCP server in Active Directory (long and not very interesting story - I promise I wasn't randomly clicking things..... ).

So I just went in to reauthorize it, and I get a very helpful (ahem) message that tells me that "The specified servers are already present in the Directory Service".

Yay Google, turns out that there's a very cool part of Active Directory Sites and Service that I'd never even seen before!

From a DC, open up Active Directory Sites and Services. Normally this is where you do all of the fancy site replication stuff if you have multiple AD sites, but if you highlight the root of the structure, choose the "View" menu, and select "Show Services Node", there's a lot more to see.

Most of this stuff I wouldn't touch without explicit instructions, of course, but still neat. You can see Exchange stuff, and your certificate info under the Public Key Services folder.

What I needed to do was to delete my server's entry under the NetServices folder, and then I was able to again authorize my DHCP server.

Gathering Important NTP Settings with Powershell

I started a new gig recently, and noticed some time issues. So, once I got all of the servers set up to speak Powershell, I pieced together the following script to do an audit of all of my servers and what they were doing for NTP. I found a really nice blog entry that explains what all of this means....

One thing I'm much more cognizant about is putting any constant variables at the TOP of my scripts. This allows me to more easily reuse my scripts, and also to change the variables quickly without having to look all over the place.

#BEGIN SCRIPT

<#
REQUIRED: Make a folder called c:\lists, and put a file in it named ServerNTPSettingsAudit.txt that contains your servers' names (one per line). Also, this script assumes that you have a C:\temp folder.
#>

#Variables
$List = "C:\Lists\ServerNTPSettingsAudit.txt"
$Attachment = "C:\Temp\NTPSettings.csv"

#Email Variables
$To = "reporting@contoso.com"
$From = "reporting@contoso.com"
$SMTPServer = "mail.contoso.com"
$Subject = "PS Report - NTP Settings Audit"
$Body = "See Attached"

#Get the list of servers
$Servers = Get-Content $List

#Create an empty array to hold the data
$NTPSettings = @()

#Foreach server, get some NTP settings from the registry (remotely, obviously)
Foreach ($Server in $Servers){
    
    $HKLM = 2147483650 #HKEY_LOCAL_MACHINE

    $reg = [wmiclass]"\\$Server\root\default:StdRegprov"

    $key = "SYSTEM\CurrentControlSet\Services\W32Time\Parameters"
    $value = "Type"
    $NTPType = $reg.GetStringValue($HKLM, $key, $value)  ## REG_SZ

    $key = "SYSTEM\CurrentControlSet\Services\W32Time\Config"
    $value = "AnnounceFlags"
    $NTPFlags = $reg.GetDWordValue($HKLM, $key, $value)  ## REG_DWORD

    $key = "SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NTPServer"
    $value = "Enabled"
    $NTPServer = $reg.GetDWordValue($HKLM, $key, $value)  ## REG_DWORD

    $ServerItem = New-Object System.Object
    $ServerItem | Add-Member -type NoteProperty -Name "Server Name" -value $Server
    $ServerItem | Add-Member -type NoteProperty -Name "NTP Type" -value $NTPType.sValue
    $ServerItem | Add-Member -type NoteProperty -Name "AnnounceFlags" -value $NTPFlags.uValue
    $ServerItem | Add-Member -type NoteProperty -Name "IsNTPServer" -value $NTPServer.uValue
    
    $NTPSettings += $ServerItem

} #End Foreach

#Export the array to CSV
$NTPSettings | Export-csv -NoTypeInformation $Attachment

#Send me the list as an email attachment
Send-mailmessage -To $To -From $From -SmtpServer $SMTPServer -Subject $Subject -Body $Body -Attachments $Attachment

#Delete the temp file
Remove-Item $Attachment -Force -ErrorAction SilentlyContinue

#END SCRIPT

Windows Allegedly Fixes Slow Windows 7 Updating Issues

In case any of you have seen Windows 7 take literally HOURS to scan for updates when the Check For Updates is initiated manually, try out the newest fix from MS:

http://www.infoworld.com/article/3086811/microsoft-windows/microsoft-releases-kb-3161647-kb-3161608-to-fix-slow-windows-7-update-scans.html

Report on Cisco VPN Logins from Syslog..... Logs....

Man it's been a long time! Don't know what to say; sometimes I feel like writing and sometimes I don't. I definitely have a long list of things to Blog about. Maybe I'm just destined to be an "in spurts" type of blogger.

The rest of this is a script I created to keep track of people that were using the VPN for licensing purposes, though it does have security implications as well. I wanted to get rid of accounts that very rarely or never used our VPN capabilities.

<#
What we want is to parse the ASA syslog files stored in the syslog folder. These are in txt format and are rather large.

This Powershell script is scheduled to run after midnight on the syslog server every day.

The script autogenerates a new CSV file if it doesn't exist. Results should append to the CSV file daily, and we pull down and remove the csv file weekly.

On Mondays (or a day of your choosing, see the variables section), it counts the entries, keeps only unique logins, and sends the file as an attachment to me. It then deletes the concatenated csv file.

#>

############
# Variables
############

#Get Today's Date
$Today = Get-Date

#Get yesterdays date
$Yesterday = $Today.AddDays(-1)

#Path to txt Syslog Files
$SyslogFilePath = "D:\ASA Syslog Files\ASA1\"

#Name of output file
$OutputFile = "D:\PowershellLogData\ASA1_VPNConnections.csv"

#Specify the day of the week to report (Monday by default)
$ReportDayOfWeek = "Monday"

#Build filename of yesterday's log file, with the path
$FileName = $SyslogFilePath + ($Yesterday.ToString('yyyy-MM-dd')) + ".txt"

#Mail Variables
$To = "me@contoso.com"
$From = "me@contoso.com"
$SMTPServer = "mail.contoso.com"

#Get the day of the week
$DayOfWeek = ((Get-Date).DayOfWeek).ToString()

#If the Output CSV File Doesn't Exit, create one
If ((Test-Path $OutputFile) -eq $False){
    $Headers = @()
    $HeadersEntry = New-Object psobject
    $HeadersEntry | Add-Member -MemberType NoteProperty -Name Timestamp -Value "ScriptEntry"
    $HeadersEntry | Add-Member -MemberType NoteProperty -Name Group -Value "ScriptEntry"
    $HeadersEntry | Add-Member -MemberType NoteProperty -Name User -Value "ScriptEntry"
    $HeadersEntry | Add-Member -MemberType NoteProperty -Name IPAddress -Value "ScriptEntry"
    $Headers += $HeadersEntry
    $Headers | Export-CSV $OutputFile -NoTypeInformation
}

#Parse Yesterday's log file for only VPN connection entries
$ConnectionEvents = select-string -path $FileName -Pattern "722022"

#Create an array
$LogInfo = @()

#Cycle through each VPN Login entry and extract the data, adding to the array
$ConnectionEvents | Foreach-Object {
    
    #Extract the Info
    $infos = $_ -split '\t'
    $TimePre = $Infos[0] -split ':'
    $Time = $TimePre[3] + ":" + $TimePre[4] + ":" + $TimePre[5]
    $BetterInfo = $Infos[3] -split '<'
    $Group = ($BetterInfo[1] -split '>')[0]
    $User = ($BetterInfo[2] -split '>')[0]
    $IPFrom = ($BetterInfo[3] -split '>')[0]
    
    #Build the Object
    $LogInfoItem = New-Object psobject
    $LogInfoItem | Add-Member -MemberType NoteProperty -Name Timestamp -Value $Time
    $LogInfoItem | Add-Member -MemberType NoteProperty -Name Group -Value $Group
    $LogInfoItem  | Add-Member -MemberType NoteProperty -Name User -Value $User
    $LogInfoItem  | Add-Member -MemberType NoteProperty -Name IPAddress -Value $IPFrom
    $LogInfo += $LogInfoItem
    
    } #End Foreach-Object

#Append the array to the csv output file
$LogInfo | Export-CSV -Append $OutPutFile

#If it's Monday, clean up the file and send it out, then remove the original CSV so it's rebuilt for the next reporting week
#If it's NOT Monday, just do the data conversion and leave the file intact.
If ($DayOfWeek -like $ReportDayOfWeek){

    #Import the Output CSV File
    $Entries = Import-CSV $OutputFile

    #Keep only entries that have populated username fields and weren't created on CSV initialization (ScriptEntry piece)
    $Entries = $Entries | select-object  | Where-Object {$_.user -notlike "" -and $_user -notlike "ScriptEntry"}

    #Create Report information
    $ReportObject = $Entries | select user -unique | sort user
    
    #Create HTML Report
    $ReportHTML = $ReportObject | ConvertTo-Html | out-string
    
    #Count the Entries
    $VPNCount = (($Entries | Measure-Object).Count).ToString()

    #Craft the Email Subject wit the count
    $Subject = "PS Report - Cisco ASA VPN Logs - $VPNCount Logons Last Week"

    #Send the email
    Send-MailMessage -To $To -From $From -SmtpServer $SMTPServer -Body $ReportHTML -BodyAsHTML -Subject $Subject -Attachments $OutputFile

    #Remove the CSV file
    Remove-Item $OutputFile -force -ErrorAction 0

} #End If Monday

VMware HA Testing Tool/Site

Something I ran across today is this VMware tool/site that allows you to upload a DRS dump file from your environment and simulates a host failure in your VMware HA cluster. It's always nice to have a test on what you think will happen, and this is much easier and less (possibly) disruptive than pulling out the power cords from your ESXi server.....

Good stuff!

PS, I passed, except for one VM that will only boot on one host.

Command by Command: My Standard Ubuntu Server Build

These are the instructions that I use to build my Ubuntu Server VMs. Once I get done with these, I then add any other software that the server needs. I live in a Microsoft world, so creating this was quite an exercise and took a long time. It was initially created on Ubuntu 14.04.1, but I just ran through it on 14.04.3 and it was fine. To that end, I used to have a section on installing VMware Tools manually, but I got a prompt on 14.04.3 that I should use open-vm-tools, so I'm going that route.

Here we go:

Ubuntu 64-bit (14.04.3 tested (original written on 14.04.1)

My Standard VM build:
60GB HDD
Network Connection (with internet)
4GB RAM
1 CPU
Obviously change depending on your ultimate use case.

During Installation:
All defaults except:
Hostname
Non-Root User Account
Password
Proxy, if needed
Security Automatic Updates only
No Package Installation
Remove disk, reboot

Log in
Change login to root:
sudo su -

Install open-vm-tools with
apt-get update
apt-get install open-vm-tools

Configure static IP Address, etc:
nano /etc/network/interfaces
change 'iface eth0 inet dhcp' to 'iface eth0 inet static'
add the following lines:
address <ipaddress>
gateway <gateway>
netmask <Mask>
dns-nameservers <DNSServersSeperatedByASpace>
Restart the computer
Log in as non-root user
Verify connectivity using ifconfig, ping, nslookup

Update apt-get
sudo apt-get update
Install Ubuntu patches:
sudo apt-get upgrade

Install and configure OpenSSH:
apt-get install openssh-server
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.default
chmod a-w /etc/ssh/sshd_config.default
addgroup sshusers
nano /etc/ssh/sshd_config
Change the following:
X11Forwarding no
LogLevel VERBOSE
LoginGraceTime 30
                        MaxStartups 2:30:10
Add the following lines:
AllowTcpForwarding no
AllowGroups sshusers
usermod -a -G sshusers <Non-Root User>
Restart the SSH service:
sudo restart ssh
Run this command to rate limit the SSH Connections (if more than 10 attempts within 30 seconds, all the following attempts will fail since the connections will be DROPped.)
sudo ufw limit ssh

Created DNS A and PTR records
Verified SSH works for Non-Root user

Set up UFW (Uncomplicated Firewall) (AS ROOT):
ufw allow ssh
ufw logging on
ufw enable
TO SHOW STATUS: ufw status
LOG FILE LOCATION: /var/log/ufw.log

Prior to setting up sendmail, ensure your mailserver will accept anonymous mail from this server's IP address.

Set up the ability to send emails:
Install sendmail:
apt-get install sendmail
Create a copy of the default file before editing:
cp /etc/mail/sendmail.mc /etc/mail/sendmail.mc.defaults
Configure sendmail:
nano /etc/mail/sendmail.mc
Your last two lines are as follows:
MAILER('local')dnl
MAILER('smtp')dnl
Put this code before those two lines:
define('SMART_HOST','mailserver.contoso.com')dnl
Save and exit
Enable changes:
cd /etc/mail
m4 sendmail.mc > sendmail.cf
make
/etc/init.d/sendmail reload
Test sendmail functionality:
echo "My test email being sent from sendmail" | /usr/sbin/sendmail youremail@contoso.com

NTP Client Setup:
apt-get install ntp
nano /etc/ntp.conf
comment all lines that begin with 'server' by placing a # in front of them
Add the following line before the first 'server' line:
server <NTPServerFQDN>
Restart NTP:
service ntp restart
Test NTP:
ntpq --numeric --peers
In the results, you will see the remote IP of the server you configured.

Fail2Ban setup:
apt-get install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
nano /etc/fail2ban/jail.local
change destemail = alertEmail@contoso.com
action = %(action_mwl)s
RESTARTING: /etc/init.d/fail2ban restart
LOG FILE AT: /var/log/fail2ban.log

Using Powershell to Sift Through My Email

Every morning I have over 100 new emails. Most of these I glance at and archive, because I only need to know that the processes ran. After thinking about how to better optimize this, it occured to me that I wasn't getting the information I needed.

For example, I get over 20 emails from Veeam about Backup Jobs and BackupCopy Jobs. They're successful (if they aren't there's a rule that forwards the offending email to my normal email address), so what's the problem? Well, the problem is that I see that they're successful, grab the whole chunk, and mark as read/archive. Was there 20? Or only 19? Now, in this example I would know if one of the jobs was hung or something because I'm running another script to check for snapshots before the workday, but what about other things like MySQL backups, or Backup Exec jobs (yes, shudder)?

So, here's what I'm doing. These emails are all sent to a reporting mailbox, that forwards any emails with issues to the appropriate personnel. I will use Veeam email as an example.

An email with a subject of "Veeam Job [Success] Daily-Job" comes in. A rule on my reporting mailbox marks it as read and throws it into an "Archive" subfolder.

At 7:15AM, a scheduled task runs on my computer at work, with Outlook open and the reporting mailbox loaded. I'm going to do this script in pieces, explaining each part in between.

#Mailbox Name
$account_address = "reporting"

#Folder in that Mailbox
$mails_folders = "archive"

#Email Variables
$To = "me@contoso.com"
$From = "reporting@contoso.com"
$SMTPServer = "mail.contoso.com"
$Subject = "Reporting Mailbox Summary"

##########################################################################
#          Date/Time Variables                                           #
##########################################################################

#Begin: 5pm yesterday 
$BeginningDateTimeString = (((Get-Date).AddDays(-1)).ToString("yyyy-MM-dd") + " 17:00:00")
[datetime]$BeginningDateTime = $BeginningDateTimeString

#End 7am today
$EndDateTimeString = ((Get-Date).ToString("yyyy-MM-dd")) + " 07:00:00"
[datetime]$EndDateTime = $EndDateTimeString

##########################################################################
#          Stuff with Outlook                                            #
##########################################################################

#Create outlook.application object
$outlook = new-object -com outlook.application
$MailNameSpace = $outlook.GetNameSpace("MAPI")
$MailFolders = $MailNameSpace.Folders |? {$_.Name -eq $account_address}

#Getting main inbox folder
$inbox = $MailFolders.Folders |? {$_.Name -eq "Inbox"}

#Specify folder of mails to calculate
$folder_to_calculate = $inbox.Folders |? {$_.Name -eq "$mails_folders"}

#Get the mail
$Emails = $folder_to_calculate.Items


At this point, $Emails has all of my mail in it (I have autoarchive enabled on the mailbox to delete after 30 days). Now, I'm only interested in a subset of this data. Searching through a months worth of email would take awhile, so I constrain the dataset with the following line, so I only get email received last night after 5pm and before this morning at 7am (see date/time variables above):

$TimePeriodMails = $Emails | Where-Object {$_.ReceivedTime -gt $BeginningDateTime -and $_.ReceivedTime -lt $EndDateTime}

Now I filter what I'm interested in by subject. Here's 4 lines:

$ApplicableMails = $TimePeriodMails | where-object {
    $_.TaskSubject -like 'Backup Exec Alert: Job Success *' -or `
    $_.TaskSubject -like "PS Report - GPO Backup Report" -or `
    $_.TaskSubject -like "PS Report - MySQL Backup Status - SUCCESS - *" -or `
    $_.TaskSubject -like "Veeam Job ``[Success``] Daily-*"}

One interesting tidbit I discovered, through much gnashing of teeth and Googling, is that when you do a string comparison, and the string includes square brackets you have to double-escape them (using the backtick)!

Once I have my emails, I just need to gather a count of the data, which I did like so:

#Create an array to hold the data
$ResultArray = @()

#Look for the Backup Exec Emails and count them

$ArrayItem = New-Object psobject

$ArrayItem | Add-Member -MemberType NoteProperty -Name Name -Value "BE Backup Successful"

$BESuccessMails = $ApplicableMails | where-object {$_.TaskSubject -like 'Backup Exec Alert: Job Success (Server: *'}

#Count those, and convert that number to a string

$BESuccessMailsCount = (($BESuccessMails | measure-Object).count).ToString()

#This next line is for my reference, once I get the process down, I'll put in here how many emails I should see. 

#You'll see why this is important (to me) later.

$BESuccessMailsCountShouldBe = "777"

#Make an array item and add the data I want to the result array

$ArrayItem | Add-Member -MemberType NoteProperty -Name ShouldBe -Value $BESuccessMailsCountShouldBe

$ArrayItem | Add-Member -MemberType NoteProperty -Name Is -Value $BESuccessMailsCount

$ResultArray += $ArrayItem

I'll spare you the other 4 search blocks; they're the same format, just with different names and data to look for.

The last step is to add some formatting, because who doesn't like a nice table to look at? You'll see here that I've included the number of emails I SHOULD see, so that with minimal effort I can deduce that all of my stuff ran.

##########################################################################

#          Format and Send                                                                                                           #

##########################################################################

#HTML Style Formatting

$style = "<style>BODY{font-family: Arial; font-size: 10pt;}"

$style = $style + "TABLE{border: 2px solid black; border-collapse: collapse;}"

$style = $style + "TH{border: 2px solid black; background: #dddddd; padding: 5px; }"

$style = $style + "TD{border: 2px solid black; padding: 5px; }"

$style = $style + "</style>"

#Export the array, with the style, to HTML

$Body = $ResultArray | ConvertTo-Html -Head $style | out-string

Send-Mailmessage -To $To -From $From -SMTPServer $SMTPServer -Subject $Subject -Body $Body -BodyAsHTML

It looks like this , which is much abbreviated, and not using the same fields as above (sorry):

You'll notice here that I haven't received the number I expected, which I've since fixed (this was due to that double-escaping of square brackets!).

Thanks for reading, and Happy Thanksgiving!