Things get complex, but don't forget to check the easy stuff first

So my Veeam backup solution wasn't performing as I thought it should. Veeam had direct access to my SAN via a 4Gb FiberChannel HBA, and was only pulling around 40MB/s. I was pulling my hair out and finally posted on a forum for assistance. The advice? Check the driver on the HBA card. Lo and behold, the QLogic HBA card was using a MICROSOFT driver. That's just no good. After updating the driver, my speeds went from 40MB/s to anywhere between 130-250 MB/s. Now that's a speed boost!

I think it's important that we as technical people strive to maintain good troubleshooting skills. Sure, the infrastructure gets complicated, but don't ever forget that the best fix might be the easiest. Your internet isn't working? Start from the bottom and work your way up. Sure, it might be a routing issue, but put traceroute away for a minute and check that the network cable is plugged in. Keep it simple, stupid.

Ice Cream Sandwich and my battle with backups (Backup Exec, Veeam, and SQL Oh My!!!)

I've been using Firefox for, well, a very long time - IE6 era, I think. I've been hanging on because Firefox could sync my bookmarks and passwords between all of my devices. My Samsung Galaxy S2 was on Android version 2.3.3 (Gingerbread) and I couldn't run Chrome on it. I was pretty close to rooting it, but I fix computing devices often enough, and my phone is a device that just has to work. My life is too busy to have it go down and immediately be able to sit down and fix it. Aaaanyway, I got the Ice Cream Sandwich update from Sprint this past week (which was out in, what, March?) and NOW I can run Chrome!!!! My favorite thing so far is Chrome-to-Phone, and my biggest pet peeves are the lack of a decent FTP program like FireFTP on Firefox, and the absence of any command to sort my bookmarks alphabetically. I've also installed Google Cloud Print, but haven't needed to use it yet. The extensions are pretty amazing:

• I can post to this blog, my blog on Tumblr (The LAG), Twitter, Facebook, etc with AddThis
• X-Notifier checks all of my email accounts on a regular basis
• Facebook, Google Phone, and Skype all have good add-ons as well

I really like Ice Cream Sandwich, too; I'm just more excited about being able to switch the default browser on all of my computers to Chrome. My notifications are much more descriptive, and it looks like I've gotten a battery boost.

At work, I've rolled out PRTG for network monitoring. Having a good network monitoring solution in place saves the IT department A LOT of headaches, especially with software as robust and flexible as PRTG. Having the ability to see all of your problems (with some exceptions) at a glance saves a ton of time. I turn off monitoring while I run the monthly Microsoft updates, then when I'm done rebooting the servers I can flip PRTG back on and see almost immediately if anything didn't come back on correctly. The BEST thing about something like PRTG though, and the most indispensable feature, as far as I'm concerned, is the tracking of sensors over time. Now, I can look at some graphs and predict when we might need to add drive space, or if someone copies a giant 20GB file to my server I can detect it immediately and possibly head things off before they bring down the server (this has happened more than once).

Also, I'm completely redesigning the backup situation. We were using Symantec Backup Exec 2010 R3 exclusively for backups, and we weren't using it right. We needed a lot more licenses to comply with the licensing requirements for myriad SQL servers, and the backup process just wasn't working well. Backup Exec is.... difficult to use. For me, anyway. Maybe with some training.... oh never mind. So I sold the boss-man on plunking down about $14,000 for the Veeam Management Studio, which includes Veeam ONE monitoring for our VMware Infrastructure, which plays into the previous topic. So I've finally got Veeam Backup and Replication 6.1 in place and it's doing it's first backup as we speak. Well, technically it's the second, but the first backup was so slow I had to scrap it. You see, Veeam B&R can talk directly to the SAN over a Fiber Channel HBA, instead of trying to move 5 Terabytes (in my case) over a gigabit network link. I misconfigured it the first time and was in network mode. Veeam also does Reverse Incremental backups, Change-Block Tracking (only backs up things that have changed), and is really good at deduplication (REALLY good). I'm hoping to cut my storage needs in half, and shrink my backup window big time. For example, my current backup has now read 3.4 TB, and only written 1.9 TB to disk. A little over half. Monday's backup will only back up what's changed from this one, so it should run pretty fast. I'll post some more stats once I get the process going and smoothed out. When Veeam's done, it will automagically launch my Backup Exec job, which will write the Veeam backup to tape. So, here's how to launch a backup exec job with Veeam: Edit the job, select storage, click the advanced button, and click on the advanced tab. The bottom section allows you to run commands at the end of a backup job. The command to launch a backup exec job looks like this: "C:\Program Files\Symantec\Backup Exec\bemcmd.exe" -o1 -jJobName. Use the quotes in there, by the way. Here's a handy web page that outlined the rest of the details.

While I was auditing what was being backed up, I kept finding little SQL Express instances everywhere. Symantec requires a special agent to properly back these up. Instead of plunking down around $9,000 more for the appropriate licenses to back up around 15 more SQL Express servers with Backup Exec, I'm skipping the SQL instance in my backup selections, and will just run a scheduled SQL script as a scheduled task to back up the database to a .bak file before the regular file backups run. Here's the page that gave me the instructions on putting this together. It's talking about backing up your VMware vCenter database (if you store it in SQL Express), but the same technique applies to other SQL Express instances.

Working with Scheduled Tasks

So I've been getting the IT "general purpose" server up and running. I have all of my management tools installed, and now it's time to automate some reporting and maintenance functions. Right now, I have 3 scheduled tasks:

• Windows Service Monitor: This Powershell script looks at many of my servers and the services that I deem "important". It emails me if it finds any that aren't running. It's a terrible script; I pretty much did a Get-WMIObject query, ran some logic with a where-object statement and then dumped it out to a file. The beast is 900 lines long or something ridiculous (done through copying and pasting, not by hand). I KNOW that there's a more elegant way to do it, but I haven't got the time to spend on it. It does what it needs to do.
• Website Monitor: This script downloads the html code from a couple of web pages, then runs select-string to return a true or false based on whether certain text appears in the code. If the expected code isn't there, the script emails the admins, because something's not right.
• The "Start Patching" script: We run this script before we start patching to turn off our monitoring software. First, it disables our SNMP monitoring software, then it stopd the Spiceworks service to take spiceworks offline. We don't want a bunch of emails that so-and-so system is down. The last thing it does is to disable the above two scheduled tasks, and let me tell you that this took me a bit to figure out. Yeah, you use powershell, but you use it to call a CLI program called schtasks.exe, which I think is pretty weird for Windows Server 2008 R2. You'd think they'd have some powershell commands for the Tasl Scheduler, but no. So here's the syntax for disabling a script:

 $CommandDisable = 'schtasks.exe /Change /S ComputerName /TN "Windows Service Monitor" /Disable'
Invoke-Command $CommandDisable

So first, I put the command into a string variable. Using single quotes forces Powershell to interpret the double quotation marks literally (as part of the string itself). Then I can use Invoke-Command to run the string as a command. Schtasks.exe has a ton of syntax options. Way back when I remember using it to set up scheduled tasks from scratch from a batch file; it's got really great functionality (too bad it hasn't been ported to powershell (grumble grumble). Every Windows computer has schtasks.exe, so if you're curious pop into a cmd shell and type schtasks /?. The help function is layered, so you can type schtasks /Change /? and get help specific to the subset of the command structure. Fun stuff.

Just some links

While I've learned a lot, I really haven't had much time to actually write about any of it. I'm in the process of rolling out Spiceworks as a helpdesk and inventory solution. It's kind of clunky to work with, honestly. My employer wants layers of helpdesk categories, and that functionality isn't built-in, so I have to rely on volunteer developers for a solution. I LOVE that people like to code, and I love free stuff (who doesn't) but in a year when I have to upgrade this those people might have moved on, and the project abandoned. I just don't have the time to get intimate with a solution right now, so I'm taking the position that if we want all the features of a for-pay helpdesk system, then we pay for it. If you want something nice and flat, then sure, we can do Spiceworks and save a few grand.

I'm also working on learning about our SAN, and our vendor (Mass Mountain) gave me a flash drive with their software on it (it's a 60 day trial once you create a volume group). You just boot from the flash drive and run it on whatever hardware you have lying around, which is pretty cool. I wanted to run it in VMware Workstation 8, where I have a VMware lab - 3 ESXi hosts in a cluster with High Availability and Distributed Resource Scheduling configured, a domain controller, a vCenter server, a FreeNAS iSCSI server, an OpenFiler NFS server, and a Server running Veeam that's backing up 3 servers running ON the ESXi cluster. Unfortunately, I had some problems getting VMware Workstation to boot from the USB drive. I found this workaround using a boot manager ISO called Plop, which did the trick.