When I started at my current employer during the summer of 2000, we had 4 servers, three of which had with thier own 20G DDS4 tape drive managed by independent installations of ArcServe.  Combined, we had less than 50G of data being backed up by the daily full backup.  The backups were usually flawless and completed in a few hours at most, and sending a monthy set offsite was no big deal and that set typically contained the data set from the last day of the month.  I miss those days.

By 2002 we were growing our datacenter at a rate never imagined, and we quickly outgrew our backup solution.  Faced with the decision to maintain multiple backup platforms or consolidate, we consolidated.  Exit the internal tape drives on each server with multiple intallations of ArcServe, enter the AIT-3 library with Backup Exec.  This solution was great and allowed tapes to be swapped weekly rather than daily, and all backups could now be managed from a single instance of the backup software.  It was great!  We still didn’t have an unmanageable quantity of data to be backed up so scheduling of weekly/monthly/yearly sets was no hassle at all.

By 2004 our data growth was moving right along and we needed more capacity.  When we bought the AIT-3 library we got AIT-2 tapes (on purpose), allowing for capacity increase by switching over to the larger AIT-3 tapes.  New tapes, problem solved.

In the spring of 2006 we had outgrown the AIT-3 tapes and were once again forced to upgrade the backup infrastructure.  The new game in town?  LTO-3.  With massive capacity and screeaming fast transfer rates, this was the solution I needed.  We purchased an Overland Storage library with 17 slots and 2 LTO-3 drives, and we were back on track with plenty of headroom to grow into.  This would become a double-edged sword.

We currently have just shy of 50 servers and a backup set well north of 2 TeraBytes.  The time window required to back up our data is a problem all by itself, but the real-world challenges go well beyond that.  In late 2007 I began attending some backup schools and data storage seminars, and learned that data theft via backup tapes was running rampant!  Clearly, I needed to start encrypting our tapes.  I did, but this came with some caveats, most importantly that encrypted data does not compress.  Our backup got more secure, but got slower and used more tapes at the same time.  The time window issue was now much worse, but for the sake of my job security I would not go back to un-encrypted tapes. 

We began evaluating other backup solutions, of both hardware and software varieties and two stick out in my memory as being noteworthy.  Data Domain offered data deduplication for backup-to-disk solutions and while we loved it we really wanted to make better use of our existing infrastructure rather than replace the hardware altogether.  Next up on the Software side was CommVault.  I can’t say enough great things about this product and it looks like it will do everything we want – deduplication for disk backups, easy automation of set duplication to tape for offsite storage, and easy integration with their file server data archiving solution.  The problem is that I can’t get any significant money to pay for any of this for the forseeable future. 

I’m now stuck in a reactionary position with the backups, and we are slowly sliding away from having an acceptable backup solution.  The backup schedule is now at the mercy of the weekday schedule, rather than the calendar week/month schedule, because the full backup takes so long to run that we have to start it Friday night.  This means that month-end finance data is not necessarily on a monthly tape set unless we manually move data around to ensure that the right data is on the right tape, and we get another step further from automation.

In hindsight, I feel like dropped the ball as far as making sure that our backup solution could keep up with our network growth, but at the same time we were doing a great job of keeping the two in sync for so long.  It was working great and then BANG it just wasn’t anymore, and the cost to get on track is now more than we can spend.  We are about to buy some used dedicated direct-attached SCSI storage on Ebay to attempt running a ”backup-to-disk with a duplicate-to-tape” scenario without deduplication, but this is just a band-aid and can’t be considered a long-term solution.  We will be forced to take advantage of advanced technologies like deduplication in order to survive the future.

• Previous Entry: OpenDNS is DOWN!!!
• Next Entry: Disaster Recovery Simulation