Thursday, April 30, 2009

The devil is in the details

One of my first real jobs was for a mainframe computer company that designed systems for the stock market and all of the major banks. They were extremely complex computers designed to never fail.

One weekend after working crazy hours all week, I dialed in from home to perform some clean up and maintenance. I had to delete some files and was tired and accidentally typed the asterisk, which in computer language means "everything."

That was how, on a Sunday afternoon, I deleted the contents of 1200 people's email.

I called my boss in a panic and he calmly told me that it was okay, I would just have to execute the recovery programs our systems were famous for. I took a deep breath, threw on some shoes, and drove to the office.

I took the disks offline and prepared for the recovery process. I had never done it, but had taken several classes and had some notes to follow. It started out okay, but then one of the tape drives started acting up. It kept spitting out the back up tapes and would take sometimes three tries to load. Restoring the files took nearly eight hours. I kept a watch on the system and was relieved to see the data being replaced. Finally in the wee hours of the morning it was finished. I tried to start up my email and to my complete horror found that the files were empty again. I couldn't call my boss so I had no choice but to phone our company emergency systems hot line which was manned 24/7 from all corners of the earth.

A smooth Australian voice answered and we walked through what I had done.

Then he asked if I had used the "to first purge" command. This basically meant, restore the system file until just before a delete command was executed. Three little words. Kind of a big deal.

I managed to start the whole thing over and get the files back online by 10 AM on Monday. My boss didn't fire me, but actually admired that I was able to recover the system since the process was so complicated.

I drove home delirious, but sort of proud of myself.

And isn't it always like that? Just something small that changes the entire thing? Sure the devil is in the details, but still you stop and try again.

A few words, a sentence, a paragraph even, that becomes the difference between failure and success.

1 comment:

latisha said...

damn those details.