Friday, February 5, 2010

Oh crud -- RAID just failed at work

Crud is not the word I'm thinking but no sense being vulgar in the title. We are having RAID fun at work this week. Monday we switched back to our old raid server from a unit borrowed from the sister group. Well it was painful, NFS was off in some way only sys admins understand and it made all the processing slow because disk writes etc were taking forever. It got into a loop of too many timeouts and failed early tuesday morning -so the sys admin managed to switch NFS back to the borrowed raid remotely on his day off and processing was easy to restore.

Today the sister group had lunch out and then a meeting at the pub, and I was dinking around in an ate too much fog (I skipped beer with lunch but it didn't help much) I tardily checked on the routine processing and found the RAID failure at 3:20 pm. Its not so bad for me since I come in and work late anyway but that is horrible timing to announce a big fat problem to the sys admin who likes to come in and LEAVE early. Ugh ugh ugh.

Now I'm spinning my wheels. Stressed that I didn't catch things in better time but with no action that I can take on the immediate problem, I have to wait until the sys admin decides whether he thinks he can fix the failed RAID quickly or would prefer to switch the processing CPU's to looking at the old RAID. I hope I can get things going tonight so I can enjoy saturday. We are supposed to get some snow and a big dose of COLD air sunday-tuesday, then wednesday I'm committed to attending a workshop and have to arrive in Boulder extra early.

No comments: