Friday, November 20, 2009

Job (hopefully well) Done

For those of you whose eyes gloss over when the talking turns to a technical, here's an executive summary for non-nerds:

Today was my last day at my now-former company. This was a tough two weeks, and I unfortunately got done very little of what I set out to do.
Non-nerds can stop here, if you wish, as the rest of this post will describe the last two weeks of my employment. You have been warned...

I gave notice of leaving the company on Friday, Nov 6. I had planned to spend the next two weeks cleaning up (and in many cases writing) varying documentation for which I'm responsible, so that those who have to pick up where I left off could make a smooth transistion. As my main job involved testing our application, I figured on fleshing out as much of the testing plan that is rattling around in my head as possible. That plan lasted until Monday when I got to work and found that I had no internet connection. You see, one of the other hats that I have been wearing is IT Brigade member, so I set to work figuring out what went wrong. It didn't take long. Our domain controller, which for varying and sundry reasons did not have a backup, had died over the weekend. Domain logins, DNS, DHCP, all gone. And since the domain controller had died a rather permanent death, recovery wasn't going to be easy, especially since none of us IT Brigade folks actually have any idea how any of those things mentioned above really work.

First thing was first, try to restore some semblance of connectivity, at least within the office. We had no spare hardware anywhere, other than a 32-bit Ubuntu box which was designated to be a new firewall. We really would have liked to have installed a Windows server, since that had been what we'd used before, but we only had available licenses for a 64-bit server, so for the time being, we did the best with what we had. My coworker and I managed, through a lot of trial and error, to get DHCP and DNS mostly working by the end of the day, but we still had no way of getting people logged into our network. We spent part of Tuesday trying to figure out how to turn our new DHCP/DNS server into a domain controller, but ended up going out and purchasing a new 64-bit box anyway (what a bargain, managed to get a low-end system for under $300). Got the domain controller set up by the end of Tuesday using our 64-bit server license, then spent Wednesday through Friday moving DNS/DHCP over to the Windows server and trying to configure the Ubuntu box to be our firewall and VPN server. By Monday, we had most everything mostly working. Then came Tuesday.

Just before lunch on Tuesday, people started complaining that they were losing their internet connections again. We spent the rest of the day Tuesday poking and prodding everything with absolutely no success, and were finally successful in getting permission to bring in outside help. Wednesday was spent with the consultant we brought in, and by the end of the day, we'd gotten our network talking to itself again, and everyone once again connected to the outside world. Thursday and today were spent tweaking things so that the VPN worked, and the servers which need to be externally accessible still were.

So, all those tests and other documentation that needed to be written? Not so much. I was able to do a little verbal and demo-style knowledge transfer, so not everything is lost. But the coworker that is taking over for me is smart. Way smart. Smarter than I've ever been, even when I was smart. I think he'll do OK. And he has my number, just in case.

Here's hoping that the new job has no networking responsibilities for me...

2 comments:

Anonymous said...

Welcome to the world of the full time IT geek. Those are the kind of things they pay me to avoid.

Sorry to hear that your last week as not as you planned and here's hoping that the new job starts of well.

--ken

Darrin said...

ooooooooof what a pain. at least you set out with the best intentions. some people would have just sandbagged. i'm sure you had fun those last 2 weeks though. networking is SOOOOOO much fun! thank goodness it's all in the past.