by Frank Bitterlich
Let me tell you a story. I am sure that you have heard similar stories before, without doubt in the days around Halloween, and I am sure that the person telling it would swear that it was true. That it happened to someone they know, maybe a friend of a friend, or a cousin of their coworker.
I won't make such assertions. Of course, this story is complete fiction. After all, who would believe that such things could happen in reality? If I told you that, you would call me crazy, a lunatic. So, of course, none of this ever happened. Ever. Right?
It was hard to believe. If I were a superstitious person, I might have thought that higher powers were at work here. It always happened just before the end of my shift, and always on a Friday, when I was ready to go home for the weekend. Just one last look at my email inbox, and there it was, the dreaded [CRITICAL] Server monitor: alert in the subject line. For about two seconds I wondered whether I should open that email. Just shut down my iMac, pretend it arrived after I left. But... no. That probably meant that I would be called in tomorrow or Sunday, and the weekend would be messed up.
Server Monitor v3.5
Critical: Alert
Condition: Disk Full (0.3 % free)
Server: Accounting 1
IP address: 192.168.4.137
Location: Main Data Center, Rack 4
Device: /dev/disk4s0
Free space: 34 MB (0.3 %)Dammit. Again? The same server as last week? How would that disk get full so fast? But the cause was not as important. If I didn't fix that and get more free space on that server hard disk, the main web server would crash, and our company would be completely offline. That would get me into lots of trouble, as our clients expect to be able to use our services around the clock. The company expected me to prevent this from happening, to keep our servers running all the time. Even though they did not pay me nearly enough for that.
Okay, that should not be too difficult. Just log in to the server, delete some old junk, and make some room on the disk. I fired up the server control program, clicked on the symbol for that server, and clicked [Connect]. And... nothing happened. The program would not connect to the server.
"Ding"... another email.
Server Monitor v3.5
Critical: Shutting down
Condition: Disk Full (0.0 % free)
Server: Accounting 1
IP address: 192.168.4.137
Location: Main Data Center, Rack 4
Device: /dev/disk4s0
Free space: 0 MB (0.0 %)
*** Server shutting down ***My head almost hit the keyboard. "Oh god... why me? Why always on Friday? What have I done wrong, dear god?" You see, the servers are programmed to automatically shut down and turn off when a main disk has no space left at all; that is a safety feature so that no data gets lost when somebody tries to store something on that server. But it also meant that I had to go down to the basement, into our data center, and manually restart that server. Even though this is a big company, at this time on a Friday, close to midnight, there is hardly anybody here any more. And such a big building can get pretty spooky when it is completely empty. I sighed, grabbed my keychain, and went for the staircase.
Using the elevator late on a Friday night was out of the question. If it were to get stuck, I might be trapped in there until Monday morning - or, in this case, Tuesday. Monday was a holiday, and everyone was enjoying a long weekend. Except me.
So I went down the stairs from the 3rd story all the way to the basement, past several fire protection doors, and to the highly-secured door of the data center. To prevent unauthorised entry, the door had a magnet card reader, a fingerprint sensor, and a keypad where you had to enter the four-digit code only known to the systems administrators.
So I swiped my company ID card through the reader, entered the four-digit code (our clever security department changed the entry code each year - to the number of the year. So in 2007, you would enter 2-0-0-7 to get in), and pressed my index finger on the reader for a second. A double beep sounded, I opened the door, and I became pale in the face.
From one of the racks between aisle 3 and 4, there was clearly smoke coming out the top. It was not much smoke, but I knew right away what that meant. Probably another one of those crappy Hewlett-Packard server power supplies crapped out and decided to go with a bang. I was just barely in the room when the fire alarm went off - it was a very loud siren, and numerous red strobe lights flashed on the ceiling all over the data center. The smoke was coming out of the rack where the Accounting servers were housed. I grabbed the CO2 fire extinguisher from the wall and ran as fast as I could over to the smoking rack.
You know, most data centers these days have a special fire suppression system to prevent damage to the servers and data storage if a fire breaks out. When the smoke detectors go off, the power to all hardware in the room was interrupted, and then the whole room would be flooded with an inert gas called Halon. This would remove all oxygen from the room and thereby put out any fire within seconds, without the usual water damage to the electronics you get with normal sprinkler systems. But this has a nasty side effect: Since it displaces all oxygen, any human in the room would suffocate. The whole IT staff got extra training when the system was installed, that we had to leave the room as fast as possible as soon as the red-blue flashing warning lights of the Halon system went off.
I knew that I had only seconds to prevent that system from going off. If that happened, all systems would be shut down and our whole company would be a mess for weeks, until we get everything running again. And all that probably just because an old power supply shorted out and bellowed some smoke out the back of a server.
It was the topmost server, which I could clearly see because it was the only one whose lights were off. I took the extinguisher and shot a blast of cold CO2 into the top of the rack to put out any small fire that might have developed inside of the server, and when I did this, I heard a series of unpleasant noises from the rack on the other side of the one that I was at. It started with a crackling, sizzling sound, and finished in a loud bang and very bright flash. And then the whole data center went dark, and very, very silent.
A mere second after the lights went out, the emergency illumination turned on. It was enough to see where I was, but still it was very dark. The greenish light coming from just a few emergency lights scattered over the room made the many racks cast long shadows, so much that most of the data center was in the dark. But the most eerie thing was the silence. Where just a few seconds earlier the dozens of racks full with server and storage hardware had made just as much noise as a factory floor, now there was just silence. When it becomes that silent so quickly, you hear every remaining sound. I admit that I was scared. The shock of the loud bang when the lights went out was sitting deep in my bones. What had just happened? Somehow the main power supply must have been affected by the fire. I went to the other aisle on the back of the rack where the fire was to see what was installed there. I needed some light, so I fished for my iPhone in the back pocket of my pants, but it wasn't there. Damn it, left it on the desk. Great.
I slowly made my way through the semi-dark aisles. Just when I went left around the corner, I thought that I saw someone out of the corner of my eye, hushing int the next aisle. Somebody else in here? At this time? Impossible. All my coworkers said their goodbyes much earlier. And anyway, they would have made their presence known a long time ago. I said, loudly (probably way too loud), "Hello? Is there anybody in here?" and almost managed not to sound totally creeped out. – Nothing. No reply. It was probably nothing, just my mind playing tricks on me. There were still some sparks coming from that other rack, which made the shadows of the server racks jump back and forth a bit – quite normal to see things that aren't really there in such a situation.
When I arrived at the scene of the crime, I saw what was there. The Mitsubishi 300 KVA backup power supply. That was a system that was supposed to provide emergency power to the whole data center if there were a problem with the mains power. It is, effectively, built like an enormous car battery - the size of a large refrigerator - and could provide enough power to run twenty households. But only if one idiot system administrator didn't use a CO2 fire extinguisher to freeze the topmost battery module to below freezing and thereby made it explode with a loud bang.
Since the whole electrical power to whole data center went through this installation, now it had neither the normal mains electricity, nor backup power that the Mitsubishi would normally provide. Now all that came out of that cabinet were a few remaining blue sparks and some smoke. SMOKE! F*CK!!!
I had just finished that thought when a loud beeping noise, blue-red flashing lights, and an artificial female voice made me jump. "FIRE ALERT. HALON SYSTEM WILL ACTIVATE IN SIXTY SECONDS. EVACUATE DATA CENTER IMMEDIATELY. THIS IS NOT A TEST. EVACUATE IMMEDIATELY. HALON SYSTEM WILL ACTIVATE IN FIFTY SECONDS."
That's it, I'm out of here, I thought, and abandoned my heroic efforts to save the data center. I made a dash for the door, and froze. The normally illuminated button that would open the massive steel door was looking dark and dead. I literally felt the blood leaving my upper body and hide down somewhere in my legs and feet. I pushed the button. Nothing, no double-beep, no click to indicate the door was open. I pushed against the door. It wouldn't move. I threw myself against the door, nothing. Not a millimetre.
BEE-BEEP. FIRE ALERT. HALON SYSTEM WILL ACTIVATE IN THIRTY SECONDS. EVACUATE DATA CENTER IMMEDIATELY. THIS IS NOT A TEST. EVACUATE IMMEDIATELY. HALON SYSTEM WILL ACTIVATE IN TWENTY SECONDS.
I rushed to the phone of the wall next to the door, took the handset, but when I picked it up, I knew it was dead before I even put it to my ear. The silence coming from that useless phone handset was what pushed me over the edge. Now I was in full-blown panic mode. I had to get out here! If the Halon system activated while I was still trapped in here, I would be unconscious in about sixty seconds, and dead in three minutes.
I started banging on the door, and yelling for help. I knew that there was little chance that anybody would hear me, but it was my only chance. That was when somebody put a hand on my shoulder, and I freaked out.
When I felt that very large hand on my shoulder, I'm pretty sure that I screamed like a girl. I turned around and saw this massive guy standing there. He was large. Probably taller than two meters, and built like a cement truck. He wore a business suit with a tie, had a briefcase in his hand, and he had a panicky look on his face. "Who the f*ck are you? What are you doing in here?" I almost screamed at him. "You almost stopped my heart!"
"It doesn't matter who I am. Come with me! Hurry! I know a way out!" he said with a funny accent. What did that sound like - Swedish? Danish? Something like that. He grabbed my shoulder and practically pulled me with him. I didn't care who this guy was any more, if he knew a way out, we would be saved. So I ran after him.
BEE-BEEP. FIRE ALERT. HALON SYSTEM WILL ACTIVATE IN TEN SECONDS. EVACUATE DATA CENTER IMMEDIATELY. THIS IS NOT A TEST. EVACUATE IMMEDIATELY. HALON SYSTEM WILL ACTIVATE NOW. EVACUATE. EVACUATE.
He ran to the backside of the data center where the spare parts and unused equipment was stored. One of the large floor tiles that separated the room we were in from the large under-floor area where the massive trunks of data and power cables were installed, was slightly ajar. Something was jammed in there so that the tile wouldn't fall back into its place and become a seamless part of the floor again.
He grabbed into the gap and easily lifted the large floor tile, which must have weighed about 60 kilograms, as if it were a bit of cardboard. He jumped down into the space between the false floor and the real concrete foundation of the room, ducked down, and was gone. I took hold of the floor tile, and it nearly crushed me. How could that guy so easily lift that massive tile?
I jumped in after him, and saw where he had gone. Leading through the wall next to this tile, was a large maintenance shaft, where pipes, ducts and cables were running into the data center. But it still left enough space for even this massive, big guy. I followed him, crawling, through the shaft.
"I'm Gunnar. I work in Accounting. I was working on the quarterly financial report when the server went down. So I came down here. To fix it." he panted, while crawling through this shaft after him. "Fix it? Are you crazy? That's not your job, why didn't you call Systems Administration? And anyway, how did you get in here?" I asked him, somewhat confused.
"I make that everytime.", he replied in his strange nordic, broken English. When server goes down, I crawl in here to fix it. I go to server, delete old files, and reboot it. I works again then." In any other situation I would have been baffled, but the situation was so strange that this conversation almost seemed normal.
After a few more meters, the shaft we had been crawling through ended and we came out in a corner of the basement where I had never been before.
"So, you're telling me, instead of calling the IT department you break into the data center, 'fix' the server yourself, and reboot it?" –– "Yes, I must. IT department veeery slow. I need hurry. If I not get financial report done until tomorrow, I be fired! I must go now! I don’t want be fired!"
With that, he took off for the staircase, leaving me standing there, open-mouthed, and baffled. I didn't even get a chance to thank this guy. He saved my life! Who was that guy?
"Good morning everybody. I hope you all had a wonderful long weekend; now let's start with our weekly meeting. I think you all heard about the events that took place in the data center last Friday. We had some, well, technical trouble which took down practically all servers and caused a discharge of the Halon fire suppression system. Luckily nobody was harmed. Frank, the sysadmin on duty, managed to find a way out of the data center in time. You security guys will have a hard week ahead of you, management wants some answers on why the emergency exit system didn't work, and why there is an unsecured maintenance shaft running into the data center. Yes, Frank, you have something to say?"
"Well, actually, it wasn't me who found the exit. It was this guy from accounting..."
"Frank, I understand that these events have been traumatic for you, and maybe... you know, a few things... did not happen exactly as you remember them. I have checked with HR, and there is nobody matching your description working for the company."
"But I talked to that guy! He was in the data center! How would he get in there if he didn't have access to the building, anyway?"
"I don't know, Frank. The closest match to what you described would be that guy from Iceland who worked here a long time ago. I think it was in the 1990s, right? Anne?"
The lady from HR stood up. "Yes, that was Gunnar Olafsson. He worked here from 1991 to 2002. He was an accountant, and he was a pretty big guy. But there is no way that you could have seen Gunnar."
2002... that was long before I started working for that company. "Why? He told me that was his name."
"Well, you know, Gunnar had a bit trouble keeping up with his work assignments. He never finished the quarterly reports on time, and one day his manager had enough. He put him on notice. One more late report and he would be fired.
So, that one late Friday evening in October 2002, he was - once more - late with his report. So he worked overtime, did his best to finish the report. Nobody knows exactly what happened that night, but he seemed to have trouble with his server storage. The cleaning lady was the only other employee still in the building, and she said she saw him get in panic, and take off from his office for the staircase, hurrying down like a madman.
Monday morning, he didn't show up for work. His PC was still on, his desk in a mess. Nobody knew where he was. Monday afternoon a secretary went down to the basement storage room to get some office supplies. She found him on the bottom of the staircase, dead with a broken neck. Apparently he went down the stairwell, slipped and fell."
A few weeks after all that happened, I needed some CAT5 cable to fix a Wifi hotspot in the lobby, so I went down to the data center, where we kept the surplus cable.
When fetching the cable I bumped my foot on a tile and almost stumbled. The tile that literally saved my life by providing an emergency exit. It still was slightly ajar, so I tried to close it -- but it wouldn't drop down. There was something still crammed into the gap, so I lifted it and took out a small metal object. It was a Zippo lighter.
I turned it this way and that way. It was a bit dented, almost as if it had been used as a "door opener" for that tile many times. I held it into the light and saw that it had an engraving. It took me a while to make out what it said from all the dents and scratches. But when I did, I dropped it to the floor. I calmly left the data center, went to my desk, fired up my email program, and wrote my termination notice. There was no way that I would work any longer in this building.
The engraving said:
From the Accounting Dept.
To Gunnar Olafsson
10 Years Anniversary
You can submit a comment or question related to this blog post. Your post may be subject to moderation and therefor not appear immediately. Please be patient.