Last year while I was on Reddit, I saw a post this laptop in a closet that was a server. At the time, I wrote this comment, which to this day is my highest rated comment on Reddit. I decided to reproduce it here, because it’s probably one of my top 3 “war stories.” Some edits have been made to correct a few misspellings and deliberately anonymizing embellishments.
I’m told this sticky note on top of this laptop says “Don’t touch, it’s a server”.
This happens more frequently than you would imagine.
Someone installs a mail server in a laptop, and shoves the laptop in a closet just because it’s the fastest way to stand something up for some new system that’s months behind schedule. The new system goes to production, and everyone who was involved in the project moves on to something else.
Years go by, and everyone forgets about the laptop in the closet. Decades go by, and it’s time to replace the system because it’s end of life. The new engineers come in to do requirements gathering, and ask “So, this thing sends mail… where’s the mail server for that?” It’s not postfix on the application server. It’s not the Exchange server. It’s not in the cloud. It’s a local IP somewhere in the East Overshoe office. “They have servers in East Overshoe?” someone in IT asks. “I thought all their servers got moved to the cloud.” Eventually, someone has to drive out to East Overshoe, to go on a search for the missing server. Nobody can find it. None of the people working there know what servers are; this is the finance department. “Oh, I don’t know what that cable going in that closet is. Didn’t you guys do that? We don’t have the key to it.” Two hours later, someone gets the door open, and behold, the mail server the entire system depends on is an ancient laptop that just never restarted itself or lost power. Because it’s a Windows XP machine. In 2019.
I was the guy who asked “where’s the mail server for that?” in the story. We were replacing a legacy system, which I think was originally installed in the mid-2000s. The other names were changed to protect the guilty.
One reader pointed out that there’s nothing more permanent as a temporary solution. Indeed. That might actually be the moral of this story. Possibly the moral of the entire project that the story came out of. I have good war stories from that project. I might write more of them.
Another commenter pointed out that this XKCD was relevant. This comic is absolutely correct. The gal in the right panel has a bright career in front of her. The one in the left panel has the time to make perfectly extensible stuff because she doesn’t have any users to tend to.
An especially pedantic reader was impressed that a POP3 server could send mail. To be fully accurate, it was both an SMTP MTA and a POP3 mailbox. It was some crazy proprietary mail server, which I think was chosen by the vendor of the old solution. “Crazy” in that I had never heard of that company before or since; it sounded like something you’d order from a print catalog in the 90’s an install from a stack of 5 shareware floppy disks because downloading it with your 14k baud modem would take too long. We replaced it with postfix and dovecot on the application server.
I have a few more of these to self-plagerize. Then maybe I’ll write new ones every once in a while. I’ve worked long enough now to have a few good stories before I get boring.