Actually, You Can Stop The Signal For A While
Irradiated by Stingray
In the spirit of explanation (seriously guys, the concern is appreciated, but if we go down we go down don’t pester other folks’ comments trying to internet-detective us- we’ll say so if we’re pulling the plug deliberately) what happened is someone’s account on the server we run the site off got hacked and rooted, and Jihadi Joe killed the server’s virtualization setup so the couple dozen sites that would normally get sorted and routed in software all got dumped to one site for the whole server, full of the message of islam’s peaceful and loving nature and how awesome it is to hijack servers other people are paying for, and then tech support at the colo shoved their heads up their asses and have taken the better part of a week to do what should’ve been a hour or two fix because it does not take that fucking long to change out a hot-swappable drive and re-image it.
And the first time I told that to a non-techie friend while we were down, I’m told all he heard was Charlie Brown’s Teacher, so for our non technical readers, here’s your version of what happened.
Back in the before time, it was one website per special giving-stuff computer, or “server.” Then someone got clever and came up with a way to put a bunch of websites on one server, and change things around based on special little bits of information that the users never know about, just as a basic part of how they ask to see a website. So theoretically, me, Matt, Sal, NFO, FarmGirl and Tam could all have our blogs on one physical special-giving-stuff-computer machine, but the magic elves would make sure Tractor Tracks sent out info on how not to piss off your cashier, and booksbikesetc would send out information on obscure WW1 pistols that six people have heard of including the designer, NFO’s website would give you naval aviation jokes, and so forth.
Then, one fine Saturday morning, Jihadi Joe killed those magic elves, so they couldn’t sort things to make sure the right info got to the right people, and all that particular machine’s elves knew how to offer up is Peace & Love & Stonings & Hijacking and yi-yi-yi music. People wanted their obscure WWI pistol information, and other people wanted their over-long scientific analysis of pop phenomenon and pointless profanity but there were no elves that knew where that douchecocking information lived anymore!
Then the stupid doubled-down. The tech monkeys at the place where the special-giving-stuff computer physically lives couldn’t figure out how to pull a small box out of a slot, insert a new box that looks just like it into the now empty hole, and press a button to trim and shape and form that part to behave as it’s supposed to. Really, they made a monkey fucking a football look dignified, elegant, and capable of all sorts of rocket surgery.
And now we’re back. The end.
November 16th, 2011 at 12:13 pm
Welcome back.
Wah-wanh.
November 16th, 2011 at 12:34 pm
So that’s what h4XX0r3d means!
November 16th, 2011 at 12:46 pm
Good to have you back.
That is some epic colo fail, right there.
November 16th, 2011 at 12:58 pm
I want to see the monkey.
November 16th, 2011 at 1:16 pm
“Really, they made a monkey fucking a football look dignified, elegant, and capable of all sorts of rocket surgery.”
Quotable quotes.
Glad you’re back.
November 16th, 2011 at 1:33 pm
Ah, cloud computing.
November 16th, 2011 at 2:27 pm
One of the funniest things I have read this week.
November 16th, 2011 at 2:57 pm
As annoying as Blogger/Google can be … they don’t go down. Even to DOS attacks.
Unless you are stupid about passwords, you’re blog there ain’t gettin hacked.
November 16th, 2011 at 2:57 pm
your, even.
( snarl )
November 16th, 2011 at 3:47 pm
Allah Hu Akbar!
That was all it took to get me to convert! Good on them!
Glad nothing was damaged in the attack.
November 16th, 2011 at 3:50 pm
I once worked for a web hosting firm. This web hosting firm had thousands of servers. All running a very, very out-of-date Linux distro with so many weird, custom, local patches that we just renamed it. All the key supporting infrastructure was built out of the digital version of spit and baling wire. We had a backup system, but except on the newest and emptiest of those servers, it didn’t work right, because the filesystems always filled up halfway through and gorked the whole process…users who screwed up and deleted the wrong thing could get their files back about half the time, but if the server got hacked or we lost a drive…well, we’d have been reduced to prayer, probably. RAID? Surely you jest, sir! Hell, as far as I know they _still_ don’t have hot-swap drives on most of their systems…they certainly didn’t have any of them when I left.
I was the night-shift sysadmin for North America. Final tech authority over five data centers, from 6pm until 9am every weeknight, plus round-the-clock one weekend out of every four.
It wasn’t as sucktastic as it sounds…it was way more sucktastic than it sounds.
And I can say with absolute certainty that if something like this had happened to _us_, and it had taken the better part of a _week_ to fix it, they’d have _fired_ my ass way before the end of the second day.
And I’d have deserved it, too.
Your hosting provider is too tolerant. Any place that makes my ex-employer look like a paragon of reliability by comparison is far, far too tolerant.
November 16th, 2011 at 3:51 pm
Yay, you’re back!
And I am so quoting that….
November 16th, 2011 at 3:59 pm
Matt: It took the provider (I work with him on some other stuff, so I got the behind-the-scenes version) basically having to threaten to kidnap the guy at the colo’s kids and take them to Penn State to get anything done. Once he got that far and finally got a techie, it was fixed inside a half hour. I’m gonna be leaning on him to find a better colo. And looking around to see if there’s anywhere else that’ll do as good or better a deal. We do this for a hobby, we’re not beholden to make sure people see ads or anything like that, so if we’re down it’s basically just an unplanned vacation from writing stuff. That said, it still annoys the piss out of me to have a problem resolved this badly on the colo end.
November 16th, 2011 at 4:30 pm
Finally! I UNDERSTOOD your explanation of how computers work.
Good to see you guys back. I was about to start a rumor that you left the country to work on Iran’s nuclear weapons program.
(Those godamn elves…I swear they hate my guts)
November 16th, 2011 at 5:32 pm
So, what I’m taking away from this is that Islamic jihadists hate elves. This is in addition to all of the other good stuff they hate like bacon and the Victoria’s Secret catalog.
These people must be stopped.
For the elves!!!
BGM
November 16th, 2011 at 6:46 pm
The person brokering the colo space probably had to pay the server farm techie out of his own pocket to fix this crap.
Even having the NOC monkey change a cd in a drive costs money.
So there is a big incentive to pretend nothing happened.
You might want to deal with the server farm directly, instead of someone who loses money when problems don’t just sod off and go away.
November 16th, 2011 at 7:29 pm
He killed the elves??! That bastard!!1!!
Probably used a priv escalation ‘sploit to do it, too.
November 16th, 2011 at 8:42 pm
I was wondering why you guys were down so long. Glad to have you back.
The technical version was more comprehensible, but maybe that’s just me.
November 16th, 2011 at 9:40 pm
So…. Do we has a target???
Welcome back!
November 17th, 2011 at 5:25 am
Bummer. I would definitely be looking for a new colo at that point.
Fair bit of islamohacking going around lately. This makes the third one I’ve seen in as many weeks.
November 17th, 2011 at 6:22 am
[…] The nerds got hacked […]
November 17th, 2011 at 8:02 am
I knew there were magical elves in the computer. They laughed, they laughed, but I knew the TRUTH!
Also when you turn off your computer when it’s not working it fixes it because the magical elves were tired and needed a nap. *nods*
November 17th, 2011 at 9:11 am
Just for the record, didn’t intend to cast any implied blame in your direction. Just bitching about low quality of service among people who are (presumably, anyway) getting _paid_ for hosting this.
You, of course, are not getting paid to blog, and thus owe us fuck-all. Anything greater than that is much appreciated by us. (Well, me, anyway…and it sounds like a bunch of others too.
)
Anyway, welcome back.
November 17th, 2011 at 10:47 am
No offense taken, Matt. Just tossing a couple cents into the explanation about why my case of the red-ass isn’t as bright as it might should be over all this. Two other people I know on the same server have business websites on this box. If I’d been in that boat, I’d have already moved hosts.
November 17th, 2011 at 11:09 am
Good website. I love it.
November 21st, 2011 at 6:28 pm
Glad to have you back.