Drunken Blackout
I wonder if this story about a drunken employee taking out 40 racks in a datacenter was ever considered part of a threat scenario?
I see the convergence marketing stories converting this from “urban myth / valleywag rumor” to “”irrefutable proof” within days.
Trivia: I used to work in the building behind this datacenter.
July 25, 2007 at 9:29 pm
Actually, the story going around is that a lot of quick “mini-blackouts” caused the flywheels of the generators to not get up to speed each time and therefore not give enough time for the switch-over
http://valleywag.com/tech/breakdowns/365-mains-credibility-outage-282257.php
http://www.datacenterknowledge.com/archives/2007/Jul/24/generator_failures_caused_365_main_outage.html
I dont exactly believe the “drunken employee” rumor, but it’s very difficult to protect against a malicious insider (and not just in this case).
July 25, 2007 at 10:17 pm
valleywag made up the whole “drunk employee” story. 365main wouldn’t have a “drunk employee” - all their staff is top-notch.
a transformer blew up and caused voltage fluctuations, which caused power outages across the entire neighborhood. this isn’t the first time that 365main has had a power outage. heck, the whole internet blows up about misinformation regarding a power outage in CA? remember when CA used to have these everyday?
the problem isn’t with 365main, or with CA power. the problem is with your average CEO/CTO’s ability to make decisions. they suck. they make horrible decisions that don’t make any sense, especially when it comes to data centers. clearly the people providing this information to them (Gartner, McKinsey, et al) are providing snake-oil information. I love Gartner’s IMM (Infrastructure Maturity Model), but when are they going to provide information about actually implementing such an idea to fruition without providing mis-information?
i used to spend a lot of time at 365main, but i always did like 200paul better. of course, south beach and bayview have a little bit of a different atmosphere, most businesses prefer the non-ghetto establishments. even with earthquake reinforcements, i still think it’s a bad idea to co-locate in sf or anywhere in CA for that matter. there are plenty of nice data centers here in chicago! i know of two new ones opening up in the next 1-2 months, even.
i also hear that there are 235% more programmers in Chicago than all of Silicon Valley. there are lots of good hosting companies in NYC, DC, and Chicago that provide serious stability. eBay and Google went to places like Phoenix and North Carolina to build their new infrastructures. what’s wrong with these Bay Area decision-makers? oh yeah, it’s probably the kick-backs. we need more IT whistle-blowers.
July 26, 2007 at 3:28 pm
Not sure if this is reflected in the hosting rates, but Chicago has quite expensive electricity. Totally agreed that we have some pretty spiffy data centers, though.
July 26, 2007 at 7:39 pm
http://en.wikipedia.org/wiki/Quincy,_Washington#A_role_in_the_high-tech_economy
For large datacenter consumers, moving services to PoDunk Nowheresville, USA is a no-brainer when you can get cheap and plentiful power. Power is, by far, the most expensive and elusive commodity at most datacenters in urban areas.
When you’re talking a scale of thousands or tens of thousands of machines, the extra cost of recruiting and retaining qualified employees is minimal. You already have to pay a set amount for datacenter support staff, moreso in the better datacenters or larger markets. By increasing that cost even as much as 50% in order to keep the best of the best on staff, you are still well ahead of the game.
Bonus points for locating your datacenter somewhat near remote resort communities or “cool” attractions. Quincy, WA has these features in the form of Crescent Bar, WA (resort community between two dams on the Columbia River - think lake) and the Gorge Amphitheater which is the top destination for mega-concerts in the state.
Keep the geeks happy, pay them with urban market compensation packages, and get screamin deals on rural facilities. You can beat that formula.