Okay, so I’m on this research kick to investigate the infrastructure solutions of some of largest distributed systems in the world. Given that most of those are internet companies… well, that should explain the trend in my posts (more to come).

Baseline has a good article (albeit a little dated… 2006) describing Google’s server infrastructure and some of the history behind it. Google really pioneered the use of COTS (Commodity Off-The-Shelf) hardware to reduce cost; an idea that works great in some situations. They’re up to around 450,000 servers worldwide now… quite impressive. Click HERE to read the full article.

Here’s an except I found interesting regarding their infrastructure history:

Google and its information-technology infrastructure had humble beginnings… when the server infrastructure consisted of a jumble of PCs scavenged from around campus.

“But this is the start of the story,” he adds, part of an approach that says “don’t necessarily do it the way everyone else did. Just find some way of doing it cheap and effectively—so we can learn.”

By 1999, the Google.com search engine was running in professionally managed Internet data centers run by companies like Exodus. But the equipment Google was parking there was, if anything, more unconventional, based on hand-built racks with corkboard trays…

His team assembled racks of bare motherboards, mounted four to a shelf on corkboard, with cheap no-name hard drives purchased at Fry’s Electronics. These were packed close together (like “blade servers before there were blade servers,” Merrill says). The individual servers on these racks exchanged and replicated data over a snarl of Ethernet cables plugged into Hewlett-Packard network switches.

… corkboard, huh… not a good idea:

Later Google data centers tidied up the cabling, and corkboard (which turned out to pose a fire hazard) vanished from the server racks.

I’m also interested in their “living out load” philosophy. Google’s custom project tracking software facilitates a greater sharing of knowledge between engineers. So often, engineers within even a single department have no idea what others are working on:

If Google employees found the project tracking system to be a hassle to work with, they probably wouldn’t use it, regardless of whether it was supposed to be mandatory, Merrill says. But because it’s as easy as reading and responding to an e-mail, “We get pretty high compliance.”

Those project tracking reports go into a repository—searchable, of course—so that managers can dip in at any time for an overview on the progress of various efforts. Other Google employees can troll around in there as well and register their interest in a project they want to track, regardless of whether they have any official connection to that project.

“What we’re looking for here is lots of accidental cross-pollination,” Merrill explains, so that employees in different offices, perhaps in different countries, can find out about other projects that might be relevant to their own work. Despite Google’s reputation for secrecy toward outsiders, internally the watchword is “living out loud,” Merrill says. “Everything we do is a 360-degree public discussion.”

On a more recent note, it appears Google is building their own network switches now using SPF+ for the links. Click HERE to read more.