High Scalability has a great link to a video TechTalk with Cuong Do, YouTube’s engineering manager. He talks about the challenges YouTube faces (past and present) to meet it’s skyrocketing user demand, as well as the infrastructure that allows them to scale. I enjoyed the anecdotes: especially the frantic email sent at 2am alerting the dev team that they only had 3 days of storage left… I always thought Google/YouTube would be immune to emergencies like that… ignorance on my part 
(requires Adobe Flash plugin… click HERE to watch it on YouTube)
I found this information interesting:
- The application code is written mostly in Python (the web app is not the bottleneck… the database RPC is)
- They use Apache for page content and lighttpd for serving video
- Thumbnails are now served by Google’s BigTable
- They’re running SuSE Linux with MySQL
- HW RAID-10 across multiple disks was too slow. HW RAID-1 with SW RAID-0 was faster because the Linux I/O scheduler could see the multiple volumes and would therefore schedule more I/O
You can read a good summary of the talk HERE from the High Scalability website.
Okay, so I’m on this research kick to investigate the infrastructure solutions of some of largest distributed systems in the world. Given that most of those are internet companies… well, that should explain the trend in my posts (more to come).
Baseline has a good article (albeit a little dated… 2006) describing Google’s server infrastructure and some of the history behind it. Google really pioneered the use of COTS (Commodity Off-The-Shelf) hardware to reduce cost; an idea that works great in some situations. They’re up to around 450,000 servers worldwide now… quite impressive. Click HERE to read the full article.
Here’s an except I found interesting regarding their infrastructure history:
Google and its information-technology infrastructure had humble beginnings… when the server infrastructure consisted of a jumble of PCs scavenged from around campus.
“But this is the start of the story,” he adds, part of an approach that says “don’t necessarily do it the way everyone else did. Just find some way of doing it cheap and effectively—so we can learn.”
By 1999, the Google.com search engine was running in professionally managed Internet data centers run by companies like Exodus. But the equipment Google was parking there was, if anything, more unconventional, based on hand-built racks with corkboard trays…
His team assembled racks of bare motherboards, mounted four to a shelf on corkboard, with cheap no-name hard drives purchased at Fry’s Electronics. These were packed close together (like “blade servers before there were blade servers,” Merrill says). The individual servers on these racks exchanged and replicated data over a snarl of Ethernet cables plugged into Hewlett-Packard network switches.
… corkboard, huh… not a good idea:
Later Google data centers tidied up the cabling, and corkboard (which turned out to pose a fire hazard) vanished from the server racks.
Lately, I’ve been reading some interesting papers from Google labs about their back-end file system, database, and algorithm framework. What I found most intriguing is the simplicity of their design. The software is highly specialized for their particular application… it follows the “do one thing and do it well” philosophy. It was nice to see that they correctly balanced simplicity against feature set.
