YouTube’s Architecture and Scalability

Tech and Security No Comments

High Scalability has a great link to a video TechTalk with Cuong Do, YouTube’s engineering manager. He talks about the challenges YouTube faces (past and present) to meet it’s skyrocketing user demand, as well as the infrastructure that allows them to scale. I enjoyed the anecdotes: especially the frantic email sent at 2am alerting the dev team that they only had 3 days of storage left… I always thought Google/YouTube would be immune to emergencies like that… ignorance on my part :-)

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

I found this information interesting:

  • The application code is written mostly in Python (the web app is not the bottleneck… the database RPC is)
  • They use Apache for page content and lighttpd for serving video
  • Thumbnails are now served by Google’s BigTable
  • They’re running SuSE Linux with MySQL
  • HW RAID-10 across multiple disks was too slow. HW RAID-1 with SW RAID-0 was faster because the Linux I/O scheduler could see the multiple volumes and would therefore schedule more I/O

You can read a good summary of the talk HERE from the High Scalability website.

TechCruch has a good article of the YouTube API.

Inside the Lego Factory

Oh So Random No Comments

Lego LogoYeah, Slashdot picked this up already, but it’s still worth posting. Gizmodo has a set of 3 videos that show the workings of the Lego factory. They also have a tour of the secret Lego vault.

This video shows something that very few people have had the opportunity to witness: the inside of the Lego factory, with no barriers or secrets. I filmed every step in the creation of the brick. From the raw granulate stored in massive silos to the molding machines to the gigantic storage cathedrals to the decoration and packaging warehouses, you will be able to see absolutely everything, including the most guarded secret of the company: the brick molds themselves.

Click HERE to read the full story and watch the videos.

Click HERE to watch the tour of the Lego secret vault.

C/C++: Using Bitfields Effectively

Code Monkey 2 Comments

Introduction

If you’ve ever done embedded development in C/C++, you are probably familiar with bitfields. They are a handy way to reference individual bits in things like hardware registers. The problem is that bitfields can lead to performance problems and race conditions if not used properly. I hope to highlight some of the issues you should consider when using them.

Usage

First, let’s assume you need to check various fields in a hardware register with the following layout:

Bitfield Register Example

You could define the following bitfield to represent this register:

1: struct HwReg
2: {
3:    unsigned int Base : 16;
4:    unsigned int Offset : 8;
5:    unsigned int Rsvd : 5;
6:    unsigned int Flag : 1;
7:    unsigned int Type : 2;
8: };

The total size of this data type is sizeof(unsigned int), with each line defining a different region (field) within that type (this looks confusing when you first look at it). The following code uses the HwReg bitfield to access a memory-mapped register:

1: struct HwReg* pReg = (struct HwReg*)0×80001005;
2:
3: if (pReg->Flag && pReg->Type == TYPE_1)
4: {
5:    void* address = pReg->Base + pReg->Offset;
6: }

Line 1 defines a pointer to the physical hardware register as type HwReg. We can now use this pointer to easily access the register fields. If this isn’t clear, you can read more about bitfields HERE.

Performance Problems

The compiler doesn’t know how to optimize bitfield accesses (especially because the pointers to memory-mapped hardware registers are almost always declared ‘volatile’). This means that every access to a member of the bitfield will require a read of the physical hardware register. This can be orders of magnitude slower than accessing main memory. In the code example above, the hardware register will be read 4 times; once for each field access.

The way to remedy this is to cache a copy of the register value and then operate on that. Consider the following code:

1: unsigned int* pFullReg = (unsigned int*)0×80001005;
2: unsigned int temp = *pFullReg;
3: struct HwReg* pReg = (struct HwReg*)&temp;
4:
5: if (pReg->Flag && pReg->Type == TYPE_1)
6: {
7:    void* address = pReg->Base + pReg->Offset;
8: }

Line 1 defines a pointer to the physical hardware register. Line 2 performs the actual read into a local variable (the slowest part). This local copy is now in main memory and the CPU cache. Line 3 casts the cached value to the bitfield for easy access. Finally, all accesses to the register fields is on the cached value, which can be read very fast from L1 cache.

Another advantage to this approach is when the hardware requires locking before the register can be accessed. By caching the value, you can keep all the locking code localized to a single area of the function. Without caching, you would hold the lock for a longer period of time (possibly forcing other operations to block) and have to make sure to release the lock on every return path (more difficult with exceptions).

NOTE: Remember you are only working with a copy of the register value. If you update a value in the bitfield, you must still copy the updated value back to the register.

Race Conditions

As stated above, each access to a field value generates its own read/write operation. Even if the CPU architecture guarantees that an individual operation is atomic, updating multiple fields are not. Thus, in a multi-threaded application you must lock the entire block of code that operates on the bitfield. I again suggest caching the value, as you only need to lock the actual read/write of the entire register.

Conclusion

Bitfields are a nice language construct that can help make it easier to write clean code (as opposed to using macros and bitmasks). Unfortunately, it’s all too easy to shoot-yourself-in-the-foot with bitfields if you don’t understand the pitfalls. As always, use caution when writing performance-critical code and make sure you understand how to use the available code constructs.

Happy coding!

Interview with Mastercard’s Rob Reeg

Tech and Security No Comments

Mastercard LogoCIO has a good interview with Rob Reeg, president of Mastercard’s Global Technology and Operations. He discusses their infrastructure and processing architecture. Definitely worth looking at if you’re interested in how credit card transactions are processed.

Interviewer: How big of an infrastructure do you have to support and maintain? It must be huge.

Reeg: Actually from a pure server footprint standpoint… we probably have fewer actual footprint servers because of techniques like virtualization that help us leverage one box to do multiple things.

Where it gets interesting is philosophically: We try to put [transaction] processing as close to our customers, the banks, as possible. When we talk about the global network, we have small servers that sit with the bank customers that connect to our network. What it does is it gives us intelligence there at the end of the network. So as a transaction comes through, we can take a look at that transaction and decide how do we best process that transaction for the benefit of all those four parties in the model.

As to processing, the majority of transactions we’re looking at relate to how do we process them as fast as possible in the most accurate way. The way to do that is by peer to peer: If you’re using your card in Europe, in London, say, and you swipe your card as you check out of hotel, we can route that transaction to the hotel’s acquiring bank in London directly to your issuing bank and get that message back for approval without ever going through St. Louis or some big data center in the middle of all that.

You can read the full article HERE.

Motivational Poster: Programming

Code Monkey, Oh So Random 3 Comments

My friend Rob (whom I share an office at work with) made the mistake of sending me the following picture, which I couldn’t resist turning into a motivational poster. I’m just having trouble coming up with the perfect tag line… Comments welcome.

Motivational Poster: Programming

Motivational Poster: Programming

Motivational Poster: Programming

… and my favorite:

Motivational Poster: Office Reality

Pixar’s Wall-E: Interview With Supervising Animator Angus MacLane

Oh So Random No Comments

Wall-EDen of Geek has an interesting interview with Angus MacLane, Pixar’s supervising animator of Wall-E. He reveils plenty of good insights into the development process of Wall-E, but my favorite parts are about the software (of course):

[Interviewer] And does the software you use alter much as you go along, over a three year production cycle?

[Angus] The software for the actual execution of the film doesn’t change that much. Because you really need to lock it to one piece of software. Maybe there’ll be an update here and there, and they’ll be individual sub-programs developed. For example there was a program developed for Wall E just to get his treads to lock to the ground, so they recognise the ground and wrap around and drive as you translate him along. That was technology that was developed as an offshoot from a very similar program from Cars, in keeping the tyres on the ground. But that’s about as automated as we get. Everything else is pretty much hand animated.

I also liked this little tidbit:

[Interviewer] Finally, for anyone looking to break into animation, what advice would you give them?

[Angus] I would say be persistent, and keep trying. A friend of mine, he’s a music composer, he’d got some advice from an old pro when he started. And the old pro said to him that people who succeed in the business are not those that are the most talented, and they’re not the people that know the most people, but they are the people who are able to endure. I think that there’s something profound about that. It’s the old saying, it doesn’t happen by mistake: it’s opportunity met with preparation. So when you get the opportunity, make sure you’re prepared.

You can read the whole interview HERE.

Video: The Simpsons Theme on Bass

Tunes and Grooves No Comments

One more thing to add to my list of things to learn on the bass :-)

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

Video: The Space Shuttle vs. A Bird

Oh So Random No Comments

… guess who lost?

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

San Francisco city officials locked out of computer network

Tech and Security No Comments

San Francisco: LockedUpdate 7/22/2008: The issue may be more complex than it first looks (of course, the media always over-simplifies things). Click HERE to read an insider’s account of the situation.

Okay, THIS is funny because of the glaring security mistakes made by San Francisco’s Department of Technology (or Department of Ignorance, after this one). From the New York Times:

A disgruntled city computer engineer has virtually commandeered San Francisco’s new multimillion-dollar computer network, altering it to deny access to top administrators even as he sits in jail…

Prosecutors say Childs, who works in the Department of Technology… tampered with the city’s new FiberWAN (Wide Area Network), where records such as officials’ e-mails, city payroll files, confidential law enforcement documents and jail inmates’ bookings are stored.

Officials also said they feared that although Childs is in jail, he may have enabled a third party to access the system by telephone or other electronic device and order the destruction of hundreds of thousands of sensitive documents.

This is like security 101… you never give this much power to any single person. On critical systems like this, you always have check-and-balances, outside security code reviews, and strict audits. The S.F. DoT was basically driving around without insurance and got in an accident… I don’t feel sorry for them. It’s really sad how ignorant the world is about security (sigh).

Marcus Miller: Frankenstein

Tunes and Grooves No Comments

Here’s another great Marcus Miller clip. You’ve gotta love the organ work. Props to Josh Brahm for the link.

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

« Previous Entries Next Entries »