Death StarFor every major event in the news, there’s someone who believes it’s a conspiracy… so why would the attack on the Death Star be any different? The guys over at Debunking911 have a great satiracal piece about this monumental Star Wars event; it was an inside job.  It’s funniest if you (a) actually remember Star Wars, (b) are kind of a nerd, and (c) are familiar with “real” conspiracy theories (like the ones surrounding 9/11… video: 1, 2, 3, 4).

And so the Death Star conspiracy goes:

We’ve all heard the “official conspiracy theory” of the Death Star attack. We all know about Luke Skywalker and his ragtag bunch of rebels, how they mounted a foolhardy attack on the most powerful, well-defended battle station ever built. And we’ve all seen the video over, and over, and over, of the one-in-a-million shot that resulted in a massive chain reaction that not just damaged, but completely obliterated that massive technological wonder.

Like many, I was fed this story when I was growing up. But as I watched the video, I began to realize that all was not as it seemed. And the more I questioned the official story, the deeper into the rabbit hole I went.

Read the full story HERE.

Tagged with:  

Microsoft’s Midori OS

On July 30, 2008, in Tech and Security, by Tom

MicrosoftThe SDTimes has an article up about a new operating system Microsoft is working on called “Midori”. It is based on their “Singularity” OS, with everything being written in managed code then natively compiled.  Rumor has it that this is the follow-on to the Windows platform… we’ll see if it ever materializes commercially. SDTimes bases the article on some internal documents they got access to, which may be why we haven’t seen this level of detail before (see the entry in Wikipedia). From the article:

According to the documentation, Midori will be built with an asynchronous-only architecture that is built for task concurrency and parallel use of local and distributed resources, with a distributed component-based and data-driven application model, and dynamic management of power and other resources.

The Midori documents foresee applications running across a multitude of topologies, ranging from client-server and multi-tier deployments to peer-to-peer at the edge, and in the cloud data center. Those topologies form a heterogeneous mesh where capabilities can exist at separate places.

In order to efficiently distribute applications across nodes, Midori will introduce a higher-level application model that abstracts the details of physical machines and processors. The model will be consistent for both the distributed and local concurrency layers, and it is internally known as Asynchronous Promise Architecture.

…operating system services, such as storage, would either be provided to the applications by the OS or be discovered across a trusted distributed environment.

Continue reading »

Tagged with:  

High Scalability has a great link to a video TechTalk with Cuong Do, YouTube’s engineering manager. He talks about the challenges YouTube faces (past and present) to meet it’s skyrocketing user demand, as well as the infrastructure that allows them to scale. I enjoyed the anecdotes: especially the frantic email sent at 2am alerting the dev team that they only had 3 days of storage left… I always thought Google/YouTube would be immune to emergencies like that… ignorance on my part :-)

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

I found this information interesting:

  • The application code is written mostly in Python (the web app is not the bottleneck… the database RPC is)
  • They use Apache for page content and lighttpd for serving video
  • Thumbnails are now served by Google’s BigTable
  • They’re running SuSE Linux with MySQL
  • HW RAID-10 across multiple disks was too slow. HW RAID-1 with SW RAID-0 was faster because the Linux I/O scheduler could see the multiple volumes and would therefore schedule more I/O

You can read a good summary of the talk HERE from the High Scalability website.

TechCruch has a good article of the YouTube API.

Tagged with:  

Inside the Lego Factory

On July 22, 2008, in Oh So Random, by Tom

Lego LogoYeah, Slashdot picked this up already, but it’s still worth posting. Gizmodo has a set of 3 videos that show the workings of the Lego factory. They also have a tour of the secret Lego vault.

This video shows something that very few people have had the opportunity to witness: the inside of the Lego factory, with no barriers or secrets. I filmed every step in the creation of the brick. From the raw granulate stored in massive silos to the molding machines to the gigantic storage cathedrals to the decoration and packaging warehouses, you will be able to see absolutely everything, including the most guarded secret of the company: the brick molds themselves.

Click HERE to read the full story and watch the videos.

Click HERE to watch the tour of the Lego secret vault.

Tagged with:  

C/C++: Using Bitfields Effectively

On July 21, 2008, in Code Monkey, by Tom

Introduction

If you’ve ever done embedded development in C/C++, you are probably familiar with bitfields. They are a handy way to reference individual bits in things like hardware registers. The problem is that bitfields can lead to performance problems and race conditions if not used properly. I hope to highlight some of the issues you should consider when using them.

Usage

First, let’s assume you need to check various fields in a hardware register with the following layout:

Bitfield Register Example

You could define the following bitfield to represent this register:

1: struct HwReg
2: {
3:    unsigned int Base : 16;
4:    unsigned int Offset : 8;
5:    unsigned int Rsvd : 5;
6:    unsigned int Flag : 1;
7:    unsigned int Type : 2;
8: };

The total size of this data type is sizeof(unsigned int), with each line defining a different region (field) within that type (this looks confusing when you first look at it). The following code uses the HwReg bitfield to access a memory-mapped register:

1: struct HwReg* pReg = (struct HwReg*)0×80001005;
2:
3: if (pReg->Flag && pReg->Type == TYPE_1)
4: {
5:    void* address = pReg->Base + pReg->Offset;
6: }

Line 1 defines a pointer to the physical hardware register as type HwReg. We can now use this pointer to easily access the register fields. If this isn’t clear, you can read more about bitfields HERE.

Performance Problems

The compiler doesn’t know how to optimize bitfield accesses (especially because the pointers to memory-mapped hardware registers are almost always declared ‘volatile’). This means that every access to a member of the bitfield will require a read of the physical hardware register. This can be orders of magnitude slower than accessing main memory. In the code example above, the hardware register will be read 4 times; once for each field access.

The way to remedy this is to cache a copy of the register value and then operate on that. Consider the following code:

1: unsigned int* pFullReg = (unsigned int*)0×80001005;
2: unsigned int temp = *pFullReg;
3: struct HwReg* pReg = (struct HwReg*)&temp;
4:
5: if (pReg->Flag && pReg->Type == TYPE_1)
6: {
7:    void* address = pReg->Base + pReg->Offset;
8: }

Line 1 defines a pointer to the physical hardware register. Line 2 performs the actual read into a local variable (the slowest part). This local copy is now in main memory and the CPU cache. Line 3 casts the cached value to the bitfield for easy access. Finally, all accesses to the register fields is on the cached value, which can be read very fast from L1 cache.

Another advantage to this approach is when the hardware requires locking before the register can be accessed. By caching the value, you can keep all the locking code localized to a single area of the function. Without caching, you would hold the lock for a longer period of time (possibly forcing other operations to block) and have to make sure to release the lock on every return path (more difficult with exceptions).

NOTE: Remember you are only working with a copy of the register value. If you update a value in the bitfield, you must still copy the updated value back to the register.

Race Conditions

As stated above, each access to a field value generates its own read/write operation. Even if the CPU architecture guarantees that an individual operation is atomic, updating multiple fields are not. Thus, in a multi-threaded application you must lock the entire block of code that operates on the bitfield. I again suggest caching the value, as you only need to lock the actual read/write of the entire register.

Conclusion

Bitfields are a nice language construct that can help make it easier to write clean code (as opposed to using macros and bitmasks). Unfortunately, it’s all too easy to shoot-yourself-in-the-foot with bitfields if you don’t understand the pitfalls. As always, use caution when writing performance-critical code and make sure you understand how to use the available code constructs.

Happy coding!

Tagged with:  

Mastercard LogoCIO has a good interview with Rob Reeg, president of Mastercard’s Global Technology and Operations. He discusses their infrastructure and processing architecture. Definitely worth looking at if you’re interested in how credit card transactions are processed.

Interviewer: How big of an infrastructure do you have to support and maintain? It must be huge.

Reeg: Actually from a pure server footprint standpoint… we probably have fewer actual footprint servers because of techniques like virtualization that help us leverage one box to do multiple things.

Where it gets interesting is philosophically: We try to put [transaction] processing as close to our customers, the banks, as possible. When we talk about the global network, we have small servers that sit with the bank customers that connect to our network. What it does is it gives us intelligence there at the end of the network. So as a transaction comes through, we can take a look at that transaction and decide how do we best process that transaction for the benefit of all those four parties in the model.

As to processing, the majority of transactions we’re looking at relate to how do we process them as fast as possible in the most accurate way. The way to do that is by peer to peer: If you’re using your card in Europe, in London, say, and you swipe your card as you check out of hotel, we can route that transaction to the hotel’s acquiring bank in London directly to your issuing bank and get that message back for approval without ever going through St. Louis or some big data center in the middle of all that.

You can read the full article HERE.

Tagged with:  

My friend Rob (whom I share an office at work with) made the mistake of sending me the following picture, which I couldn’t resist turning into a motivational poster. I’m just having trouble coming up with the perfect tag line… Comments welcome.

Motivational Poster: Programming

Motivational Poster: Programming

Motivational Poster: Programming

… and my favorite:

Motivational Poster: Office Reality

Tagged with:  

Wall-EDen of Geek has an interesting interview with Angus MacLane, Pixar’s supervising animator of Wall-E. He reveils plenty of good insights into the development process of Wall-E, but my favorite parts are about the software (of course):

[Interviewer] And does the software you use alter much as you go along, over a three year production cycle?

[Angus] The software for the actual execution of the film doesn’t change that much. Because you really need to lock it to one piece of software. Maybe there’ll be an update here and there, and they’ll be individual sub-programs developed. For example there was a program developed for Wall E just to get his treads to lock to the ground, so they recognise the ground and wrap around and drive as you translate him along. That was technology that was developed as an offshoot from a very similar program from Cars, in keeping the tyres on the ground. But that’s about as automated as we get. Everything else is pretty much hand animated.

I also liked this little tidbit:

[Interviewer] Finally, for anyone looking to break into animation, what advice would you give them?

[Angus] I would say be persistent, and keep trying. A friend of mine, he’s a music composer, he’d got some advice from an old pro when he started. And the old pro said to him that people who succeed in the business are not those that are the most talented, and they’re not the people that know the most people, but they are the people who are able to endure. I think that there’s something profound about that. It’s the old saying, it doesn’t happen by mistake: it’s opportunity met with preparation. So when you get the opportunity, make sure you’re prepared.

You can read the whole interview HERE.

 

One more thing to add to my list of things to learn on the bass :-)

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

Tagged with:  

Video: The Space Shuttle vs. A Bird

On July 15, 2008, in Oh So Random, by Tom

… guess who lost?

(requires Adobe Flash plugin… click HERE to watch it on YouTube)

Tagged with: