How does GC, work, generally? Why is it important? The GC inside of the CLR is of a specfic type – ephemeral, concurrent (the server version has always been concuurent and now with Background GC on the client in CLR 4, GC is concurrent on the client as well, but there are differences…)

>4. Is it possible to include a Visual Studio solution in the
>distrubution to make it more convenient for Windows developers to
>use live555?
>5. Is it possible for live555 to generate adequate makefiles for
>Windows systems with development environments newer than Visual
>Studio 2003?

I have no current plans to change this. (These days, fewer and fewer people seem to be using Windows for development of system software.)

Regardless, Live555 works fine on Windows and is actually quite easy to build. Simply do the following:

1. Open the ‘win32config’ file and change the TOOLS32=... variable to your VS2008 install directory. For me, it’s TOOLS32=C:\Program Files\Microsoft Visual Studio 9.0\VC
2. In ‘win32config’, modify the LINK_OPTS_0=... line from msvcirt.lib to msvcrt.lib. This fixes the link error:
   LINK : fatal error LNK1181: cannot open input file 'msvcirt.lib'
3. Open the Visual Studio command prompt.
4. From the ‘live’ source directory, run genWindowsMakefiles
5. Now you’re ready to build. Simply run the following commands:

   cd liveMedia
   nmake /B -f liveMedia.mak
   cd ..\groupsock
   nmake /B -f groupsock.mak
   cd ..\UsageEnvironment
   nmake /B -f UsageEnvironment.mak
   cd ..\BasicUsageEnvironment
   nmake /B -f BasicUsageEnvironment.mak
   cd ..\testProgs
   nmake /B -f testProgs.mak
   cd ..\mediaServer
   nmake /B -f mediaServer.mak

That’s it. You should be good to go.

Outline

1. Video Window Class Definition
2. Creating and Destroying the Window
3. The Window Thread
4. The Message Pump
5. Rendering a Video Frame

Things to note:

• You will need a thread in your window class to “pump” window events.
• GDI does NOT do any color conversion, so you will need to convert each frame into the same format as the display (always RGB, usually 24 or 32 bits).

Video Window Class Definition

class VideoWindow
{
public:
    ...
private:
    static DWORD _ThreadMain(void*);
    static LRESULT WINAPI _MsgProc(HWND, UINT, WPARAM, LPARAM);

    CRITICAL_SECTION _lok;
    HANDLE _thread;
    HWND _hWnd;
    HDC _hDC;
    HDC _hFrameDC;
    HFONT _hFont;
    int _bpp;   // bit depth
    volatile bool _exit;
    int _x, _y, _w, _h; // Window dimensions
    int _imgWidth, _imgHeight, _imgLength; // Frame dimensions
    unsigned char *_imgBits;
};

Creating and Destroying the Window

void VideoWindow::Create(int x, int y, int w, int h)
{
    EnterCriticalSection(&_lok);
    _x = x;
    _y = y;
    _w = w;
    _h = h;
    _hWnd = 0;
    _exit = false;
    _thread = CreateThread( NULL, NULL, (LPTHREAD_START_ROUTINE)_ThreadMain,
        this, NULL, 0 );
    LeaveCriticalSection(&_lok);

    // Wait for window to be created. Cheap, but it works.
    while (!_hWnd) {
        ::Sleep(100);
    }
}

void VideoWindow::Destroy()
{
    // Safe to call from the destructor.

    EnterCriticalSection(&_lok);
    if(_hWnd)
    {
        _exit = true;
        LeaveCriticalSection(&_lok);
        WaitForSingleObject( _thread, INFINITE );
        CloseHandle(_thread);
        EnterCriticalSection(&_lok);
        DestroyWindow(_hWnd);
        _hWnd = 0;
    }
    if (_hFont) ::DeleteObject(_hFont);
    if (_hFrameDC) ::DeleteDC(_hFrameDC);
    if (_hDC) ::DeleteDC(_hDC);
    LeaveCriticalSection(&_lok);
}

The Window Thread

DWORD VideoWindow::_ThreadMain(void* ctx)
{
    VideoWindow *p = (VideoWindow*)ctx;
    EnterCriticalSection(&p->_lok);

    LPCSTR className = "VideoWndClass";
    LPCSTR windowName = "MyWindowText";
    WNDCLASSEX wcex = {
        sizeof(WNDCLASSEX),
        CS_OWNDC | CS_HREDRAW | CS_VREDRAW,
        _MsgProc,
        0L,
        0L,
        NULL,
        NULL,
        NULL,
        (HBRUSH)GetStockObject(BLACK_BRUSH),
        NULL,
        className,
        NULL
    };

    RegisterClassEx(&wcex);   

    p->_hWnd = CreateWindow(className, windowName, WS_OVERLAPPEDWINDOW,
        p->_x, p->_y, p->_w, p->_h, ::GetDesktopWindow(), NULL, NULL, NULL);

    p->_hDC = ::GetDC(p->_hWnd);
    p->_bpp = ::GetDeviceCaps(p->_hDC, BITSPIXEL);
    p->_hFrameDC = ::CreateCompatibleDC(p->_hDC);

    p->_hFont = ::CreateFont(18, 0, 0, 0, FW_BOLD, FALSE, FALSE, FALSE,
        ANSI_CHARSET, OUT_DEFAULT_PRECIS, CLIP_DEFAULT_PRECIS,
        ANTIALIASED_QUALITY, DEFAULT_PITCH | FF_DONTCARE, "Arial");

    ShowWindow(p->_hWnd,SW_SHOW);
    UpdateWindow(p->_hWnd);

    while( !p->_exit )
    {
        MSG msg;
        if(PeekMessage(&msg,p->_hWnd,0,0,PM_REMOVE))
        {
            TranslateMessage(&msg);
            DispatchMessage(&msg);
        }

        LeaveCriticalSection(&p->_lok);
        Sleep(50); // A better way to do this is to use a message
        EnterCriticalSection(&p->_lok);
    }
    LeaveCriticalSection(&p->_lok);
    return 0;
}

The Message Pump

LRESULT WINAPI VideoWindow::_MsgProc(HWND hWnd, UINT msg, WPARAM w, LPARAM l)
{
    if (msg == WM_PAINT && l)
    {
        VideoWindow *p = (VideoWindow*)l;
        EnterCriticalSection(&p->_lok);

        HGDIOBJ  oldFont  = ::SelectObject(p->_hDC, p->_hFont);
        COLORREF oldFtColor = ::SetTextColor(p->_hDC, RGB(255,255,255));
        COLORREF oldBkColor = ::SetBkColor(p->_hDC, RGB(64,64,64));

        // Render image
        //
        // NOTE: This assumes you've already coverted the image to the proper
        // RGB bit-depth (_bpp) and copied the data into _imageBits.
        //
        HBITMAP hBMP = ::CreateCompatibleBitmap(p->_hDC, p->_imgWidth,
            p->_imgHeight);
        ::SetBitmapBits(hBMP, p->_imgLength, p->_imgBits);
        HGDIOBJ oldBMP = ::SelectObject(p->_hFrameDC, hBMP);
        ::BitBlt(p->_hDC, 0, 0, p->_imgWidth, p->_imgHeight, p->_hFrameDC,
            0, 0, SRCCOPY);
        ::SelectObject(p->_hFrameDC, oldBMP);

        // Draw some text over the image
        RECT rect;
        ::SetRectEmpty(&rect);
        ::DrawText(p->_hDC, p->_imageText, -1, &rect, DT_CALCRECT);
        ::DrawText(p->_hDC, p->_imageText, -1, &rect, DT_CENTER | DT_VCENTER);

        if (hBMP) ::DeleteObject(hBMP);

        LeaveCriticalSection(&p->_lok);
    }
    return DefWindowProc(hWnd, msg, w, l);
}

Rendering a Video Frame

void VideoWindow::Render(const unsigned char* frame, int w, int h, int stride)
{
    // Convert frame to the proper RGB bit depth to match _bpp and copy it
    // into _imgBits buffer. Update _imgWidth, _imgHeight, and _imgLength as
    // necessary.

    ::PostMessage(_hWnd, WM_PAINT, 0, (LPARAM)this);
}

#ifdef WIN32
    #ifndef NAN
        static const unsigned long __nan[2] = {0xffffffff, 0x7fffffff};
        #define NAN (*(const float *) __nan)
    #endif
#endif

This code is adapted from an MSDN example.

Note: This post assumes you are already familiar with the socket extensions for IPv6 (RFC 3493).

Linux and Mac
Good news… they both fully support RFC 3493.

Windows
Windows IPv6 support varies based on which version you’re targeting. Microsoft started adding IPv6 in Windows 2000, and they’ve continued adding more of the socket extensions as time went on. Most of the core functionality is present in XP, and what’s missing is easily replaced by using Winsock calls directly (more on this later).

Windows gained IPv6 support while RFC 2553 was still the supported standard. Since then, it has been deprecated by RFC 3493. However, Microsoft doesn’t want to break existing code written against it’s API, so the older API lives on. The main impact of this is that sockaddr_in6 and sockaddr_storage are slightly different on Windows than Mac and Linux. The size of the structures across platforms is the same (the sa_family_t member was shortened), it’s just that the Windows structures don’t begin with the length member. For example:

// Linux and Mac
struct sockaddr_in6 {
    uint8_t      sin6_len;    /* Added in RFC 3493 */
    sa_family_t  sin6_family;
    ...
};
struct sockaddr_storage {
    uint8_t      ss_len;      /* Added in RFC 3493 */
    sa_family_t  ss_family;
    ...
};

// Windows
struct sockaddr_in6 {
    sa_family_t  sin6_family;
    ...
};
struct sockaddr_storage {
    sa_family_t  ss_family;
    ...
};

I’ve never had a problem with this, because the size of sockaddr_in6 is easily determined (sizeof(sockaddr_in6)) and I always end up casting sockaddr_storage to the specific type (sockaddr_in or sockaddr_in6) based on ss_family.

Besides the data structure differences, it’s important to remember that Microsoft added IPv6 support over multiple versions. Support first appeared in Windows 2000, but more of the extensions have been added over time. Most of the core functionality was present in XP (including multicast), but not everything is implemented as of Windows 7. It’s annoying, but I will say that what’s missing is easily replaced by using Winsock calls directly.

Here’s the breakdown of IPv6 socket extensions by Windows version:

As you can see, you may still have to call Winsock directly depending on what version of Windows you are targeting. In my opinion, programming IPv6 on Windows is a lot easier if you only support XP and later, but I know that’s not always possible.

Summary

Modern operating systems all support IPv6. However, for business reasons, Windows has a slightly older version of the socket API which requires special consideration. My goal was to enumerate those differences to help make the transition to IPv6 smoother. Writing cross-platform code can be a fun challenge at times, but it’s also a little tedious. Hopefully, this post helps ease the pain.

Note: I’m talking here about interval timing (i.e. accurately measuring the duration between 2 events). This is different than synchronizing different clocks or maintaining accurate wall time.

Anatomy of the Clock

The first mistake most people make when doing timing is to use functions like gettimeofday(), GetSystemTime(), etc. These functions return what is called “wall time”… time that corresponds to a calendar date/time. These clocks suffer from the follow limitations:

1. They have a low resolution: “High-performance” timing, by my definition, requires clock resolutions into the microseconds or better.
2. They can jump forwards and backwards in time: Computer clocks all tick at slightly different rates, which causes the time to drift. Most systems have NTP enabled which periodically adjusts the system clock to keep them in sync with “actual” time. The adjustment can cause the clock to suddenly jump forward (artificially inflating your timing numbers) or jump backwards (causing your timing calculations to go negative or hugely positive).

For interval timing, all that’s needed for a clock is a simple counter that increments at a stable rate. For high-performance timing, the rate this counter increments should be high. A related constraint is that the counter must be monotonic (can never “tick” backwards… ever). The counter may overflow and wrap back to 0, but using unsigned math in your timing calculations can compensate for that (see example below).

Something to note: since we are usually measuring short durations, the drift of the clock is so small that we aren’t concerned by it (what matters is the drift between successive reads, not total drift over time).

Using the Clock

To use the clock, you need access to 2 pieces of information: the clock value, and the rate (frequency) at which it increments. Once you have that information, it’s trivial to calculate the duration between 2 successive reads of the clock.

For example, consider the following:

-        start equals the clock value at the beginning of the interval to measure.

-        end equals the clock value at the end of the interval.

-        frequency equals the frequency that the clock increments per second.

Calculating the duration of the interval (in seconds) is as simple as:

duration = (end – start) / frequency

duration will equal the floating-point interval time in seconds (e.g. 0.000237 = 237us).

A common use case I’ve seen is wanting to measure the duration between successive calls to a function. Here’s one way to implement that:

float freq = (float) get_frequency();

Foo()
{
    unsigned now = read_clock();
    float duration = (float)(now – last) / (float)freq;
    last = now;

    /* Your code here */
}

It is important to use unsigned variables for now and last or your calculations will go haywire if the clock ever wraps back to 0. Consider what happens if now is less than last. Using unsigned variables allows the math operation to underflow, which produces the correct result. If you don’t believe me, just write some test code and see for yourself… it’s an important concept to understand.

Don’t Use RDTSC As Your Clock

I’m dismayed by how many forum posts suggest that newbie’s use RDTSC for timing. Don’t get me wrong; the TSC isn’t bad in-and-of-itself. It’s just way too hard to get timing right with it unless you’re an expert… and if you’re reading this, chances are you aren’t . Stick to the higher level API’s I present below.

Even if you are an expert, I still suggest avoiding RDTSC for the following reasons:

1. It’s processor core specific: Using RDTSC will give different values on different processor cores. This causes non-monotonic clock behavior as a thread is migrated across cores during execution (i.e. the clock can tick backwards on two successive reads). Conversely, the clock may also appear to jump forwards if the thread moves to a different core.
2. It doesn’t always tick at the same rate: On some systems, the clock frequency will vary as the CPU load changes. This is due to power saving features in the processor that throttle down the clock speed when the load is low (e.g. Intel SpeedStep).
3. It’s unnecessary: there are better alternatives to RDTSC that are readily available (HPET and APIC).

“Game Timing and Multicore Processors” has more info, though it’s centered around Windows.

“Guidelines For Providing Multimedia Support” has a good summary of the hardware clocks on the PC platform, and justification for the creation of the HPET.

A more technical look into the past problems with RDTSC can be found in this email from Rick Brunner (AMD Fellow).

If you’re having trouble sleeping, you can look at Intel’s HPET design document.

Linux Clock

POSIX.1b defines realtime clock methods that you’ll find on most *NIX systems (the full spec can be viewed HERE). Specifically, you want to use clock_getres() and clock_gettime(). clock_getres() returns the resolution (frequency) of the clock, and clock_gettime() returns the current value of the clock. Most systems implement the CLOCK_MONOTONIC type, which provides a frequency-stable, monotonically-increasing counter. The resolution of CLOCK_MONOTONIC is high on the 2.6 kernel, in my experience. I recommend using this clock when building a high-performance timing solutions on Linux.

The methods are defined in time.h, and you need to link against librt (pass ‘-lrt’ to gcc). The prototypes for the functions are:

int clock_getres(clockid_t clock_id, struct timespec *res);
int clock_gettime(clockid_t clock_id, struct timespec *tp);

A detailed description of these methods can be found HERE.

I also suggest looking at clock_nanosleep(), but that’s a separate topic.

Windows Clock

On Windows, QueryPerformanceFrequency() and QueryPerformanceCounter() are the obvious choice. QueryPerformanceFrequency() returns (surprise!) the frequency of the counter. QueryPerformanceCounter() returns the current value of the counter. Just like CLOCK_MONOTONIC on Linux, the Windows performance counter is a high-frequency, stable, monotonically-increasing counter.

The methods are defined in windows.h. The prototypes are:

BOOL QueryPerformanceFrequency(LARGE_INTEGER *lpFrequency);
BOOL QueryPerformanceCounter(LARGE_INTEGER *lpPerformanceCount);

Windows Timing Errata

Note: The follow items do NOT affect the resolution of the Window’s performance counter, but I think it’s still important to know when doing timing on Windows.

One problem you may run into on Windows is that the default system clock interval defaults to 10 or 15ms (depending of the OS version). This clock is what drives all the timers and sleep functions for that platform. What this means is that, left to the default, your timers will trigger 5 to 7.5ms late, on average.

You can fix this by increasing the system clock resolution to 2ms (I remember reading a tech article by Microsoft saying that 2ms gave better system performance than 1ms, but I can’t find it for the life of me). You do this by using the timeGetDevCaps(), timeBeginPeriod() and timeEndPeriod(). Sample code can be found HERE. You have to include mmsystem.h and link against Winmm.lib.

I feel it necessary to note that increasing the system clock interval negatively affects power consumption. The Windows 7 blog as an interesting breakdown of “Windows 7 Energy Efficiency”. Specifically, they noticed a 10% drop in battery life when the clock resolution was set to 1ms using timeBeginPeriod().

Maybe this last section belongs in a separate post, but whatever…

Fault bucket 42424242, type 1
Event Name: APPCRASH
Response: None
Cab Id: 0

Problem signature:
P1: MyApp.exe
P2: 1.42.42.42
P3: 598773cf
P4: StackHash_ac62
P5: 0.0.0.0
P6: 00000000
P7: c0000007
P8: 00000000
P9:
P10:

We spent some time wondering if our crypto libraries were the problem (we just made some changes recently), but concluded that was unlikely. So what the heck is the “StackHash” module? Did our trashed stack cause the kernel to think we were a different module? Nope.

The answer is that the Windows executive couldn’t identify the module we were in when the application crashed (it uses the instruction pointer to determine what code was executing). In this case, the kernel simply takes a hash of the stack so at least we might be able to identify if we’ve seen this exact crash before. Here’s the answer summarized by an engineer from Microsoft:

In the OS when I try to get a faulting module name it is possible that there is no module laoded (sic) at that address. For example in this case the EIP was zero. So in those cases where a module is not loaded and it is not also in the unloaded module list, I take a stack hash of the stack so that we can identify this crash from other crashes where also the module is not known.

Analogy is a typographic clock which fuses the immediacy of digital with the visual-spatial quality of analogue into a hybrid format. It presents an everyday object with a fresh twist.

Click on the image below to visit his site and download it. Enjoy!

#define _CRTDBG_MAP_ALLOC
// Include other header files here
#include <crtdbg.h>

Then, you call _CrtDumpMemoryLeaks() before your application exits. If your program exits at many points, you can alternatively call _CrtSetDbgFlag( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF ) at the beginning of you application, which will cause the leaks to also be printed when it exits. The results are printed to the Debug Window and look like the following:

Detected memory leaks!
Dumping objects ->
C:\PROGRAM FILES\VISUAL STUDIO\MyProjects\leaktest\leaktest.cpp(20) :
{18} normal block at 0x00780E80, 64 bytes long.
Data: <                > CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD
Object dump complete.

Cool, Huh?! However, some libraries don’t play nice with this, as I explain below.

Memory Leak Detection and ACE:

I’ve mentioned ACE before in a previous post evaluating portable runtime environments. It’s a pretty cool set of libraries that provides a portable OS abstraction layer and communication environment. If you write your application using ACE, it will compile and run on a whole host of system types… but I digress.

One problem with ACE is that it conflicts with the Windows leak check code. If you define _CRTDBG_MAP_ALLOC before the ACE headers, you will get build errors like the following:

1>..\include\ace\os_ns_unistd.h(154) : error C2059: syntax error : 'constant'

What’s happening is that the ACE headers try to enable the leak detection code themselves, which causes the overridden memory functions to conflict with the original versions. In essence, ACE is trying to define malloc (for example) after crtdbg.h has already overridden it.

So, how do you get your application to build with leak detection and ACE? Define _CRTDBG_MAP_ALLOC after the ACE headers (Remember: the ACE headers must always be included before other headers… another complaint I have).

#include "ace/ACE.h"
#define _CRTDBG_MAP_ALLOC
// Include other headers here
#include <crtdbg.h>

Okay, your application should compile now. Run it and look at the output window. For a simple application that does nothing but return, this is what I see:

Detected memory leaks!
Dumping objects ->
{178} normal block at 0x00CDBD78, 50 bytes long.
Data: <---TEST---      > 2D 2D 2D 48 45 4C 4C 4F 2D 2D 2D 00 CD CD CD CD
{173} normal block at 0x002EAAC8, 8 bytes long.
Data: <  .     > C0 A1 2E 00 CD CD CD CD
{172} normal block at 0x002EAA78, 20 bytes long.
Data: <4  Z       Z    > 34 90 F3 5A 0A 01 00 00 F3 17 DA 5A 03 00 00 00
{171} normal block at 0x002EAA18, 32 bytes long.
Data: <   Z  ]         > C4 91 F3 5A C0 FB 5D 00 FF FF FF FF 00 00 00 00
{170} normal block at 0x002EA9B8, 32 bytes long.
Data: <   Z  ]         > C4 91 F3 5A 88 FB 5D 00 FF FF FF FF 00 00 00 00
{169} normal block at 0x002EA958, 32 bytes long.
Data: <   ZP ]         > B4 91 F3 5A 50 FB 5D 00 FF FF FF FF 00 00 00 00
{168} normal block at 0x002EA910, 8 bytes long.
Data: <   Z    > D4 91 F3 5A 00 00 00 00

... Truncated for sanity (30 more leaks follow) ...

Object dump complete.
The program '[14288] simple.exe: Native' has exited with code 0 (0x0).

These are very likely false-positives, but they are annoying. There are other problems too:

1. There are no file/line numbers printed, so we can’t track down the code to verify if the errors are real or not.
2. File and line numbers don’t show up for errors outside of ACE (the first leak is mine).
3. All this output adds noise that can hide other memory leaks (I wish Microsoft would add a way to suppress errors we don’t care about… props to Valgrind for doing this).

Anyways, all of this is quite frustrating. I was hunting around on the ACE newsgroup for solutions and could only find this thread:

“In general, we track memory issues on Windows using Purify, so you might try going that route.”
Douglas Schmidt

“Correct. This memory tracking scheme and the ACE restrictions onheader file placement are at odds here. You have two choices:

1. Use a different memory tracking scheme such as Purify, which Doug
suggested.

2. Develop, or sponsor, necessary changes to remove ACE’s restrictions
on header file placement.”
Steve Huston

So there you have it: ACE and Window memory leak detection don’t play nice together. I hate encountering a problem without a viable solution… it bugs me (no pun intended). I even tried modifying the ACE headers with marginal results (it came close to compiling). Oh well…

Of particular interest to me are the changes to DirectX 10, Media Foundation, and the new DirectX 11. Here are some highlights.

DirectX 11:

• “…resource creation and management has been optimized for multithreaded use, enabling more efficient dynamic texture management for streaming.”
• Several improvements have been made to the high-level shading language (HLSL), such as a limited form of dynamic linkage in shaders to improve specialization complexity, and object-oriented programming constructs like classes and interfaces.”

DirectX 10 improvements:

• “The pipeline also introduces the geometry shader stage, which offloads work entirely from the CPU to the GPU. This new stage enables you to create geometry, stream the data to memory, and render the geometry with no CPU interaction.”
• Predicated rendering performs occlusion culling to reduce the amount of geometry that is rendered. Instancing APIs can dramatically reduce the amount of geometry that needs to be transferred to the GPU by drawing multiple-instances of similar objects. Texture arrays enable the GPU to do texture swapping without CPU intervention.”

Media Foundation improvements:

• “…Media Foundation has been enhanced to provide better format support, including MPEG-4, as well as support for video capture devices and hardware codecs.”
• “In Windows 7, Media Foundation provides extensive format support that includes codecs for H.264 video, MJPEG, and MP3; new sources for MP4, 3GP, MPEG2-TS, and AVI; and new file sinks for MP4, 3GP, and MP3.”
• “In Windows Vista, Media Foundation exposed a relatively low-level set of APIs. These APIs are flexible, but may not be appropriate for performing tasks. Windows 7 adds new high-level APIs that make it simpler to write media applications in C++.”

It seems that Lenovo (the PC supplier for the games) chose Windows XP instead fo Vista. From the article:

Lenovo chairman, Yang Yuanqing, was quoted as saying that because of the complexity of the IT functions at the Games, it was decided to not use the the more recent operating system. “If it’s not stable, it could have some problems,” he said.

Ironically, former Microsoft CEO Bill Gates was in the crowd (he can run but he can’t hide).

Gizmodo has some more images and links to the incident.

According to the documentation, Midori will be built with an asynchronous-only architecture that is built for task concurrency and parallel use of local and distributed resources, with a distributed component-based and data-driven application model, and dynamic management of power and other resources.

The Midori documents foresee applications running across a multitude of topologies, ranging from client-server and multi-tier deployments to peer-to-peer at the edge, and in the cloud data center. Those topologies form a heterogeneous mesh where capabilities can exist at separate places.

In order to efficiently distribute applications across nodes, Midori will introduce a higher-level application model that abstracts the details of physical machines and processors. The model will be consistent for both the distributed and local concurrency layers, and it is internally known as Asynchronous Promise Architecture.

…operating system services, such as storage, would either be provided to the applications by the OS or be discovered across a trusted distributed environment.

The programming model and API are also changing to help developers develop in the new model (goodbye Win32):

The Midori documents indicate that the proposed OS would have a non-blocking object-oriented framework API. This would have strong notions of immutability—in the sense of objects that cannot be modified once created—and strive to foster application correctness through deep verifiability by using .NET programming languages.

The Midori programming model will tackle state management, which Microsoft admits in its documentation is a challenge in Windows, by migrating APIs, applications and developers to a constrained model.

Other objectives are eliminating dynamic loading and in-process extensions; developing a failure model based on reliable transactions, so the system understands exactly which processes are impacted by a cascading failure and how to restart the computation; and having a standard way of dealing with latency, asynchronous behavior and cancellation, throughout the stack.

To provide better modularity (and to support mobile devices), Midori will be a micro-kernel:

Unlike Windows, Microsoft intends for Midori to be componentized from the beginning to achieve performance and security benefits. It will have strong isolation boundaries and enforced contracts between components, to ensure that servicing one component will not cause others to fail, while keeping overhead minimal.

At its lowest level, Midori has two separate kernel layers: a microkernel comprised of unmanaged code that controls hardware and environment abstracts, and higher-level managed kernel services that provide the full set of operating system functionality.

There are a lot more gems in the article… definitely worth a read. Click HERE to read it.

Picture source

When writing a shared library, it is sometimes useful to have a set of functions that get called when the library is loaded and unloaded. In Windows, this is done by implementing the DllMain function. This function is called by the loader whenever a DLL is loaded or unloaded into the address space of a process (and also when the process creates a new thread, but it is less common to handle this case). A value is passed in as an argument to the DllMain function that indicates which event is occurring: DLL load or unload.

On Linux, one must use the GCC __attribute__((constructor)) and __attribute__((destructor)) keywords (double underscores before and after) to explicitly declare functions to be called on load and unload. These keywords cause the compiler/linker to add the specified functions to the __CTOR_LIST__ and __DTOR_LIST__ (“ConstrucTOR LIST” and “DestrucTOR LIST” respectively) in the object file. Functions on the __CTOR_LIST__ are called by the loader when the library is loaded (either implicitly or by dlopen()). The main purpose for this list is to call the constructors on global objects in the library. Conversely, functions on the __DTOR_LIST__ are called when the library is unloaded (either implicitly or by dlclose()). By adding initialization and clean-up functions to this list, one can effectively replicate the DllMain functionality on Linux.

NOTE: There are many ways to “shoot yourself in the foot” with these methods (on both Windows and Linux) because certain things aren’t available to your library until loading is complete. Don’t use these methods unless you have a real need… just export an Initialize() and Destroy() function instead, and force the consuming application to call them. Please read the “Gotcha’s” section below.

External Resources:

• MSDN: DllMain
• Misc. Linux Library Information
• Extended GCC Features
• Dynamic Linking on Linux and Windows

Example: Linux

Example: Windows

Gotcha’s

Linux

From www.dwheeler.com:

Shared libraries must not be compiled with the gcc arguments “-nostartfiles” or “-nostdlib”. If those arguments are used, the constructor/destructor routines will not be executed (unless special measures are taken).

Windows

From Microsoft’s MSDN page on DllMain:

The entry-point function should perform only simple initialization or termination tasks. It must not call the LoadLibrary or LoadLibraryEx function (or a function that calls these functions), because this may create dependency loops in the DLL load order. This can result in a DLL being used before the system has executed its initialization code. Similarly, the entry-point function must not call the FreeLibrary function (or a function that calls FreeLibrary) during process termination, because this can result in a DLL being used after the system has executed its termination code.

Because Kernel32.dll is guaranteed to be loaded in the process address space when the entry-point function is called, calling functions in Kernel32.dll does not result in the DLL being used before its initialization code has been executed. Therefore, the entry-point function can call functions in Kernel32.dll that do not load other DLLs. For example, DllMain can create synchronization objects such as critical sections and mutexes, and use TLS. Unfortunately, there is not a comprehensive list of safe functions in Kernel32.dll.

Windows 2000: Do not create a named synchronization object in DllMain because the system will then load an additional DLL. This restriction does not apply to subsequent versions of Windows.

Calling functions that require DLLs other than Kernel32.dll may result in problems that are difficult to diagnose. For example, calling User, Shell, and COM functions can cause access violation errors, because some functions load other system components. Conversely, calling functions such as these during termination can cause access violation errors because the corresponding component may already have been unloaded or uninitialized.

Because DLL notifications are serialized, entry-point functions should not attempt to communicate with other threads or processes. Deadlocks may occur as a result.

For information on best practices when writing a DLL, see http://www.microsoft.com/whdc/driver/kernel/DLL_bestprac.mspx.

If your DLL is linked with the C run-time library (CRT), the entry point provided by the CRT calls the constructors and destructors for global and static C++ objects. Therefore, these restrictions for DllMain also apply to constructors and destructors and any code that is called from them.

Inside the Windows Vista Kernel: Part 1

Inside the Windows Vista Kernel: Part 2

Inside the Windows Vista Kernel: Part 3

Well, there’s a subtle reason why this is not good, and I believe Bruce Schneier offers a good summary as to why this is:

It’s called the “equities issue.” Basically, the NSA has two roles: eavesdrop on their stuff, and protect our stuff. When both sides use the same stuff — Windows Vista, for example — the agency has to decide whether to exploit vulnerabilities to eavesdrop on their stuff or close the same vulnerabilities to protect our stuff. In its partnership with Microsoft, it could have decided to go either way: to deliberately introduce vulnerabilities that it could exploit, or deliberately harden the OS to protect its own interests.

So, which choice did they make? We’ll probably never know, but given the current administration’s feeling about privacy and warrentless eavesdropping, this whole thing doesn’t make me feel any better about Vista security.

The real irony of the whole thing is that this could make Vista seem more secure, when actually the opposite is true. There’s an old saying in the security field: “No security is better than poor security.” When there’s no security, at least people are cautious with their data. With the “illusion” of security, people tend to act as if they are truly secure.

As a side note, this is an example of why security is so hard to get right. In many ways, true security is counter-intuitive… that’s part of what makes this field so interesting.