The picture below shows the 2 most common configurations for a Netem box:

I’ve always used the first configuration, but it doesn’t really matter.

1. Find a Suitable System

• Any “reasonable” machine that can run Fedora 14 (I use an old Pentium 4 box since I don’t need to simulate a high-speed link)
• It must have 2 network interfaces
• It’s nice to use a smaller box if you want it to be portable

2. Install bridge-utils

Make sure you are root, and run:

brctl

to see if bridge-utils is installed. If it isn’t, run:

yum install bridge-utils.

3. Bridge the 2 Network Interfaces

First make sure to clear the network configuration for your interfaces:

ifconfig eth0 0.0.0.0
ifconfig eth1 0.0.0.0

Then create the bridge and bring it up:

brctl addbr br0
brctl setfd br0 0
brctl addif br0 eth0
brctl addif br0 eth1
ifconfig br0 up

Note that we disable the forwarding delay (‘setfd’). This makes the bridge pass traffic through it immediately instead of the configured delay time.

One final set I had to perform is disabling kernel-level filtering on the bridge. This is done by writing 0 to the bridge nodes under proc:

for f in /proc/sys/net/bridge-*; do echo 0 > $f; done

Note: If for some reason none of this works for you, check out the Linux Foundation page on network bridging.

4. Configure Netem

Netem is actually used in conjunction with the Traffic Control application, tc. I’m not going to go into detail here, but suffice it to say that tc allows you to do packet shaping and adjust packet scheduling. Check out the tc man page for more information.

tc allows you to specify the “queueing discipline” (or “qdisc” for short) used for sending outbound packets on an interface (the fact that it operates on outbound packets only is important to remember). Basically, a qdisc defines how outbound packets are ordered and sent. To view the current qdisc setup on your box, type:

tc qdisc

The default qdisc is pfifo_fast. We’re going to change this to use a combination of Netem and Token Bucket Filter.

Note: For the following examples, I assume eth0 is connected to the network (client-side), and eth1 is directly connected to the server.

Limiting Bandwidth

The Token Bucket Filter (tbf) is used to limit how much data can exit the network interface per second… perfect for simulating WAN bandwidth limitations. Let’s assume we want to emulate a client-side WAN connection of 768kbps down and 128kbps up. Assuming the server is connected to eth1, the eth1 interface receives inbound traffic from the server and eth0 sends that traffic outbound to the client. Since we know the a qdisc works on outbound traffic only, we need to limit eth0 for our download speed of 768kbps. Conversely, we configure eth1 for our upload speed of 128kbps.

tc qdisc replace dev eth0 root handle 1:0 tbf rate 768kbit burst 2048 latency 100ms

tc qdisc replace dev eth1 root handle 2:0 tbf rate 128kbit burst 2048 latency 100ms

We use the ‘replace’ command to overwrite any qdisc setting that’s there (you can use the ‘del’ command to simple remove qdiscs). We set the qdisc as the ‘root‘ of the tree, and configure the tbf ‘rate‘ accordingly. The ‘burst‘ and ‘latency‘ parameters control the initial number of tokens in the bucket and how long queued packet can hang around before being dropped, respectively.

Adding Latency

We can append qdiscs that will allows us to use different tools to control how our simulated WAN behaves. In this example, I’ll use Netem to artifically add 57ms of latency to the download connection with a random variantion of +/- 13ms:

tc qdisc add dev eth0 parent 1:1 handle 10: netem delay 57ms 13ms

And So Much More…

The Netem page on the Linux Foundation website has many other great examples, so there’s no point in me copying them here. If you’ve made it this far and are still interested, I highly encourage you to check out their page.

Note: This post assumes you are already familiar with the socket extensions for IPv6 (RFC 3493).

Linux and Mac
Good news… they both fully support RFC 3493.

Windows
Windows IPv6 support varies based on which version you’re targeting. Microsoft started adding IPv6 in Windows 2000, and they’ve continued adding more of the socket extensions as time went on. Most of the core functionality is present in XP, and what’s missing is easily replaced by using Winsock calls directly (more on this later).

Windows gained IPv6 support while RFC 2553 was still the supported standard. Since then, it has been deprecated by RFC 3493. However, Microsoft doesn’t want to break existing code written against it’s API, so the older API lives on. The main impact of this is that sockaddr_in6 and sockaddr_storage are slightly different on Windows than Mac and Linux. The size of the structures across platforms is the same (the sa_family_t member was shortened), it’s just that the Windows structures don’t begin with the length member. For example:

// Linux and Mac
struct sockaddr_in6 {
    uint8_t      sin6_len;    /* Added in RFC 3493 */
    sa_family_t  sin6_family;
    ...
};
struct sockaddr_storage {
    uint8_t      ss_len;      /* Added in RFC 3493 */
    sa_family_t  ss_family;
    ...
};

// Windows
struct sockaddr_in6 {
    sa_family_t  sin6_family;
    ...
};
struct sockaddr_storage {
    sa_family_t  ss_family;
    ...
};

I’ve never had a problem with this, because the size of sockaddr_in6 is easily determined (sizeof(sockaddr_in6)) and I always end up casting sockaddr_storage to the specific type (sockaddr_in or sockaddr_in6) based on ss_family.

Besides the data structure differences, it’s important to remember that Microsoft added IPv6 support over multiple versions. Support first appeared in Windows 2000, but more of the extensions have been added over time. Most of the core functionality was present in XP (including multicast), but not everything is implemented as of Windows 7. It’s annoying, but I will say that what’s missing is easily replaced by using Winsock calls directly.

Here’s the breakdown of IPv6 socket extensions by Windows version:

As you can see, you may still have to call Winsock directly depending on what version of Windows you are targeting. In my opinion, programming IPv6 on Windows is a lot easier if you only support XP and later, but I know that’s not always possible.

Summary

Modern operating systems all support IPv6. However, for business reasons, Windows has a slightly older version of the socket API which requires special consideration. My goal was to enumerate those differences to help make the transition to IPv6 smoother. Writing cross-platform code can be a fun challenge at times, but it’s also a little tedious. Hopefully, this post helps ease the pain.

// Returns the address family of an address or hostname.
// AF_INET, AF_INET6, or -1 on error.
int getaddrfamily(const char *addr)
{
    struct addrinfo hint, *info =0;
    memset(&hint, 0, sizeof(hint));
    hint.ai_family = AF_UNSPEC;
    // Uncomment this to disable DNS lookup
    //hint.ai_flags = AI_NUMERICHOST;
    int ret = getaddrinfo(addr, 0, &hint, &info);
    if (ret)
        return -1;
    int result = info->ai_family;
    freeaddrinfo(info);
    return result;
}

See RFC 3493 for more information on the latest socket API for dealing with IPv6.

Here’s the mapping, taken from a Microsoft TechNet article:

To support IP multicasting, the Internet authorities have reserved the multicast address range of 01-00-5E-00-00-00 to 01-00-5E-7F-FF-FF for… media access control (MAC) addresses.

To map an IP multicast address to a MAC-layer multicast address, the low order 23 bits of the IP multicast address are mapped directly to the low order 23 bits in the MAC-layer multicast address. Because the first 4 bits of an IP multicast address are fixed according to the class D convention, there are 5 bits in the IP multicast address that do not map to the MAC-layer multicast address. Therefore, it is possible for a host to receive MAC-layer multicast packets for groups to which it does not belong. However, these packets are dropped by IP once the destination IP address is determined.

For example, the multicast address 224.192.16.1 becomes 01-00-5E-40-10-01.

Happy hacking!

1. Set the hostname: `# hostname <hostname>`
2. Add ‘HOSTNAME=<hostname>’ to ‘/etc/sysconfig/network’ (makes the change permanent).
3. Add ‘DHCP_HOSTNAME=<hostname>’ to ‘/etc/sysconfig/network-scripts/ifcfg-eth0′
4. Restart the networking service: `# service network restart`

Volia! Now you can connect to your box via hostname.

References:

• Configuring Linux Static DHCP Clients by Sending Host Name
• Linux setting hostname and domain name of my server

Interviewer: How big of an infrastructure do you have to support and maintain? It must be huge.

Reeg: Actually from a pure server footprint standpoint… we probably have fewer actual footprint servers because of techniques like virtualization that help us leverage one box to do multiple things.

Where it gets interesting is philosophically: We try to put [transaction] processing as close to our customers, the banks, as possible. When we talk about the global network, we have small servers that sit with the bank customers that connect to our network. What it does is it gives us intelligence there at the end of the network. So as a transaction comes through, we can take a look at that transaction and decide how do we best process that transaction for the benefit of all those four parties in the model.

As to processing, the majority of transactions we’re looking at relate to how do we process them as fast as possible in the most accurate way. The way to do that is by peer to peer: If you’re using your card in Europe, in London, say, and you swipe your card as you check out of hotel, we can route that transaction to the hotel’s acquiring bank in London directly to your issuing bank and get that message back for approval without ever going through St. Louis or some big data center in the middle of all that.

You can read the full article HERE.