Hi everybody,

I'm playing with a 4 socket NUMA system. Each NUMA node consist of a 4 core Opteron 8378 processor. Three HyperTransport links are available in each node, so the resulting NUMA topology is a square:

N0 -- N1

| |

N2 -- N3

I want to known the NUMA distance between nodes. I expect the following distances:

N0 N1 N2 N3

N0 0 1 1 2

N1 1 0 2 1

N2 1 2 0 1

N3 2 1 1 0

Under Linux I use numactl --hardware to read distances from hardware. It reports:

node 0 1 2 3

0: 10 20 20 20

1: 20 10 20 20

2: 20 20 10 20

3: 20 20 20 10

The misbehavior is a known problem. It seems that the Opteron BIOS does not export right distances (see

http://kerneltrap.org/mailarch.../4/3/4428878/thread).
In order to observe the real topology I've run some test throuh the numademo program, distributed within the numactl package. Consider the tests:

pin current process to node i, exec memset on all other nodes

I expect results are better when the memory accessed is near the current node. So I've started by pinning a process to node 0 and writing to other nodes (Each test wrote 128mb of data):

alloc on node 1 memset Avg 2920.52 MB/s Max 2938.22 MB/s Min 2808.14 MB/s

alloc on node 2 memset Avg 2965.80 MB/s Max 2967.84 MB/s Min 2962.34 MB/s

alloc on node 3 memset Avg 2116.58 MB/s Max 2122.49 MB/s Min 2081.02 MB/s

This confirm my ipotesis: node 0 is near node 1 and node 2, while node 3 is 2 hops distant.

I tried the dual configuration (pin process to node 3), and the results are coherent whit those of the previous test.

At last I've tried pinning the process to nodes 1 and 2. Here are the results for node 1:

alloc on node 0 memset Avg 2698.34 MB/s Max 2701.21 MB/s Min 2689.30 MB/s

alloc on node 2 memset Avg 2954.00 MB/s Max 2957.12 MB/s Min 2948.54 MB/s

alloc on node 3 memset Avg 2644.85 MB/s Max 2646.67 MB/s Min 2642.71 MB/s

The nearest node is 2, while 0 and 3 are the farest!

If I pin the process to node 2, I have:

alloc on node 0 memset Avg 2602.85 MB/s Max 2605.16 MB/s Min 2596.69 MB/s

alloc on node 1 memset Avg 2940.77 MB/s Max 2943.11 MB/s Min 2933.34 MB/s

alloc on node 3 memset Avg 2675.09 MB/s Max 2677.07 MB/s Min 2673.23 MB/s

The nearest node is 1, the farest are 0 and 3!

Only the first two test are coherent with the hardware topology. Anyone has any idea of why I got these results?

Thanks,

speziale.ettore@gmail.com