Topic Title: AMD CPUs vs IRQs unfair distribution in Linux
Topic Summary:
Created On: 03/18/2013 09:35 PM
Status: Post and Reply
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
 03/18/2013 09:35 PM
User is offline View Users Profile Print this message

Author Icon
AdrianB
Peon

Posts: 2
Joined: 03/18/2013

Hi,

I'm new on this forum. I got a strange issue about AMD CPUs.

My problem is related to the IRQ unfair distribution in Linux. All my AMD CPUs has a strange behavior on same or different mother board.

The last one is clear that the CPU is doing this job and I wondering if somebody from AMD can give me some tehnical explications.

For the begining: I'm using a mother board build by ASUS which is an M5A99X with a AMD 990X North bridge and SB950 South bridge.
The CPUs I've tested are: AMD Athlon  II X4 645 Processor and AMD FX-8120

A lspci (Athlon):

00:00.0 Host bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx0 port B) (rev 02)
00:00.2 IOMMU: Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory Management Unit (IOMMU)
00:02.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port B)
00:04.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port D)
00:05.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port E)
00:06.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port F)
00:07.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port G)
00:11.0 SATA controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
00:16.0 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Link Control
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV670 [Radeon HD 3870]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI RV670/680 HDMI Audio [Radeon HD 3690/3800 Series]
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 09)
03:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
06:05.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
07:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID PCIe (rev 01)

cat /proc/interrupts (Athlon unfair distribution):
root@Sydney:~# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3
   0:        293       5815     301622   16652740   IO-APIC-edge      timer
   1:          0          0          0          2   IO-APIC-edge      i8042
   8:          0          0          0          1   IO-APIC-edge      rtc0
  12:          0          0          0          4   IO-APIC-edge      i8042
  16:          0          0          3        602   IO-APIC-fasteoi   snd_hda_intel
  17:          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1
  18:          0          0          0          2   IO-APIC-fasteoi   ohci_hcd:usb6, ohci_hcd:usb10
  19:          0          0          0          0   IO-APIC-fasteoi   ahci
  20:          9        426      29780     348768   IO-APIC-fasteoi   ohci_hcd:usb9, 3w-9xxx, eth0
  80:          0          0         20        245   PCI-MSI-edge      xhci_hcd
  85:          0          0          1         30   PCI-MSI-edge      snd_hda_intel
 NMI:         24         24         24         29   Non-maskable interrupts
 LOC:    6850562    7039838    6869557    2874394   Local timer interrupts
 PMI:         24         24         24         29   Performance monitoring interrupts
 RES:     915760     921301     906348     873543   Rescheduling interrupts
 CAL:      32441      32218      32494      21018   Function call interrupts
 TLB:       1517       1436       1608       1496   TLB shootdowns
 MCP:          5          5          5          5   Machine check polls
 ERR:          1
 MIS:          0
cat /proc/interrupts (FX-8120 fair IRQ distribution over all cores):
root@Sydney:/home/vbox/VirtualBox VMs/LMQoS Hosting# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
   0:         40          1          0          0          0          0          0          1   IO-APIC-edge      timer
  16:         76         76         78         78         77         77         77         76   IO-APIC-fasteoi   snd_hda_intel
  20:       2958       2971       2963       2966       2967       2970       2978       3055   IO-APIC-fasteoi   ohci_hcd:usb8, 3w-9xxx, eth0
  83:         45         41         42         40         40         44         41         44   PCI-MSI-edge      xhci_hcd
  91:          4          4          3          4          4          4          4          4   PCI-MSI-edge      snd_hda_intel
 LOC:       8768       9209       9266       8613       8656       7799       8115      10953   Local timer interrupts
 RES:      46182      15146      46987      15525      46999      15146      47537      13963   Rescheduling interrupts
 CAL:        774        779        781        851        845        790        803        854   Function call interrupts
 TLB:       2538       1944       2544       2158       2369       2376       2600       2177   TLB shootdowns
 MCP:          2          2          2          2          2          2          2          2   Machine check polls
 ERR:          1
As you can see the same hardware different behaviors for IRQ distribution!
For many years I've put the problem on mother board but with this FX CPU I saw that the main problem wasn't the MB but the CPU.
The unfair IRQ distribution I got with Quad-Core AMD Opteron(tm) Processor 1352, Phenom(tm) 8650, Phenom II X4 965, Athlon X2. All those CPUs have the same behavior: over 90% of IRQ goes to the last core, 2-4% goes to the last but one and so on.
My question is: why this behavior ultil FX CPUs? What is different between FX and the rest of the CPUs?
The advantage of the IRQ distribution over all cores is that in my case I'm using those PC in routing and the power of the packet proccess can rise up to 3,4 or more times.
The Opteron CPU on one core can reach over 220Kpps, but with power of 3 or 4 CPU I can reach maybe at least 2 times.
Best regards.

 03/19/2013 08:36 PM
User is offline View Users Profile Print this message

Author Icon
AdrianB
Peon

Posts: 2
Joined: 03/18/2013

Hm .. no body can give an answer to this problem?

Or a tip how to buy a CPU that is IRQ fair capable?

 

I've reproduce the same issue on a M4A89GTD motherboard:

 

root@debian:/home/user/Desktop# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : AMD Phenom(tm) II X4 965 Processor
stepping : 3
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr
bogomips : 6822.96
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

 

 

root@debian:/home/user/Desktop# lspci
00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge (ext gfx port 0)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge (PCIE port 5)
00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [IDE mode] (rev 40)
00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller (rev 40)
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller (rev 40)
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
00:15.0 PCI bridge: ATI Technologies Inc Device 43a0
00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Link Control
01:00.0 VGA compatible controller: ATI Technologies Inc Device 68ba
01:00.1 Audio device: ATI Technologies Inc Juniper HDMI Audio [Radeon HD 5700 Series]
02:00.0 SATA controller: JMicron Technology Corp. JMB361 AHCI/IDE (rev 02)
02:00.1 IDE interface: JMicron Technology Corp. JMB361 AHCI/IDE (rev 02)
03:07.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)

root@debian:/home/user/Desktop# cat /proc/interrupts 
            CPU0       CPU1       CPU2       CPU3       
   0:         91          4       2544      60573   IO-APIC-edge      timer
   1:          0          0          0          2   IO-APIC-edge      i8042
   4:          0          0          0          2   IO-APIC-edge    
   8:          0          0          0          1   IO-APIC-edge      rtc0
   9:          0          0          0          2   IO-APIC-fasteoi   acpi
  12:          0          0          0          4   IO-APIC-edge      i8042
  14:          0          0          0          0   IO-APIC-edge      pata_atiixp
  15:          0          0          0          0   IO-APIC-edge      pata_atiixp
  16:          0          0          0        444   IO-APIC-fasteoi   HDA Intel
  17:          0          3        303      22299   IO-APIC-fasteoi   ehci_hcd:usb1, ehci_hcd:usb3, ehci_hcd:usb5
  18:          0          0         87      13512   IO-APIC-fasteoi   ohci_hcd:usb2, ohci_hcd:usb4, ahci, ohci_hcd:usb6, ohci_hcd:usb7
  19:          0          7        287       4113   IO-APIC-fasteoi   pata_jmicron, ahci, HDA Intel
  22:          0          0          0          8   IO-APIC-fasteoi   firewire_ohci
  27:          0          1       2050     113321   PCI-MSI-edge      eth0
 NMI:          0          0          0          0   Non-maskable interrupts
 LOC:      17036      16376      13146       3264   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0   Performance monitoring interrupts
 PND:          0          0          0          0   Performance pending work
 RES:      68802      60152      21946      12150   Rescheduling interrupts
 CAL:        207        220        217         49   Function call interrupts
 TLB:        536       1426       1042        746   TLB shootdowns
 TRM:          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0   Machine check exceptions
 MCP:          2          2          2          2   Machine check polls
 ERR:          0
 MIS:          0

 

 

root@debian:/home/user/Desktop# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 21
model : 1
model name : AMD FX(tm)-8120 Eight-Core Processor          
stepping : 2
cpu MHz : 3110.753
cache size : 2048 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 8
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc nonstop_tsc extd_apicid pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt nodeid_msr arat
bogomips : 6221.50
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate [9]

 

root@debian:/home/user/Desktop# lspci
00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge (ext gfx port 0)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge (PCIE port 5)
00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [IDE mode] (rev 40)
00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller (rev 40)
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller (rev 40)
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
00:15.0 PCI bridge: ATI Technologies Inc Device 43a0
00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices [AMD] Device 1600
00:18.1 Host bridge: Advanced Micro Devices [AMD] Device 1601
00:18.2 Host bridge: Advanced Micro Devices [AMD] Device 1602
00:18.3 Host bridge: Advanced Micro Devices [AMD] Device 1603
00:18.4 Host bridge: Advanced Micro Devices [AMD] Device 1604
00:18.5 Host bridge: Advanced Micro Devices [AMD] Device 1605
01:00.0 VGA compatible controller: ATI Technologies Inc Device 68ba
01:00.1 Audio device: ATI Technologies Inc Juniper HDMI Audio [Radeon HD 5700 Series]
02:00.0 SATA controller: JMicron Technology Corp. JMB361 AHCI/IDE (rev 02)
02:00.1 IDE interface: JMicron Technology Corp. JMB361 AHCI/IDE (rev 02)
03:07.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller
(rev c0)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)

 

root@debian:/home/user/Desktop# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7      
   0:         47          0          0          0          0          0          0          0   IO-APIC-edge      timer
  1:          2          0          0          0          0          0          0          0   IO-APIC-edge      i8042
  4:          1          0          0          0          0          0          0          0   IO-APIC-edge    
  8:          0          0          0          0          0          0          0          1   IO-APIC-edge      rtc0
  9:          1          0          0          0          1          0          0          0   IO-APIC-fasteoi   acpi
 12:          4          0          0          0          0          0          0          0   IO-APIC-edge      i8042
 14:          0          0          0          0          0          0          0          0   IO-APIC-edge      pata_atiixp
 15:          0          0          0          0          0          0          0          0   IO-APIC-edge      pata_atiixp
 16:         49         51         51         56         56         59         58         65   IO-APIC-fasteoi   HDA Intel
 17:       3139       3118       3148       3161       3125       3129       3205       3608   IO-APIC-fasteoi  ehci_hcd:usb1, ehci_hcd:usb3, ehci_hcd:usb5
 18:       2236       2298       2228       2211       2235       2214       2314       2969   IO-APIC-fasteoi   ohci_hcd:usb2, ahci, ohci_hcd:usb4, ohci_hcd:usb6, ohci_hcd:usb7
 19:        607        617        615        575        624        605        642        905   IO-APIC-fasteoi   ahci, pata_jmicron, HDA Intel
 22:          2          1          0          2          1          1          2          1   IO-APIC-fasteoi   firewire_ohci
 27:      56766      56736      56796      56861      56767      56788      57015      59108   PCI-MSI-edge      eth0
NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC:      33652      29861      24530      20631      13037      11751      11629      12299   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0          0          0          0          0   Performance monitoring interrupts
PND:          0          0          0          0          0          0          0          0   Performance pending work
RES:     121979     118723      28845      14445       9946       7037       5266       5974   Rescheduling interrupts
CAL:        140        160        132        160        163        156        170        165   Function call interrupts
TLB:        391       1779       1162        687        558        493        491        449   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:          2          2          2          2          2          2          2          2   Machine check polls
ERR:          0
MIS:          0

 

 

 

Statistics
84954 users are registered to the AMD Support and Game forum.
There are currently 5 users logged in.

FuseTalk Hosting Executive Plan v3.2 - © 1999-2014 FuseTalk Inc. All rights reserved.