Choosing a nic for a MIPS
There is a not so well known interaction between the processor architecture and many NICs which may have serious impact on networking performance. This article explain the issue and under its consideration grades several NICs for use on a Linux/MIPS system.
Alignment on RISC processors
The MIPS architecture specifies that the address used for a load or store instruction must be aligned to a multiple of the size of the object that is being access. That is a 16-bit integer needs to be on a 16-bit boundary etc. If this constraint is violated the processor will not perform the load or store as normal but instead take an address error exception. The Linux kernel has an exception handler which then will perform the operation in software. This comes at a serious performance impact; the exception handler might be on the order of 300 instructions. The alternative offered by the architecture is to replace the offending loads and stores by lwl/lwr, ldl/ldr, swl/swr or sdl/sdr combinations. The design choice to not support misaligned loads was made in the early 80's in order to keep the processor pipeline simple; several other RISC architectures such as the Alpha, Sparc or Sparc64 architectures also decided to not support missalignment. Some early ARM processor even went as far as to silently corrupt data for the misalignment case.
Alignment on network controllers
The easy thing to do for a hardware designer is to limit addresses to the width of the bus which in case of PC and many other I/O busses is 32-bit. So the network adapter shovels the packet into memory. Destination MAC at offset 0, source MAC at offset 6, packet type at offset 8 then the IP header starts at offset 14 with the source IP another 12 later. Oh wait. 14 + 12 bytes, that's 26 bytes and that not dividable by 4 so that IP is misaligned. And followed by another missaligned 32-bit field, the destination IP address. You see where the plot is going, as per the previous paragraph the processor is going to take even two of those expensive address error exceptions.
- Tracking down all loads in the kernel that may potentially trigger address error exceptions due to misalignment and replace them with lwl/lwr sequences is possible, would provide always have some performance impact even when the packet alignment is correct but best possible worst case performance.
- copying misaligned buffers to a properly aligned ones. Some drivers already do that; this feature is called rx_copybreak. It also helps conserving some memory which helps with, memory-wise, a bit more claustrophobic systems.
- Pick another NIC. Modern NICs are sophisticated pieces of technologies with many features that reduce the workload for the CPU. But which NICs are recommendable?
- AMD PCnet32 This is an old-timer but AMD is still manufacturing it; it exists in like 8 variants. Not terribly feature-ful but it does avoid the big mistakes that would make performance suffer big time.
- Intel E100 Another fairly old NIC. Not very featureful either but has a well supported driver as well.
- NS DP81816 is another relatively popular discrete and cheap NIC with on-chip PHY.
- 3com 3c59x. Another well supported and reasonably performing Fast Ethernet card.
- Intel E1000. A modern fast NIC with a reasonably well maintained driver. Has a little history of making trouble with certain PCI bridges.
- Broadcom BCM5902, BCM5705F etc. Tigon 3 family members. These are limited to 100Mbit/s which makes them more attractive for embedded applications. Aside of that the entire Tigon family is a family of modern decent NICs. Just like the E1000 the Tigon 3 also has a bit of a troubled history with some PCI bridges.
This card includes all cards suffering from the alignment issue.
- DEC Tulip 2104x, 2114x series. This is generally a decent mid-90's NIC but will perform badly on MIPS systems due to the alignment. There are numerous makers of cards based on these chips but in the end the heart are the same chips so they share strengths and weaknesses. Among the more popular Tulip-based cards are the Linksys EtherFast LNE100TX and several models of D-Link cards. At least some Tulips will also need a separate PHY chip which makes them relatively expensive. So all fair cards just not on a Linux/MIPS system.
Nightmare class of cards
Choose these where price is the only thing that matters.
- Realtek RTL8139. On receive and transmit every packet has to be copied. So kiss performance goodbye.
- Realtek RTL8129. Making the RTL8139 look good.
- NE2000. This NIC has been cloned numerous times and not always quite right. Originally available as an ISA card eventually even NE2000-based PCI Gigabit versions were manufactured. But the Gigabit word was mostly something for the advertisement material. Paper planes don't go supersonic. And pigs don't fly.
- Philips SAA9730 So broken never got a MIPS Atlas eval board to work reliably with NFS root.
- RFC 1925: The Twelve Networking Truths