Cisco Systems Routers
Cisco Systems manufactures several routers with MIPS processors. Most of these routers are relatively modern, and sometimes are still prohibitively expensive on eBay or even via alternative channels of purchase. However, as many of these routers get decomissioned, they do appear on eBay at reasonable prices. A popular model with Linux/MIPS developers is the Cisco 3600-series.
Cisco has a very bland history when it comes to the class of CPUs in their hardware, often known for their (ab)use of m68k CPUs for their routers, as well as x86 CPUs for their firewalling and VPN solutions.
Cisco later switched to MIPS-based solutions from a variety of vendors, most notably IDT, PMC-Sierra and Broadcom, as well as some higher-end models being based on PowerPC processors.
|Model Number||CPU||CPU Clockspeed||Controller||Slot/Bus|
|Aironet 1100||IBM PowerPC 405GP||200Mhz||-||Mini-PCI|
|Aironet 1200||IBM PowerPC 405GP||200Mhz||-||Mini-PCI/Cardbus|
|Aironet 1300||IBM PowerPC 405GP||200Mhz||-||Mini-PCI|
|1600||Motorola QUICC 68360||33MHz||-||WIC|
|1720||Motorola PowerQUICC MPC860||40MHz||-||WIC|
|1750||Motorola PowerQUICC MPC860||40MHz||-||WIC|
|1841||RM5261A-256H||250MHz||Marvell GT96103A||NM (PCI)|
|2600||Motorola PowerQUICC MPC860||40MHz||-||NM (PCI)|
|2610||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2610XM||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2611||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2611XM||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2620||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2620XM||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2621||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2621XM||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2650||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2650XM||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2651||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2651XM||Motorola PowerQUICC MPC860||?||-||NM (PCI)|
|2801||RM5261A||250MHz||Marvell MV96103A||AIM, HWIC, PVDM|
|2811||RM5261A||350MHz||Marvell MV96340||NM (PCI), AIM, HWIC, PVDM|
|2821||RM7065C||466MHz||Marvell MV96340||NM (PCI)|
|2851||RM7065C-466T||466MHz||Marvell MV96340||NM (PCI)|
|3620||IDT R4600||80MHz||Galileo||NM (PCI)|
|3640||IDT R4600||100MHz||Galileo||NM (PCI)|
|4000||Motorola 68030||40MHz||?||NP ?|
|7200 NPE100||R4700||150MHz||?||PA ?|
|7200 NPE150||R4700||150MHz||?||PA ?|
|7200 NPE175||RM5270||200MHz||?||PA ?|
|7200 NPE200||R5000||200MHz||?||PA ?|
|7200 NPE225||RM5271||225MHz||?||PA ?|
|7200 NPE300||RM7000||262MHz||?||PA ?|
|7200 NPE400||RM7000||350MHz||?||PA ?|
|7200 NSE-1||RM7000||263MHz||?||PA ?|
|7200 NPE-G1||BCM-1250||800MHz||?||PA ?|
|7200 NPE-G2||PowerPC 7448||1.67GHz||?||PA ?|
|12000 PRP-1||PowerPC 7450||667MHz||?||?|
|12000 PRP-2||PowerPC 7455||1GHz||?||?|
The Boot ROM
The Cisco Boot ROM firmware contains startup diagnostic code (ROM monitor, or ROMmon) as well as the boot loader for the Cisco Internetworking Operating System, or IOS for short.
During early boot, the code in the boot ROM performs a Power-on Self Test (POST) and, if all tests are passed, boot into IOS from the flash.
The Boot ROM CLI
In order to gain access to the monitor in the Boot ROM, send a break sequence to the device early on in the boot process. This can be done with a C-a f (Ctrl+a f) in Minicom, for example.
Recently boot ROMs have had an undocumented
priv command. This command then can be used to gain access to several additional commands, including a debugger, disassembler and additional hardware tests. To use this command, a secret password is required, known only to Cisco representatives. However, in recent times this has been cracked and the password for many models of Cisco routers can be calculated using the tool at http://ers.pp.ru/cisco/priv.html.
This is the list of commands available on the C3640 :
rommon 6 > help addrloop walk 1 thru range of addresses alias set and display aliases command alter alter locations in memory berrscan scan range of addresses for bus errors boot boot up an external process break set/show/clear the breakpoint call call a subroutine at address with converted hex args cat concatenate files checksum checksum a block of memory clrerr clear the error log compare compare two blocks of memory confreg configuration register utility cont continue executing a downloaded image context display the context of a loaded image cookie display contents of cookie PROM in hex cpu cpu / system information and control cycles excercise the hardware with all possible cycles dev list the device table dir list files in file system dis disassemble instruction stream dnld serial download a program module dram verify DRAM dump display a block of memory echo monitor echo command errlog display the error log fdump file dump utility fill fill a block of memory flash flash services command frame print out a selected stack frame help monitor builtin command help history monitor command history ifill fill a block of memory w/incrementing pattern initfs re-initialize the file system access structures jump call a subroutine at address with argc/argv launch launch a downloaded image leds check out the error LED memdebug write/read/verify scope loop meminfo main memory information memloop write or read scope loop memtest simple memory test menu main diagnostic menu move move a block of memory partest memory parity test repeat repeat a monitor command reset system reset set display the monitor variables sleep millisecond sleep command speed timed performance loop stack produce a stack trace sync write monitor environment to NVRAM sysret print out info from last system return tcal timer calibration test tlbdump display the cpu TLB tlbflush flush the TLB tlbmap initialize a TLB mapping tlbpid set/display process ID number tlbphy search TLB for physical translation tlbtest test the TLB tlbscan scan for TLB exceptions tlbvir search TLB for a virtual translation tscope timer scope loop unalias unset an alias unset unset a monitor variable watchdog test watchdog rebooting of the box xmodem x/ymodem image download
Boot ROM maintains certain system configuration parameters in environment variables. For example, the $MONRC can contain a starup command sequence, $PS1 contains the command prompt, and so forth. As well, the BootROM supports basic command aliasing. Both the environment variables and the alias table are stored in NVRAM so that their values persist, even when power is off.
The Boot ROM API
The Boot ROM API provides some simple APIs for IOS (for example, putchar and version information commands). Unlike many other firmwares in the world of MIPS, the
syscall opcode is used to call these firmware APIs. Note that register a0 must contain the syscall number.
A sample "Hello, world!" program can be found here.
To load this hello world program, you will need to enter in the private mode of the rommon. To accomplish this, follow the instructions from http://ers.pp.ru/cisco/priv.html.
Once done, you can test sending the file using xmodem :
rommon 8 > xmodem -r Do not start the sending program yet... Invoke this application only for disaster recovery. Do you wish to continue? y/n [n]: y Ready to receive file help ... Download Complete! program load complete, entry point: 0x80008000, size: 0x4c Hello World!
Note that testing your programs with Dynamips works as well :
./dynamips -P 3600 ciscohello/hello.bin Cisco Router Simulation Platform (version 0.2.7-x86) Copyright (c) 2005-2007 Christophe Fillot. Build date: Aug 8 2008 09:02:48 IOS image file: ciscohello/hello.bin ILT: loaded table "mips64j" from cache. ILT: loaded table "mips64e" from cache. ILT: loaded table "ppc32j" from cache. ILT: loaded table "ppc32e" from cache. CPU0: carved JIT exec zone of 64 Mb into 2048 pages of 32 Kb. NVRAM is empty, setting config register to 0x2142 C3600 instance 'default' (id 0): VM Status : 0 RAM size : 128 Mb NVRAM size : 128 Kb Chassis : 3640 IOS image : ciscohello/hello.bin Loading ELF file 'ciscohello/hello.bin'... ELF entry point: 0x80008000 C3600 'default': starting simulation (CPU0 PC=0xffffffffbfc00000), JIT enabled. ROMMON emulation microcode. Launching IOS image at 0x80008000... Hello World! Image returned to ROM. % No memory map for code execution at 0x0 % Unable to create instruction block for vaddr=0x0 insn_page_compile: unable to create JIT block. VM 'default': unable to compile block for CPU0 PC=0x0 Shutdown in progress... Shutdown completed.
Be forewarned -- the Dynamips emulation of hardware is far from complete and it doesn't quite emulate all the quirks of the firmware. For example, the Dynamips loader is actually capable of loading ELF files with multiple segments.
IOS executables are shipped in a raw binary format (known as a .bin file to many). For MIPS-based devices, this is just conventional MIPS Big-Endian ELF, however Cisco does play a dirty trick in using a non-standard
e_machine value in the ELF header. This seems to be based on the router model. For example, the Cisco 3620/40 routers have an
e_machine value of
0x1e, while the 3660 routers have a value of
0x34. Cisco likely does this so people don't attempt to run IOS images intended for different models of routers.
In order to alter the
e_machine value, a recent version of objcopy can be used with the switch
--alt-machine-code 0x1e (in this example for a Cisco 3620/3640 series router).
As well, there are limitations placed on the binary format due to the behaviour of the software in the Boot ROM. ROMMON cannot load multiple ELF program headers, as it is very broken. Thus, to work around this problem, toolchains must be built with --target=mips-elf. CiscoLoad also helps to work around this issue, by allowing you to chainload ELF images with multiple program headers.
Finally, all symbol tables must be removed.
The Boot ROM can load and execute a block of executable code (such as IOS) from various internal locations: the internal FLASH module(s), a PCMCIA Linear Flash card and (unofficially) TFTP. More recent models also support PCMCIA IDE, CompactFlash and USB flash. Also the Boot ROM supports compressed images (.gz) with embedded helper and text files that contain a command sequence (like a shell script). Nowadays -mz- IOS distributions are compressed with ZIP and have a built-in ELF ZIP decompressor "piggybacked" on top, since ZIP provides better compression.
To boot from the PCMCIA ATA or CompactFlash the media should be formatted on the CISCO router with format disk0: command. This command creates a FAT structure with second bootloader (MONLIB) resides in the hidden FAT area. More inforamtion may be found in the ATA Monlib Enhancements article. Note that this does not apply to devices with Linear Flash, which use the Cisco IFS filesystem.
The Cisco 3600-series routers were designed with limited expandability in mind. As such, the Cisco 3620 has two expansion slots; the 3640 has four (as is reflected in the size of the units). These expansion modules sit on the NM expansion bus, which is essentially a modified PCI 2.1 bus with some proprietary extensions relating to EEPROM identication and what Cisco calls OIR, or Online Insertion and Removal.
Deeper technical information on the Cisco 3600-series can be found at http://www.cisco.com/warp/public/63/36xx-arch.pdf. Cisco also has made available a document describing the memory map of these routers, which can be found at .
The NM cards used in many Cisco products have a Motorola QUICC on them. While the purpose of this is unknown at this time, it is likely that the protocol decode core within the QUICC is used to assist in protocol functions; firmware appears to be loaded into the QUICC at startup time by IOS (via the IOFPGA on the board), but I have yet to capture the firmware being loaded.
One possiblity is that once Linux/MIPS is booting on this hardware, the QUICCs can be leveraged to improve routing/networking performance of the 3600-series devices if they are to be used in a networking context.
Cisco IOS Flash Filesystem (IFS)
The Cisco IOS Flash Filesystem file entries start at the first byte of flash (no additional structures). Each file is denoted by a magic number, 0xbad00b1e.
A file is denoted by a 64 byte header with:
- 32-bit magic number (so it seems, I've never seen it change)
- 32-bit file length (confirmed)
- 64 bits of flags/modification time/crc (haven't bothered to figure out which byte(s) is/are which)
- 48 bytes of filename
This is very primitive, however effective and means that code can be executed in place from the flash; the CiscoLoad bootloader actually takes advantage of this with the second stage bootloader essentially running straight from flash.
Presently Linux does not boot on the Cisco 3600 series in any usable form. A fairly complete bootloader has been written, and can be found on Phil's server. Unless you're looking for something specific, use the latest version of CILO here. Bugginess guaranteed. ;-)
Do not use versions of CILO prior to 0.4 -- a major bug in the segment loading code caused issues with interrupt handling, among other things.
Thus far, the boot gets this far:
[ 0.000000] Linux version 2.6.28-rc9 (philippe@euclid) (gcc version 4.1.2) #8 [ 0.000000] Cisco 3620 Multiservice Router with 67108864B of RAM. [ 0.000000] console [early0] enabled [ 0.000000] CPU revision is: 00002110 (R4700) [ 0.000000] FPU revision is: 00002110 [ 0.000000] Determined physical RAM map: [ 0.000000] memory: 04000000 @ 00000000 (usable) [ 0.000000] Initrd not found or empty - disabling initrd [ 0.000000] Zone PFN ranges: [ 0.000000] Normal 0x00000000 -> 0x00004000 [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map active PFN ranges [ 0.000000] 0: 0x00000000 -> 0x00004000 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pa6 [ 0.000000] Kernel command line: console=ttyS0 [ 0.000000] Primary instruction cache 16kB, VIPT, 2-way, linesize 32 bytes. [ 0.000000] Primary data cache 16kB, 2-way, VIPT, cache aliases, linesize 32s [ 0.000000] DEBUG: iofpga_irq = 00 [ 0.000000] PID hash table entries: 256 (order: 8, 1024 bytes) [ 0.000000] MIPS counter frequency 50008020Hz [17179569.188000] Console: colour dummy device 80x25 [17179569.192000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes) [17179569.196000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes) [17179569.240000] Memory: 61168k/65536k available (2651k kernel code, 4300k res) [17179569.244000] SLUB: Genslabs=6, HWalign=32, Order=0-3, MinObjects=0, CPUs=11 [17179569.248000] Calibrating delay loop... 98.81 BogoMIPS (lpj=197632) [17179569.332000] Security Framework initialized [17179569.336000] SELinux: Disabled at boot. [17179569.336000] Mount-cache hash table entries: 512 [17179569.344000] Initializing cgroup subsys ns [17179569.348000] Initializing cgroup subsys cpuacct [17179569.364000] net_namespace: 704 bytes [17179569.368000] NET: Registered protocol family 16 [17179569.500000] pci 0000:00:06.0: unknown header type 5e, ignoring device [17179569.524000] pci 0000:00:05.0: BAR 0: error updating (0x12001000 != 0x0010) [17179569.528000] pci 0000:00:05.0: BAR 1: error updating (0x12001010 != 0x1100) [17179569.532000] pci 0000:00:0d.0: BAR 0: error updating (0x12001020 != 0x0010) [17179569.536000] pci 0000:00:0d.0: BAR 1: error updating (0x12001030 != 0x1100) [17179569.540000] pci 0000:00:0e.0: BAR 0: error updating (0x12001040 != 0x1001) [17179569.544000] pci 0000:00:0e.0: BAR 1: error updating (0x12001050 != 0x0010) [17179569.548000] pci 0000:00:0f.0: BAR 0: error updating (0x12001060 != 0x1001) [17179569.552000] pci 0000:00:0f.0: BAR 1: error updating (0x12001070 != 0x0010) [17179569.556000] pci 0000:00:14.0: BAR 0: error updating (0x12001080 != 0x1100) [17179569.576000] NET: Registered protocol family 8 [17179569.580000] NET: Registered protocol family 20 [17179569.624000] NET: Registered protocol family 2 [17179569.668000] IP route cache hash table entries: 1024 (order: 0, 4096 bytes) [17179569.676000] TCP established hash table entries: 2048 (order: 2, 16384 byt) [17179569.684000] TCP bind hash table entries: 2048 (order: 1, 8192 bytes) [17179569.688000] TCP: Hash tables configured (established 2048 bind 2048) [17179569.696000] TCP reno registered [17179569.712000] NET: Registered protocol family 1 [17179569.724000] Registering console device: [17179569.744000] audit: initializing netlink socket (disabled) [17179569.748000] type=2000 audit(0.566:1): initialized [17179569.956000] msgmni has been set to 119 [17179569.976000] alg: No test for stdrng (krng) [17179569.980000] io scheduler noop registered [17179569.984000] io scheduler anticipatory registered [17179569.992000] io scheduler deadline registered [17179569.996000] io scheduler cfq registered (default) [17179573.116000] Serial: 8250/16550 driver4 ports, IRQ sharing disabled [17179573.164000] serial8250.0: ttyS0 at MMIO 0x1e840000 (irq = 5) is a 16550A *hang*
Still a ways to go.
You can grab the preliminary kernel support patch here