From LinuxMIPS
Revision as of 10:01, 17 November 2008 by Philv (Talk | contribs)

Jump to: navigation, search

Cisco Systems Routers

Cisco Systems manufactures several routers with MIPS processors. Most of these routers are relatively modern, and sometimes are still prohibitively expensive on eBay or even via alternative channels of purchase. However, as many of these routers get decomissioned, they do appear on eBay at reasonable prices. A popular model with Linux/MIPS developers is the Cisco 3600-series.


Cisco has a very bland history when it comes to the class of CPUs in their hardware, often known for their (ab)use of m68k CPUs for their routers, as well as x86 CPUs for their firewalling and VPN solutions.

Cisco later switched to MIPS-based solutions from a variety of vendors, most notably IDT, PMC-Sierra and Broadcom, as well as some higher-end models being based on PowerPC processors.


Model Number CPU CPU Clockspeed Controller Slot/Bus
Aironet 1100 IBM PowerPC 405GP 200Mhz - Mini-PCI
Aironet 1200 IBM PowerPC 405GP 200Mhz - Mini-PCI
Aironet 1300 IBM PowerPC 405GP 200Mhz - Mini-PCI
1600 Motorola QUICC 68360 33MHz - WIC
1720 Motorola PowerQUICC MPC860 40MHz - WIC
1750 Motorola PowerQUICC MPC860 40MHz - WIC
1841 RM5261A-256H 250MHz Marvell GT96103A NM (PCI)
2500 Motorola 680EC30 20MHz - -
2600 Motorola PowerQUICC MPC860 40MHz - NM (PCI)
2610 Motorola PowerQUICC MPC860 ? - NM (PCI)
2610XM Motorola PowerQUICC MPC860 ? - NM (PCI)
2611 Motorola PowerQUICC MPC860 ? - NM (PCI)
2611XM Motorola PowerQUICC MPC860 ? - NM (PCI)
2620 Motorola PowerQUICC MPC860 ? - NM (PCI)
2620XM Motorola PowerQUICC MPC860 ? - NM (PCI)
2621 Motorola PowerQUICC MPC860 ? - NM (PCI)
2621XM Motorola PowerQUICC MPC860 ? - NM (PCI)
2650 Motorola PowerQUICC MPC860 ? - NM (PCI)
2650XM Motorola PowerQUICC MPC860 ? - NM (PCI)
2651 Motorola PowerQUICC MPC860 ? - NM (PCI)
2651XM Motorola PowerQUICC MPC860 ? - NM (PCI)
2691 RM7061A 160MHz ? NM (PCI)
2801 RM5261A 250MHz Marvell MV96103A AIM, HWIC, PVDM
2811 RM5261A 350MHz Marvell MV96340 NM (PCI), AIM, HWIC, PVDM
2821 RM7065C 466MHz Marvell MV96340 NM (PCI)
2851 RM7065C-466T 466MHz Marvell MV96340 NM (PCI)
3620 IDT R4600 80MHz Galileo NM (PCI)
3631 RM7061a 240MHz ? NM/WIC
3640 IDT R4600 100MHz Galileo NM (PCI)
366x RM5271 225MHz Galileo NM (PCI)
3725 RM5200 240MHz GT96100 NM (PCI)
3745 RM5200 350MHz ? NM (PCI)
3825 BCM1125H 500MHz AMD1813 NM (PCI)
3845 BCM1250 ? ? NM (PCI)
4000 Motorola 68030 40MHz ? NP ?
4500M R4700 100MHz ? NP ?
4700M R4700 133MHz ? NP ?
AS5300 R4700 150MHz ? ?
AS5350 RM7000 250MHz ? ?
AS5350XM BCM-1250 750MHz ? ?
AS5400 RM7000 250MHz ? ?
AS5400HPX RM7000 390MHz ? ?
AS5400XM BCM-1250 750MHz ? ?
7000 Motorola 68040 25MHz ? ?
7010 Motorola 68040 25MHz ? Cbus
7120 RM5271 225MHz ? PA ?
7140 RM7000 262MHz ? PA ?
7200 NPE100 R4700 150MHz ? PA ?
7200 NPE150 R4700 150MHz ? PA ?
7200 NPE175 RM5270 200MHz ? PA ?
7200 NPE200 R5000 200MHz ? PA ?
7200 NPE225 RM5271 225MHz ? PA ?
7200 NPE300 RM7000 262MHz ? PA ?
7200 NPE400 RM7000 350MHz ? PA ?
7200 NSE-1 RM7000 263MHz ? PA ?
7200 NPE-G1 BCM-1250 800MHz ? PA ?
7200 NPE-G2 PowerPC 7448 1.67GHz ? PA ?
7500 RSP1/2 R4600 100MHz ? CyBus
7500 RSP4 R5000 200MHz ? CyBus
7500 RSP8 RM7000 250MHz ? CyBus
VIP4-50 RM7000 ? ? ?
VIP4-80 RM7000 ? ? ?
12000 GRP R5000 200MHz ? ?
12000 PRP-1 PowerPC 7450 667MHz ? ?
12000 PRP-2 PowerPC 7455 1GHz ? ?

The Boot ROM

The Cisco Boot ROM firmware contains startup diagnostic code (ROM monitor, or ROMmon) as well as the boot loader for the Cisco Internetworking Operating System, or IOS for short.

During early boot, the code in the boot ROM performs a Power-on Self Test (POST) and, if all tests are passed, boot into IOS from the flash.

The Boot ROM CLI

In order to gain access to the monitor in the Boot ROM, send a break sequence to the device early on in the boot process. This can be done with a C-a f (Ctrl+a f) in Minicom, for example.

Recently boot ROMs have had an undocumented priv command. This command then can be used to gain access to several additional commands, including a debugger, disassembler and additional hardware tests. To use this command, a secret password is required, known only to Cisco representatives. However, in recent times this has been cracked and the password for many models of Cisco routers can be calculated using the tool at

This is the list of commands available on the C3640 :

rommon 6 > help
addrloop            walk 1 thru range of addresses
alias               set and display aliases command
alter               alter locations in memory
berrscan            scan range of addresses for bus errors
boot                boot up an external process
break               set/show/clear the breakpoint
call                call a subroutine at address with converted hex args
cat                 concatenate files
checksum            checksum a block of memory
clrerr              clear the error log
compare             compare two blocks of memory
confreg             configuration register utility
cont                continue executing a downloaded image
context             display the context of a loaded image
cookie              display contents of cookie PROM in hex
cpu                 cpu / system information and control
cycles              excercise the hardware with all possible cycles
dev                 list the device table
dir                 list files in file system
dis                 disassemble instruction stream
dnld                serial download a program module
dram                verify DRAM
dump                display a block of memory
echo                monitor echo command
errlog              display the error log
fdump               file dump utility
fill                fill a block of memory
flash               flash services command
frame               print out a selected stack frame
help                monitor builtin command help
history             monitor command history
ifill               fill a block of memory w/incrementing pattern
initfs              re-initialize the file system access structures
jump                call a subroutine at address with argc/argv
launch              launch a downloaded image
leds                check out the error LED
memdebug            write/read/verify scope loop
meminfo             main memory information
memloop             write or read scope loop
memtest             simple memory test
menu                main diagnostic menu
move                move a block of memory
partest             memory parity test
repeat              repeat a monitor command
reset               system reset
set                 display the monitor variables
sleep               millisecond sleep command
speed               timed performance loop
stack               produce a stack trace
sync                write monitor environment to NVRAM
sysret              print out info from last system return
tcal                timer calibration test
tlbdump             display the cpu TLB
tlbflush            flush the TLB
tlbmap              initialize a TLB mapping
tlbpid              set/display process ID number
tlbphy              search TLB for physical translation
tlbtest             test the TLB
tlbscan             scan for TLB exceptions
tlbvir              search TLB for a virtual translation
tscope              timer scope loop
unalias             unset an alias
unset               unset a monitor variable
watchdog            test watchdog rebooting of the box
xmodem              x/ymodem image download

Environment variables

Boot ROM maintains certain system configuration parameters in environment variables. For example, the $MONRC can contain a starup command sequence, $PS1 contains the command prompt, and so forth. As well, the BootROM supports basic command aliasing. Both the environment variables and the alias table are stored in NVRAM so that their values persist, even when power is off.

The Boot ROM API

The Boot ROM API provides some simple APIs for IOS (for example, putchar and version information commands). Unlike many other firmwares in the world of MIPS, the syscall opcode is used to call these firmware APIs. Note that register a0 must contain the syscall number.

A sample "Hello, world!" program can be found here.

To load this hello world program, you will need to enter in the private mode of the rommon. To accomplish this, follow the instructions from

Once done, you can test sending the file using xmodem :

rommon 8 > xmodem -r
Do not start the sending program yet...

Invoke this application only for disaster recovery.
Do you wish to continue? y/n  [n]:  y
Ready to receive file help ...

Download Complete!
program load complete, entry point: 0x80008000, size: 0x4c
Hello World!

Note that testing your programs with Dynamips works as well :

./dynamips -P 3600 ciscohello/hello.bin
Cisco Router Simulation Platform (version 0.2.7-x86)
Copyright (c) 2005-2007 Christophe Fillot.
Build date: Aug  8 2008 09:02:48

IOS image file: ciscohello/hello.bin

ILT: loaded table "mips64j" from cache.
ILT: loaded table "mips64e" from cache.
ILT: loaded table "ppc32j" from cache.
ILT: loaded table "ppc32e" from cache.
CPU0: carved JIT exec zone of 64 Mb into 2048 pages of 32 Kb.
NVRAM is empty, setting config register to 0x2142
C3600 instance 'default' (id 0):
  VM Status  : 0
  RAM size   : 128 Mb
  NVRAM size : 128 Kb
  Chassis    : 3640
  IOS image  : ciscohello/hello.bin

Loading ELF file 'ciscohello/hello.bin'...
ELF entry point: 0x80008000

C3600 'default': starting simulation (CPU0 PC=0xffffffffbfc00000), JIT enabled.
ROMMON emulation microcode.

Launching IOS image at 0x80008000...
Hello World!
Image returned to ROM.
% No memory map for code execution at 0x0
% Unable to create instruction block for vaddr=0x0
insn_page_compile: unable to create JIT block.
VM 'default': unable to compile block for CPU0 PC=0x0
Shutdown in progress...
Shutdown completed.

Be forewarned -- the Dynamips emulation of hardware is far from complete and it doesn't quite emulate all the quirks of the firmware. For example, the Dynamips loader is actually capable of loading ELF files with multiple segments.

Binary Format

IOS executables are shipped in a raw binary format (known as a .bin file to many). For MIPS-based devices, this is just conventional MIPS Big-Endian ELF, however Cisco does play a dirty trick in using a non-standard e_machine value in the ELF header. This seems to be based on the router model. For example, the Cisco 3620/40 routers have an e_machine value of 0x1e, while the 3660 routers have a value of 0x34. Cisco likely does this so people don't attempt to run IOS images intended for different models of routers.

In order to alter the e_machine value, a recent version of objcopy can be used with the switch --alt-machine-code 0x1e (in this example for a Cisco 3620/3640 series router).

As well, there are limitations placed on the binary format due to the behaviour of the software in the Boot ROM. ROMMON cannot load multiple ELF program headers, as it is very broken. Thus, to work around this problem, toolchains must be built with --target=mips-elf. CiscoLoad also helps to work around this issue, by allowing you to chainload ELF images with multiple program headers.

Finally, all symbol tables must be removed.

Boot sequence

The Boot ROM can load and execute a block of executable code (such as IOS) from various internal locations: the internal FLASH module(s), a PCMCIA Linear Flash card and (unofficially) TFTP. More recent models also support PCMCIA IDE, CompactFlash and USB flash. Also the Boot ROM supports compressed images (.gz) with embedded helper and text files that contain a command sequence (like a shell script). Nowadays -mz- IOS distributions are compressed with ZIP and have a built-in ELF ZIP decompressor "piggybacked" on top, since ZIP provides better compression.

To boot from the PCMCIA ATA or CompactFlash the media should be formatted on the CISCO router with format disk0: command. This command creates a FAT structure with second bootloader (MONLIB) resides in the hidden FAT area. More inforamtion may be found in the ATA Monlib Enhancements article. Note that this does not apply to devices with Linear Flash, which use the Cisco IFS filesystem.

Cisco 3600

Cisco 3640
The Cisco 3600-series were entry-level edge routers built on the IDT R4700 MIPS CPUs. The system controller is a Galileo GT-64010.

The Cisco 3600-series routers were designed with limited expandability in mind. As such, the Cisco 3620 has two expansion slots; the 3640 has four (as is reflected in the size of the units). These expansion modules sit on the NM expansion bus, which is essentially a modified PCI 2.1 bus with some proprietary extensions relating to EEPROM identication and what Cisco calls OIR, or Online Insertion and Removal.

Deeper technical information on the Cisco 3600-series can be found at Cisco also has made available a document describing the memory map of these routers, which can be found at [1].

NM Cards

The NM cards used in many Cisco products have a Motorola QUICC on them. While the purpose of this is unknown at this time, it is likely that the protocol decode core within the QUICC is used to assist in protocol functions; firmware appears to be loaded into the QUICC at startup time by IOS (via the IOFPGA on the board), but I have yet to capture the firmware being loaded.

One possiblity is that once Linux/MIPS is booting on this hardware, the QUICCs can be leveraged to improve routing/networking performance of the 3600-series devices if they are to be used in a networking context.

Cisco IOS Flash Filesystem (IFS)

The Cisco IOS Flash Filesystem file entries start at the first byte of flash (no additional structures). Each file is denoted by a magic number, 0xbad00b1e.

A file is denoted by a 64 byte header with:

  • 32-bit magic number (so it seems, I've never seen it change)
  • 32-bit file length (confirmed)
  • 64 bits of flags/modification time/crc (haven't bothered to figure out which byte(s) is/are which)
  • 48 bytes of filename

This is very primitive, however effective and means that code can be executed in place from the flash; the CiscoLoad bootloader actually takes advantage of this with the second stage bootloader essentially running straight from flash.

Current Status

Presently Linux does not boot on the Cisco 3600 series in any usable form. A fairly complete bootloader has been written, and can be found on Phil's server. Bugginess guaranteed. ;-)

The first (known) attempt to boot went something as follows:

[    0.000000] Linux version 2.6.28-rc4 (philippe@euclid) (gcc version 4.1.2) #8
[    0.000000] Linux on the Cisco 3600 Series.
[    0.000000] console [early0] enabled
[    0.000000] CPU revision is: 00002110 (R4700)
[    0.000000] FPU revision is: 00002110
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 04000000 @ 00000000 (usable)

Still a ways to go.

External Links