This document reflects what I have learned through porting several MIPS machines and other related Linux work. Hopefully it will help beginners to get started and give the experienced a reference point.
This document goes through all the major steps to port Linux to a MIPS machine. The focus can perhaps be called "MIPS machine abstraction layer", i.e., the interface between machine-specific code and, mostly, MIPS common code. Another useful document focuses on "Linux hardware abstraction layer", i.e., the interface between Linux common code and architecture-specific code. The document is written by C Hanish Menon (www.hanishkvc.com).
There are some notations used in this document.
Reminder for incomplete part
Really need your help on this
Here is my opinion. What do you think?
Thanks to the person who pointed this out
Additional note added later, but the original comment may be still useful
- 1 An overview
- 2 Hello, world!
- 3 Add your own code
- 4 Early printk
- 5 Serial driver and console
- 6 KGDB
- Know C programming.
- Have some knowledge of OS concepts, such as interrupt handling, system calls, memory management.
- Know how to configure and make Linux kernel. You can find many helps on this if you are not very comfortable.
- Have some knowledge of MIPS CPU. More than likely you will need to deal with CP0 registers, enable or disable interrupts, etc..
- You don't have to be an expert in MIPS assembly, but total ignorance of it might make you handicapped in some situations.
- Obviously, you need a MIPS hardware to play with.
- Finally but most importantly, you need a willing-to-learn heart and perhaps many restless debugging hours. :-)
It is also highly recommanded to read through the Linux MIPS HOWTO by Ralf Bächle, firstname.lastname@example.org. By the way, as part of the pre-requisite, you should also remember Ralf's name. :-)
Kernel source trees
The common MIPS tree is the CVS tree at linux-mips.org. See the instructions in "Anonymous CVS servers" section in Linux/MIPS HOWTO. The current kernel version as of 2004/01/26 is 2.6.1. You can always check out earlier stable revisions by using "linux_2_4" or even "linux_2_2" branch tag.
For various reasons, a kernel tree may leave bugs there for a quite long time before a suitable fix is checked in. There are various places to get patches. Here are some of the more common ones:
Jun Sun patches Linux/MIPS FTP archive Maciej W. Rozyki patches Brad LaRonde's patches
Cross-compilation and toolchains
More than likely your MIPS box does not run Linux yet (why would you bother otherwise?). Therefore you will need another machine to build the kernel image. Once the image is built, you need to download this image to your MIPS machine and let it run your MIPS kernel. This is called cross-development. Your MIPS box is often called the target machine and the machine used to build the kernel image is called the host machine.
Cross-development is common for developing on embedded targets, because usually embedded targets do not have enough power or the peripherals to develop natively.
The most common host machine is probably Linux on i386/PCs
You need to have cross-development tools setup on your host before you can start. Several years of experience have show that hile you can find instructions to build cross-compilation tools, your best bet is probably to get some ready-made ones. Another reason is that a full toolchain build has become quite time consuming so you probably want to avoid it.
MontaVista used to offer for free Journeyman edition, which includes a full featured toolchain. Unfortunately, it does not offer that anymore. Instead you can download the preview kit, which includes a "slim" version of toolchain. You can get the kit from http://www.mvista.com/previewkit/index.html
Dan Kegel has a set of scripts that build cross-compiling tools. You can check it out here.
The following are links to pre-build toolchains, instructions to build your own toolchain and finally pre-compiled distributions for MIPS boards:
- Brad LaRonde's cross toolchain for Linux
- Steve Hill's toolchains for glibc and uClibc
- Toolchains from MIPS Technologies
- Distributions for MIPS
Overall porting steps
Depending on your specific cases, some of the following steps can be skipped.
- Hello World! - Get board setup, serial porting working, and print out "Hello, world!" through the serial port.
- Add your own code.
- Get early printk working - Make the first MIPS image and see the printk output from kernel.
- Serial driver and serial console - Get the real printk working with the serial console.
- KGDB - KGDB can be enormously helpful in your development. It is highly recommended and it is not that difficult to set up.
- CPU support - If your MIPS CPU is not currently supported, you need to add new code that supports it.
- Board specific support - Create your board-specific directory. Setup interrupt routing/handling and kernel timer services.
- PCI subsystem - If your machine has PCI, you need to get the PCI subsystem working before you can use PCI devices.
- Ethernet drivers - You should already have the serial port working before attempting this. Usually the next driver you want is the ethernet driver. With ethernet driver working, you can set up a NFS root file system which gives you a fully working Linux userland.
- ROMFS root file system - Alternatively you can create a userland file system as a ROMFS image stored in a ramdisk.
- Some words on Debugging.
In cross development, the serial port is usually the most important interface: That is where you can see anything happening! It might be worthwhile to make sure you get serial porting work before you even start playing with Linux. You can find the sample code or gzipped tar ball of a stand-alone program that can do printf. Such a program can even be useful in later debugging staging, e.g., printing out hardware register values.
Before you rush to type 'make', check and modify the following configurations:
- The sample code assumes R4K style CP0 structure. It should apply to most CPUs named above number 4000 and the recent MIPS32/MIPS64.
- Check if you have 1MB RAM size. (You really should have at 1MB to run Linux at all.) It is recommanded you have 8MB RAM or more.
- Is your serial port a standard UART type? If yes, modify the serial code and parameters. If not, you will have to supply your own functions to utilize the UART.
- What is your cross-tool name and path? Modify the Makefile accordingly.
Now, fire your "make" command.
Depending on your downloader on your MIPS box, you may need to generate ELF image, binary image or a SREC image.
Download the barebone image to your target and give it a run! Connect the serial port to your host machine. Start minicom and hopefully you can see the "Hello, world!" message.
- Make sure your bootloader downloads the image to uncached KSEG1 segment. If your bootloader downloads to the cached KSEG0 area, you will want to run the image from the KSEG0 area too.
- If your bootloader has already initialized the serial port, you may want to skip your own initialization.
- Did you set up minicom correctly? Test it with other machines.
- Hopefully it is not the toolchain problem...
Add your own code
Let us add some code to the tree and make a Linux image. For conveninence sake, let us say we are porting Linux to a MIPS board called Aloha.
Create the right directory for your board
Your code for a new board can be classified into board-support code (or board-specific code) and drivers. Driver code should be placed under the 'drivers' directory and board specific code should be placed under 'arch/mips' directory.
The easiest choice is to create a directory called 'arch/mips/aloha'.
However, a couple of other considerations might make it slightly complicated.
- If Aloha uses a chipset or System on a Chip (SOC) that is already supported or belongs to a bigger family, such as NEC VR41xx and gt64120, it makes sense to put Aloha code under those sub-directories. You can re-use and share a lot of common code.
- Similarly, if Aloha is the first board that uses a chipset or SOC which is expected to be used in many other boards, you may want to create similar directory structure. However, if you are not sure, just create your own board specific directory.
In the past people have created directories based on the board manufacturer's name, such as "mips-boards". This generally is not a good idea. It is almost certain that some of these boards do not share anything common at all.
To make things worse, sometimes boards made by different companies use the same chipset or SOC. Now what are you going to do? Are you going to duplicate the common code? Or are you going stick one company's board under another company's name?
For header files, you usually create similar directory or header files under include/asm-mips. [DEBATE] For board specific header files, I would encourage people to place them under the corresponding 'arch/mips' directory if possible.
In our exmaple, we will create 'arch/mips/aloha' directory.
Write the minimum Aloha code
Let us write some code for the Aloha board which can generate a complete Linux image without complaining about missing symbols.
Go to this directory to browse 'arch/mips/aloha' directory. Or download the gzipped file of the directory.
Obviously the code is not complete yet, but if you follow the following steps and everything is correct, you should be able to generate a Linux/MIPS kernel image of your very own!
Hook up your code with the Linux tree
Most of the steps are fairly straightforward:
- include/asm-mip/bootinfo.h - Add your machine group ID, machine group name and Aloha machine ID.
- arch/mips/kernel/setup.c - Add 'aloha_setup' function declaration and invocation code.
- arch/mips/Makefile - Add a section that links your Aloha code in.
# # Hawaii Aloha board # ifdef CONFIG_ALOHA SUBDIRS += arch/mips/aloha LIBS += arch/mips/aloha/aloha.o LOADADDR += 0x80002000 endif
LOADADDR is the starting address for your Linux image when it is loaded into RAM. Note that the first 0x200 bytes are used by the exception vectors on most CPUs. Some CPUs will requries a larger space, so modify the LOADADDR accordingly. Due to the linker's addressing limit, the start address is aligned on a 8KB boundary, so setting your LOADADDR to 0x80002000 should be reasonable.
- arch/mips/config-shared.in - Add necessary config information for Aloha board.
- Add the following to 'Machine selection'.
dep_bool 'Support for Hawaii Aloha board (EXPERIMENTAL)' CONFIG_ALOHA $CONFIG_EXPERIMENTAL
- Add a set of default configs for the board, which depends on the features and drivers that Linux port will supports. Here is a very simple example for our minimum Aloha board configurations.
if [ "$CONFIG_ALOHA" = "y" ]; then define_bool CONFIG_CPU_R4X00 y define_bool CONFIG_CPU_LITTLE_ENDIAN y define_bool CONFIG_SERIAL y define_bool CONFIG_SERIAL_MANY_PORTS y define_bool CONFIG_NEW_IRQ y define_bool CONFIG_NEW_TIME_C y define_bool CONFIG_SCSI n fi
There are two kinds of configuration options here. The first kind are those you cannot select interactively during 'make config' or 'make menuconfig' or 'make xconfig'. Examples are CONFIG_NEW_IRQ and CONFIG_NEW_TIME_C. You must put them here, or else they will not get selected. The second kind of options are those you can select interactively, such as CONFIG_CPU_R4X00 and CONFIG_SERIAL. However you may also put them here if you know which selection is right for the board. This way people will make fewer mistakes when they configure for the board.
For instant gratification, you can find a complete patch for adding the Aloha board support to the Linux/MIPS CVS tree checked out on January 20, 2004.
Configure and build a kernel image
Now you are ready to run your favorite configuration tool. Since we do not have much code added yet, do not be too greedy with selecting options. Just pick a couple of simple options such as the serial and serial console.
- If you denoted the Aloha board support to be EXPERIMENTAL, select 'Prompt for development and/or incomplete code/drivers' under 'Code maturity level options'.
- Select 'Support for Hawaii Aloha board' and unselect all other machines under 'Machine selection'.
- Select the right CPU. Under 'CPU selection' select your CPU. If there is no entry for the CPU on your board, you will need to add support for it. Most recent CPUs can generally run to some degree with CPU_R4X00.
- Under 'Character devices', select 'Standard/generic (8250/16550 and compatible UARTs) serial support' and 'Support for console on serial port'. Unselect the 'Virtual terminal' option.
- Under the 'Kernel hacking' option, select 'Are you using a crosscompiler'.
- For other options either take the default or select 'no'.
Here is a sample minimum config for our Aloha board.
Before you type 'make', double-check the 'arch/mips/Makefile' and make sure the cross-toolchain program names are correct and in your execution path i.e. your PATH environment variable.
Now type 'make dep' and 'make'. Then wait for a miracle to happen!
Assuming you are lucky and actually generate an image from the last chapter, don't bother running it because you won't see anything. This is not strange because all our board-specific code is empty and we have not told Linux kernel anything about our serial port or I/O devices yet.
The sign of a live Linux kernel comes from the output of printk, which is routed to the first console. Since we have configured a serial console, we should be able to see something on the serial wire if we have set it up correctly.
Unfortunately, setup of the serial console happens much later during the kernel startup process. (See Appendix A for a chart of the kernel start-up sequence). Chances are your new kernel probably dies even before that. That is where the early printk patch comes in handy. It allows you to see printk as early as the first line of C code.
By the way the first line of C code for Linux MIPS is the first line of code of 'init_arch()' function in the 'arch/mips/setup.c' file.
For kernel version earlier than 2.4.10, you can find the early printk patch here for boards with standard UART serial ports. Starting from 2.4.10 and beyond, a new printk patch is needed. If you have already got the stand-alone "Hello, world!" program running, the early printk should be easy to get going, and you should have printk output from the Linux kernel very soon.
Serial driver and console
While early printk is rather useful, you still need to get the real serial driver working.
Assuming you have a standard serial port, there are two ways to add serial support: static defines and run-time setup.
With static defines, you modify the 'include/asm-mips/serial.h' file. Looking through the code, it is not difficult to figure out how to add support for your board's serial port(s).
As more boards are supported by Linux/MIPS, the 'serial.h' file gets crowded. One potential solution is to do run-time serial setup. Sometimes run-time serial setup is necessary if any of the parameters can only be detected at the run-time where settings are read from a non-volatile memory device or an option is passed on the kernel command line.
There are two elements to consider for doing run-time serial setup:
- Reserve the 'rs_table' size, see the 'drivers/char/serial.c' file. Unfortunately there is not a clean way to accomplish this yet. A temporary workaround is to define CONFIG_SERIAL_MANY_PORTS in 'arch/mips/config-shared.in' for your board. This configuration reserves up to 64 serial port entries for your board!
- Call the 'early_serial_setup()' routine in your board setup routine. Here is a piece of sample run-time initialization code.
Most of the parameter settings are rather obvious. Here is a list of some less obvious ones:
- Only used for run-time serial configuration. It is the index into the 'rs_table array'.
- io_type determines how your serial registers are accessed. Two common types are SERIAL_IO_PORT and SERIAL_IO_MEM. A SERIAL_IO_PORT type driver uses the inb/outb macros to access registers whose base address is at port. In other words, if you specify SERIAL_IO_PORT as the io_type, you should also specify the port parameter.
For SERIAL_IO_MEM, the driver uses readb/writeb macros to access regsiters whose address is at iomem_base plus a shifted offset. The number of digits shifted is specified by iomem_reg_shift. For example, all the serial registers are placed at 4-byte boundary, then you have an iomem_reg_shift of 2.
Generally SERIAL_IO_PORT is for serial ports on an ISA bus and SERIAL_IO_MEM is for memory-mapped serial ports. There are also SERIAL_IO_HUB6 and SERIAL_IO_GSC. The HUB6 was a multi-port serial card produced by Bell Technologies. The GSC is a special bus found on PA-RISC systems. These options will not be used on any MIPS boards.
Non-standard serial ports
If you have a non-standard serial port, you will have to write your own serial driver.
Some people derive their code from the standard serial driver. Unfortunately this is a very daunting task. The 'drivers/char/serial.c' file has over 6000 lines of code!
Fortunately, there is an alternative [THANKS: Fillod Stephane]. There is a generic serial driver, 'drivers/char/generic_serial.c'. This file provides a set of serial routines that are hardware independent. Then your task is to provide only the routines that are hardware dependent such as the interrupt handler routine and low-level I/O functions. There are plenty of examples. Just look inside the 'drivers/char/Makefile' and look for drivers that link with 'generic_serial.o' file.
[HELP: Would appreciate if you can share your experience of writing proprietary serial driver code or using the 'generic_serial.c' file.]
For many Linux kernel developers, KGDB is a life-saving tool. With KGDB, you can debug the kernel while it runs! You can set breakpoints or do single stepping at the source code level.
To do this, you will need a dedicated serial port on your target, and use a crossover-cable (also known as a null-modem) to connect it to your development host. If you are also using a serial console, this implies you will need two serial ports on your target. It is possible to do both kernel debug and serial console through a single serial port. This will be mentioned later in this chapter.
When you configure the kernel, select the 'Remote GDB kernel debugging' which is listed under "Kernel hacking". Do a 'make clean' and recompile the kernel so that debugging symbols are compiled into your kernel image. Try to make a new image. You will soon discover two missing symbols in the final linking stage:
arch/mips/kernel/kernel.o: In function `getpacket': arch/mips/kernel/kernel.o(.text+0x85ac): undefined reference to `getDebugChar' arch/mips/kernel/kernel.o(.text+0x85cc): undefined reference to `getDebugChar' arch/mips/kernel/kernel.o(.text+0x8634): undefined reference to `getDebugChar' arch/mips/kernel/kernel.o(.text+0x864c): undefined reference to `getDebugChar' arch/mips/kernel/kernel.o(.text+0x8670): undefined reference to `putDebugChar' arch/mips/kernel/kernel.o(.text+0x8680): undefined reference to `putDebugChar' arch/mips/kernel/kernel.o(.text+0x8698): undefined reference to `putDebugChar' arch/mips/kernel/kernel.o(.text+0x86a0): undefined reference to `putDebugChar'
You need to supply these two functions for your own boards:
int putDebugChar(uint8 byte) uint8 getDebugChar(void)
As an example, here is the dbg_io.c for DDB5476 board. DDB5476 uses a standard UART serial port to implement those two functions.
After supplying those two functions, you are ready to debug the kernel. Run the new kernel image. If you also use the early printk patch, you should be able to see something like this on your console:
Wait for gdb client connection ...
Assuming you have already connected a cross-over serial cable between the dedicated serial port on the target and a serial port on your host (say, COM0), you can then set the appropriate baud rate and start the cross gdb on your host:
stty -F /dev/ttyS0 38400 mipsel-linux-gdb vmlinux
At the gdb prompt, type
target remote /dev/ttyS0
And, voila! You should be talking to the kernel through KGDB - if you are lucky enough!
A couple of tips on using KGDB:
- Any functions preceded with the '__init' label will not break very well with breakpoints. Sometimes it will screw up the line numbers of other functions in the same file. Try undefining __init to be an empty macro in your 'include/linux/init.h' file. Refer to the patch. [NOTE: This problem is fixed in the latest gdb version, at least in gdb 5.2]
- Sometimes if you break on a function, you cannot see the correct value in variables and cannot do back-tracing. This is probably because certain registers are still not initialized [HELP: because kernel is compiled with -O2 flag?]. Step into the function a couple of lines, and you should see the variable and back-tracing fine.
What if the board only has one serial port?
Some boards only have one serial port. If you use it as serial console, you cannot really use it for KGDB - unless you do some tricks to it.
There are two solutions. One is GDB console, and the other is to use a KGDB demuxing script.
It is easy to use the GDB console. When you select 'Remote GDB kernel debugging' under the 'Kernel hacking' sub-menu, you are also prompted for 'Console output to GDB'. Simply selecting that choice will work! In fact, this option is so easy to use you might want to use it even if you have a second serial port.
However, this option has a limit. When the kernel goes to userland, the console stops working. This is because the KGDB stub in the kernel and GDB are not designed to provide interactive output. [HELP: any volunteers?]
The second option uses a script called 'kgdb_demux' written by Brian Moyle. It creates two virtual ports, typically ttya0 and ttya1. It then listens to the real serial port (such as ttyS0). It will forward console traffic to ttya0 and KGDB traffic to ttya1. All you have to do then is to start minicom on /dev/ttya0 (port setting does not matter) and KGDB on /dev/ttya1.
You can download the tarball here. A couple of usage tips.
- Untar the file to some place.
- Copy kgdb_demux script to your execution path, and modify it properly.
- Set the port parameters properly before you start kgdb_demux.
Next page: CPU Support