Highmem is a method that increases the usable address space. Originally developed for Intel i386 where it enables the use of more than approx. 970MB of memory. Most of the code is portable to other architectures.
Memory Managment on Linux/MIPS
32-bit Linux/MIPS kernels assume the entire low memory to be accessible through KSEG0 which limits lowmem to at most 512MB. Typical MIPS systems reserve part of that address space for other uses so a lowmem limit of 256MB is a common value. Memory beyond that can be used by 64-bit kernels or 32-bit kernels with highmem support enabled.
Linux allocates a large array named mem_map in lowmem. On 32-bit kernels it contains one 32-byte (i.e. sizeof (struct page)) entry per page of memory. On Linux 2.4 struct page used to be twice as large. Pages are allocated even for reserved memory ranges. This consumption of lowmem means memory pressure on lowmem will rise with the amount of highmem. In practice at most a ratio of 1:4 to 1:8 has been found to perform well.
Highmem is accessed through temporary mappings. Creation and teardown of such mappings are not for free and mappings are only permitted to exist for a short time. This means any kernel software using highmem needs to be written with special care or performance and stability of the entire system are at stake.
Certain systems where memory may be outside the low 4GB physical address space will need to enable support for 64-bit physical addresses in pagetables also. This will nearly double the size of pagetables. Again pagetables reside in lowmem; this can be a large consumer of lowmem for system that make heavy use of mmap.
While creation and teardown of mappings already are a significant overhead, the cacheflushes necessary to deal with cache aliases would dwarf that. Therefore Linux/MIPS supports highmem only on systems that don't suffer from cache aliases. A highmem kernel will boot on such a system but only use the lowmem.
64-bit kernels use XKPHYS to access their lowmem. Due to the size of XKPHYS the entire physical memory is directly accessible so highmem isn't needed with 64-bit kernels. 64-bit kernels have somewhat larger code and data resulting in a slight slowdown. This however is by far preferable over the impact of highmem.
Jaguar ATX for some creative abuse of the kernel's highmem facilities.