From LinuxMIPS
Jump to navigationJump to search

KVM is one of several virtualization technologies available for Linux. KVM generally requires hardware support but can run an unmodified OS image but in exchange offers full system virtualization, good performance. Also since KVM is based on a standard Linux kernel the host system uses the standard I/O drivers available for the platform.

Trap and Emulate

KymaSys implemented MIPS support in KVM for MIPS32r2 processors using Trap and Emulate, which was merged in Linux 3.10. The guest runs entirely in user mode which means in confined to the lowest 2GB of the address space. As the result the address space for guest applications is shrunk so guest kernels now use a 1GB/1GB userspace/kernelspace split. Privileged instructions trap to the host kernel which emulates the instruction.

Kyma claims “We've limited the changes to the Linux guest to less than 10 lines of code and use advanced techniques like run-time binary translation to minimize the number of traps and greatly improve performance.”[1][2] for their software-only implementation. If their technology holds up to its promise they should deliver performance below but fairly close to that of a hardware-based KVM implementation.

Guest segments

The Trap & Emulate version of KVM will expose a non-traditional set of segments to the guest:

Trap & Emulate Guest Segment Layout
Name Privilege Level Attributes Address Range Size Comments
KSeg3 Kernel TLB 0xE0000000 .. 0xFFFFFFFF 512MB Linux: vmalloc / modules
KSeg2 Supervisor TLB 0xC0000000 .. 0xDFFFFFFF 512MB
KSeg1 Kernel Unmapped, Uncached 0xA0000000 .. 0xBFFFFFFF 512MB Linux: MMIO
KSeg0 Unmapped, Cached 0x80000000 .. 0x9FFFFFFF 512MB Linux: Static code / data
USeg Guest KSeg2 User Guest


TLB Guest TLB 0x60000000 .. 0x7FFFFFFF 2GB 512MB Linux Guest: vmalloc / modules
Guest KSeg0 Guest Unmapped, Cached 0x40000000 .. 0x5FFFFFFF 512MB Linux Guest: static code / data
Guest USeg Guest


Guest TLB 0x00000000 .. 0x3FFFFFFF 1GB

A Trap & Emulate guest kernel must be built with CONFIG_KVM_GUEST=y, so that it is linked at 0x40000000 (Guest KSeg0) instead of the usual 0x80000000 (KSeg0).

VZ (Hardware Assisted Virtualization)

MIPS Release 5 has the VZ module to support hardware assisted virtualization. Multiple KVM implementations utilising VZ have been proposed. An implementation based mainly on KymaSys' version was merged in Linux 4.12.


David Daney posted a patchset to add KVM/VZ support on 7th June 2013[3].

It modifies the existing exception handlers to handle guest mode, and is largely focussed on MIPS64 so far. David Daney has also ported the kvm tool to MIPS as the controlling program, so this implementation makes use of paravirtualization.


Sanjay Lal posted a patchset to add KVM/VZ support on 19th May 2013[4].

It creates a new exception vector (using the EBASE COP0 register) for use when running guests, and is largely focussed on MIPS32 so far.

KymaSys have also adding MIPS KVM support to QEMU[5], which is used to emulate the Malta platform and was merged in QEMU v2.1[6].

Notable Features by Kernel Version

Linux 3.10
Initial Trap & Emulate support for MIPS32r2 processors.
Linux 3.14
Support TLBInv capable cores with EntryHi.EHInv.
Linux 3.16
Rewrite guest timer emulation with new user API, allowing precise control of guest timer.
Support XBurst cores and <16K pages.
Expose CP0_EPC, CP0_Count, CP0_Compare, CP0_UserLocal, CP0_HWREna registers to user API.
Linux 4.1
Expose FPU & MSA to guest.
Expose CP0_PRid, FPU & MSA registers and writing of CP0_Config1, CP0_Config2, CP0_Config3, CP0_Config4, CP0_Config5 registers to user API.
Handle host MSA Disabled & Trap exceptions.
Linux 4.7
MIPS groundwork for probing and accessing VZ hardware resources.
Linux 4.8
Support MIPS32r6 host & guest.
Support MIPS64 hosts (with MIPS32 guest).
Make KVM_GET_REG_LIST dynamic, and list FPU & MSA registers.
Make Config3.ULRI writable by user API (to control exposure of CP0_UserLocal register).
Emulate CP0_KScratch registers and expose to user.
Move commpage away from 0 if possible to allow guest kernel NULL pointer dereferences to be handled by guest.
Convert guest exception entry code to use UASM (dynamically generated).
Add trace events: kvm_aux (FPU & MSA context management), kvm_asid_change, kvm_enter, kvm_reenter, kvm_hwr (guest register emulation).
Some support for Highmem host kernels.
Linux 4.9
Some fixes in preparation for supporting MIPS EVA host kernels
Linux 4.11
Add GVA->HPA page tables for T&E, to cache GVA mappings commit commit commit commit.
Generate fast-path TLB refill exception handler which loads host TLB entries from GVA page table, avoiding repeated guest memory translation and guest TLB lookups commit.
Use uaccess macros when T&E needs to access guest memory, which with GVA page tables and the Linux TLB refill handler improves robustness against TLB faults and fixes EVA hosts commit.
Use BadInstr/BadInstrP registers when available to obtain instruction encodings after a synchronous trap commit.
Add GPA->HPA page tables to replace the inflexible linear array, allowing for multiple sparsely arranged memory regions commit.
Properly implement dirty page logging commit commit commit.
Add KVM_CAP_SYNC_MMU support so that changes in GPA mappings become effective in guests even if they are already running, allowing for copy-on-write, KSM, idle page tracking, swapping, and guest memory ballooning commit.
Add KVM_CAP_READONLY_MEM support, so writes to specified memory regions are treated as MMIO commit commit commit.
Implement proper CP0_EBase support in T&E commit.
Expose a few more missing CP0 registers to userland commit commit.
Add KVM_CAP_NR_VCPUS and KVM_CAP_MAX_VCPUS support, and allow up to 8 VCPUs to be created in a VM commit.
Linux 4.12
Add basic support for the MIPS Virtualization Module (generally known as MIPS VZ), supporting the ImgTec P5600, P6600, I6400, and Cavium Octeon III commit commit cores so far commit commit commit commit commit.
Support is included for the following VZ / guest hardware features:
Properly implement the TLBR instruction for T&E commit.

Future Work

This section details some future improvements that could be made (or have yet to be fully upstreamed) to KVM for MIPS:

  • Memory management
    • Enable HTW to use GVA->HPA page table.
    • Emulate guest FTLB to speed up guest TLB lookups (set associative) and allow much larger guest TLB. (done, but needs cleaning up)
  • Further Trap & Emulate optimisations.

See Also

External Links