KVM
KVM is one of several virtualization technologies available for Linux. KVM generally requires hardware support but can run an unmodified OS image but in exchange offers full system virtualization, good performance. Also since KVM is based on a standard Linux kernel the host system uses the standard I/O drivers available for the platform.
Trap and Emulate
KymaSys implemented MIPS support in KVM for MIPS32r2 processors using Trap and Emulate, which was merged in Linux 3.10. The guest runs entirely in user mode which means in confined to the lowest 2GB of the address space. As the result the address space for guest applications is shrunk so guest kernels now use a 1GB/1GB userspace/kernelspace split. Privileged instructions trap to the host kernel which emulates the instruction.
Kyma claims “We've limited the changes to the Linux guest to less than 10 lines of code and use advanced techniques like run-time binary translation to minimize the number of traps and greatly improve performance.”[1][2] for their software-only implementation. If their technology holds up to its promise they should deliver performance below but fairly close to that of a hardware-based KVM implementation.
Guest segments
The Trap & Emulate version of KVM will expose a non-traditional set of segments to the guest:
| Name | Privilege Level | Attributes | Address Range | Size | Comments | ||||
|---|---|---|---|---|---|---|---|---|---|
| KSeg3 | Kernel | TLB | 0xE0000000 .. 0xFFFFFFFF | 512MB | Linux: vmalloc / modules | ||||
| KSeg2 | Supervisor | TLB | 0xC0000000 .. 0xDFFFFFFF | 512MB | |||||
| KSeg1 | Kernel | Unmapped, Uncached | 0xA0000000 .. 0xBFFFFFFF | 512MB | Linux: MMIO | ||||
| KSeg0 | Unmapped, Cached | 0x80000000 .. 0x9FFFFFFF | 512MB | Linux: Static code / data | |||||
| USeg | Guest KSeg2 | User | Guest
Kernel |
TLB | Guest TLB | 0x60000000 .. 0x7FFFFFFF | 2GB | 512MB | Linux Guest: vmalloc / modules |
| Guest KSeg0 | Guest Unmapped, Cached | 0x40000000 .. 0x5FFFFFFF | 512MB | Linux Guest: static code / data | |||||
| Guest USeg | Guest
User |
Guest TLB | 0x00000000 .. 0x3FFFFFFF | 1GB | |||||
A Trap & Emulate guest kernel must be built with CONFIG_KVM_GUEST=y, so that it is linked at 0x40000000 (Guest KSeg0) instead of the usual 0x80000000 (KSeg0).
VZ (Hardware Assisted Virtualization)
MIPS Release 5 has the VZ module to support hardware assisted virtualization. Multiple KVM implementations utilising VZ have been proposed. An implementation based mainly on KymaSys' version was merged in Linux 4.12.
Cavium
David Daney posted a patchset to add KVM/VZ support on 7th June 2013[3].
It modifies the existing exception handlers to handle guest mode, and is largely focussed on MIPS64 so far. David Daney has also ported the kvm tool to MIPS as the controlling program, so this implementation makes use of paravirtualization.
KymaSys
Sanjay Lal posted a patchset to add KVM/VZ support on 19th May 2013[4].
It creates a new exception vector (using the EBASE COP0 register) for use when running guests, and is largely focussed on MIPS32 so far.
KymaSys have also adding MIPS KVM support to QEMU[5], which is used to emulate the Malta platform and was merged in QEMU v2.1[6].
Notable Features by Kernel Version
- Linux 3.10
- Initial Trap & Emulate support for MIPS32r2 processors.
- Linux 3.14
- Support TLBInv capable cores with EntryHi.EHInv.
- Linux 3.16
- Rewrite guest timer emulation with new user API, allowing precise control of guest timer.
- Support XBurst cores and <16K pages.
- Expose CP0_EPC, CP0_Count, CP0_Compare, CP0_UserLocal, CP0_HWREna registers to user API.
- Linux 4.1
- Expose FPU & MSA to guest.
- Expose CP0_PRid, FPU & MSA registers and writing of CP0_Config1, CP0_Config2, CP0_Config3, CP0_Config4, CP0_Config5 registers to user API.
- Handle host MSA Disabled & Trap exceptions.
- Linux 4.7
- MIPS groundwork for probing and accessing VZ hardware resources.
- Linux 4.8
- Support MIPS32r6 host & guest.
- Support MIPS64 hosts (with MIPS32 guest).
- Make KVM_GET_REG_LIST dynamic, and list FPU & MSA registers.
- Make Config3.ULRI writable by user API (to control exposure of CP0_UserLocal register).
- Emulate CP0_KScratch registers and expose to user.
- Move commpage away from 0 if possible to allow guest kernel NULL pointer dereferences to be handled by guest.
- Convert guest exception entry code to use UASM (dynamically generated).
- Add trace events: kvm_aux (FPU & MSA context management), kvm_asid_change, kvm_enter, kvm_reenter, kvm_hwr (guest register emulation).
- Some support for Highmem host kernels.
- Linux 4.9
- Some fixes in preparation for supporting MIPS EVA host kernels
- Linux 4.11
- Add GVA->HPA page tables for T&E, to cache GVA mappings commit commit commit commit.
- Generate fast-path TLB refill exception handler which loads host TLB entries from GVA page table, avoiding repeated guest memory translation and guest TLB lookups commit.
- Use uaccess macros when T&E needs to access guest memory, which with GVA page tables and the Linux TLB refill handler improves robustness against TLB faults and fixes EVA hosts commit.
- Use BadInstr/BadInstrP registers when available to obtain instruction encodings after a synchronous trap commit.
- Add GPA->HPA page tables to replace the inflexible linear array, allowing for multiple sparsely arranged memory regions commit.
- Properly implement dirty page logging commit commit commit.
- Add KVM_CAP_SYNC_MMU support so that changes in GPA mappings become effective in guests even if they are already running, allowing for copy-on-write, KSM, idle page tracking, swapping, and guest memory ballooning commit.
- Add KVM_CAP_READONLY_MEM support, so writes to specified memory regions are treated as MMIO commit commit commit.
- Implement proper CP0_EBase support in T&E commit.
- Expose a few more missing CP0 registers to userland commit commit.
- Add KVM_CAP_NR_VCPUS and KVM_CAP_MAX_VCPUS support, and allow up to 8 VCPUs to be created in a VM commit.
- Linux 4.12
- Add basic support for the MIPS Virtualization Module (generally known as MIPS VZ), supporting the ImgTec P5600, P6600, I6400, and Cavium Octeon III commit commit cores so far commit commit commit commit commit.
- Support is included for the following VZ / guest hardware features:
- MIPS32 and MIPS64 commit commit, r5 (VZ requires r5 or later) and r6.
- TLBs with GuestID (IMG cores) or Root ASID Dealias (Octeon III) commit.
- Shared physical root/guest TLB (IMG cores) commit.
- FPU / MSA commit.
- Cop0 timer (up to 1GHz for now due to soft timer limit) commit, optionally emulated commit.
- Segmentation control (EVA) commit.
- Hardware page table walker (HTW) both for root and guest TLB commit.
- Tracing of guest mode changes (using GuestCtl0.MC bit) commit.
- MAAR emulation commit.
- Properly implement the TLBR instruction for T&E commit.
Future Work
This section details some future improvements that could be made (or have yet to be fully upstreamed) to KVM for MIPS:
- Memory management
- Enable HTW to use GVA->HPA page table.
- Emulate guest FTLB to speed up guest TLB lookups (set associative) and allow much larger guest TLB. (done, but needs cleaning up)
- Further Trap & Emulate optimisations.
See Also
External Links
References
- ↑ http://kymasys.com/wordpress/2012/08/
- ↑ http://www.virtual-strategy.com/2012/08/07/kyma-systems-announces-open%E2%80%93source-kvm-hypervisor-mips32%C2%AE-processors
- ↑ http://www.linux-mips.org/archives/linux-mips/2013-06/threads.html#00132
- ↑ http://www.linux-mips.org/archives/linux-mips/2013-05/threads.html#00144
- ↑ http://lists.gnu.org/archive/html/qemu-devel/2013-03/threads.html#00200
- ↑ https://lists.gnu.org/archive/html/qemu-devel/2014-06/msg04074.html
