Difference between revisions of "Page size"

From LinuxMIPS
Jump to: navigation, search
(Accessible address space)
 
(5 intermediate revisions by one user not shown)
Line 1: Line 1:
 
Since a while the Linux/MIPS kernel support page sizes other than 4kB.
 
Since a while the Linux/MIPS kernel support page sizes other than 4kB.
  
* The [[R6000]] processor only supports 16kB page size.
+
* Large compute workloads benefit sometimes dramatically from the increased ''TLB reach''.  Downside: the memory overhead can be significant and at times prohibitive for small systems or workloads that use a large number of small memory mappings.
* The [[R8000]] processor supports 8kB page size as it's smallest pagesize.  It's also the only MIPS processor to support 8kB page size.
+
* Large page size eleminates cache aliases and the overhead and potential for bugs associated with them.
* Large compute workloads benefit sometimes dramatically from the increased ''TLB reach''.  Downside: the memory overhead can be significant and at times prohibitive.
+
* Large pagesize eleminates cache aliases and the overhead and potencial for bugs associated with them.
+
 
* SPECbench numbers have been shown to be boosted significantly by 16kB pages and yet somewhat more by 64kB pages.
 
* SPECbench numbers have been shown to be boosted significantly by 16kB pages and yet somewhat more by 64kB pages.
  
== Accessible address space ==
+
== Hardware restrictions ==
 +
* [[R2000]] (MIPS I) processors only support 4K page size.
 +
* The [[R6000]] processor ''only'' supports 16kB page size.
 +
* The [[R8000]] processor supports 8kB page size as it's smallest page size.
 +
* Every MIPS III, MIPS IV, MIPS32 and MIPS64 processor supports 4kB, 16kB and 64kB page sizes.  Probably a few more larger sizes but these are the sizes supported by the Linux kernel.  Support for the intermediate sizes of 8kB and 32kB is optional in the MIPS architecture and only a few, notably Cavium cnMIPS cores, support them.
 +
* NEC VR41xx processors also support 1kB and 2kB page sizes.  These are very small for a modern operating system such as Linux which is why Linux does not support these page sizes.
 +
 
 +
== Addressable virtual address space ==
 
The size of the usable address space depends on the number of levels of the page tables, page size and the kernel model used.  It always is 4GB for 32-bit kernels.  Note that these 4GB are split into 2GB of userspace and 2GB of kernel addressing space.  This split is hardwired in the MIPS architecture and (with the exception of the R6000) cannot be changed.  The following table gives an overview over the situation on the 64-bit kernel.
 
The size of the usable address space depends on the number of levels of the page tables, page size and the kernel model used.  It always is 4GB for 32-bit kernels.  Note that these 4GB are split into 2GB of userspace and 2GB of kernel addressing space.  This split is hardwired in the MIPS architecture and (with the exception of the R6000) cannot be changed.  The following table gives an overview over the situation on the 64-bit kernel.
  
 +
{|
 +
|
 
{| {{PrettyTable}}
 
{| {{PrettyTable}}
 
! Page Size !! 1 Level        !! 2 Level        !! style="background-color: #ffe387;" | 3 Levels        !! 4 Levels
 
! Page Size !! 1 Level        !! 2 Level        !! style="background-color: #ffe387;" | 3 Levels        !! 4 Levels
Line 19: Line 26:
 
| 16kB      || 25 bits / 32MB  || 36 bits / 64GB  || style="background-color: #ffe387;" | 47 bits / 128TB || 58 bits / 256PB
 
| 16kB      || 25 bits / 32MB  || 36 bits / 64GB  || style="background-color: #ffe387;" | 47 bits / 128TB || 58 bits / 256PB
 
|-
 
|-
| 32kB      || 27 bits / 128MB || 39 bits / 512GB || style="background-color: #ffe387;" | 51 bits / 2PB  || 64 bits / 4EB
+
| 32kB      || 27 bits / 128MB || 39 bits / 512GB || style="background-color: #ffe387;" | 51 bits / 2PB  || style="background-color: #ffbbbb;" | 63 bits / 8EB
 
|-
 
|-
| 64kB      || 29 bits / 512MB || 42 bits / 4TB  || style="background-color: #ffe387;" | 55 bits / 32PB  || 68 bits / 256EB
+
| 64kB      || 29 bits / 512MB || style="background-color: #ffe387;" | 42 bits / 4TB  || 55 bits / 32PB  || style="background-color: #ffbbbb;" | 68 bits / 256EB
 
|}
 
|}
 +
|
 +
{| {{PrettyTable}}
 +
|-
 +
{| {{PrettyTable}} style="background: #ABCDEF;"
 +
| style="background-color: #ffe387; width:20em;" | Used by 64-bit kernels
 +
|}
 +
|-
 +
{| {{PrettyTable}} style="background: #ABCDEF;"
 +
| style="background-color: #ffbbbb; width:20em;" | Nonsenical combination
 +
|}
 +
|}
 +
|}
 +
 +
The combinations marked red are nonsenical because <tt>XUSEG</tt> limits the user address space to 62&nbsp;bits&nbsp;/ 4EB.
  
This table is mostly meant to show how the numbers work out.  In reality there are a few constraints, so it should be taken with a grain of salt:
+
This table is mostly meant to show how the numbers work out if all page table levels are one page in size.
 
* For 3&nbsp;level page tableswith 4kB pages Linux uses two pages for the page directory so the actual virtual address space capacity is 40&nbsp;bits&nbsp;/ 1TB.
 
* For 3&nbsp;level page tableswith 4kB pages Linux uses two pages for the page directory so the actual virtual address space capacity is 40&nbsp;bits&nbsp;/ 1TB.
 
* The largest virtual address space supported by any MIPS processor is 48bits on the old and rather odd R8000 processor.  Typical 64-bit MIPS processors support 36-bit, 40-bit or 44-bit address spaces but other sizes are permitted by the [[MIPS64]] architecture.
 
* The largest virtual address space supported by any MIPS processor is 48bits on the old and rather odd R8000 processor.  Typical 64-bit MIPS processors support 36-bit, 40-bit or 44-bit address spaces but other sizes are permitted by the [[MIPS64]] architecture.
 
* Address spaces of less than 2GB are not likely to be very useful either.
 
* Address spaces of less than 2GB are not likely to be very useful either.
* Only 3-level tables are supported by 64-bit MIPS kernels.
+
* Only 3-level tables are supported by 64-bit MIPS kernels.  The exception are 64kB pages where the resulting 55&nbsp;bits&nbsp; 32PB address space would be so large that cropping it to 42&nbsp;bits&nbsp;/ 4TB appeared more practical for the time being.
* As of 2.6.18 16kB pagesize is supported for 64-bit kernels but experimental for the 32-bit kernel.  64kB is experimental.
+
* 32-bit kernels only support 4kB and 16kB page size
  
 
== Software support ==
 
== Software support ==

Latest revision as of 09:39, 12 October 2012

Since a while the Linux/MIPS kernel support page sizes other than 4kB.

  • Large compute workloads benefit sometimes dramatically from the increased TLB reach. Downside: the memory overhead can be significant and at times prohibitive for small systems or workloads that use a large number of small memory mappings.
  • Large page size eleminates cache aliases and the overhead and potential for bugs associated with them.
  • SPECbench numbers have been shown to be boosted significantly by 16kB pages and yet somewhat more by 64kB pages.

Hardware restrictions

  • R2000 (MIPS I) processors only support 4K page size.
  • The R6000 processor only supports 16kB page size.
  • The R8000 processor supports 8kB page size as it's smallest page size.
  • Every MIPS III, MIPS IV, MIPS32 and MIPS64 processor supports 4kB, 16kB and 64kB page sizes. Probably a few more larger sizes but these are the sizes supported by the Linux kernel. Support for the intermediate sizes of 8kB and 32kB is optional in the MIPS architecture and only a few, notably Cavium cnMIPS cores, support them.
  • NEC VR41xx processors also support 1kB and 2kB page sizes. These are very small for a modern operating system such as Linux which is why Linux does not support these page sizes.

Addressable virtual address space

The size of the usable address space depends on the number of levels of the page tables, page size and the kernel model used. It always is 4GB for 32-bit kernels. Note that these 4GB are split into 2GB of userspace and 2GB of kernel addressing space. This split is hardwired in the MIPS architecture and (with the exception of the R6000) cannot be changed. The following table gives an overview over the situation on the 64-bit kernel.

Page Size 1 Level 2 Level 3 Levels 4 Levels
4kB 21 bits / 2MB 30 bits / 1GB 39 bits / 512GB 48 bits / 256TB
8kB 23 bits / 8MB 33 bits / 8GB 43 bits / 8TB 53 bits / 8PB
16kB 25 bits / 32MB 36 bits / 64GB 47 bits / 128TB 58 bits / 256PB
32kB 27 bits / 128MB 39 bits / 512GB 51 bits / 2PB 63 bits / 8EB
64kB 29 bits / 512MB 42 bits / 4TB 55 bits / 32PB 68 bits / 256EB
Used by 64-bit kernels
Nonsenical combination

The combinations marked red are nonsenical because XUSEG limits the user address space to 62 bits / 4EB.

This table is mostly meant to show how the numbers work out if all page table levels are one page in size.

  • For 3 level page tableswith 4kB pages Linux uses two pages for the page directory so the actual virtual address space capacity is 40 bits / 1TB.
  • The largest virtual address space supported by any MIPS processor is 48bits on the old and rather odd R8000 processor. Typical 64-bit MIPS processors support 36-bit, 40-bit or 44-bit address spaces but other sizes are permitted by the MIPS64 architecture.
  • Address spaces of less than 2GB are not likely to be very useful either.
  • Only 3-level tables are supported by 64-bit MIPS kernels. The exception are 64kB pages where the resulting 55 bits  32PB address space would be so large that cropping it to 42 bits / 4TB appeared more practical for the time being.
  • 32-bit kernels only support 4kB and 16kB page size

Software support

Hugetlbfs

Hugetlbfs is a special magic filesystem. Any file mmapped from hugetlbfs will use huge pages. The problem with hugetlbfs is that it software will need to be modified to use it and memory to be used for huge pages with hugetlbfs needs to be put aside manually.

Cavium's SDK is able to put away with the need to modify software by using an special preload library that wraps malloc. This method works for most but not all applications.

Transparent Huge Pages

Transparent Huge Pages allows normal unmodified software to use huge pages. Other than possibly having to enable THP support there is no administrative activity required. The price for this is a slight overhead.