linux关闭透明大页

DESCRIPTION

Starting with RedHat6, OL6, SLES11, and UEK2 kernels, Transparent HugePages are implemented and enabled (default) in an attempt to improve the memory management.  Transparent HugePages are similar to the HugePages that have been available in previous Linux releases.  The main difference is that the Transparent HugePages are set up dynamically at run time by the khugepaged thread in kernel while the regular HugePages had to be preallocated at the boot up time.
 

Because Transparent HugePages are known to cause unexpected node reboots and performance problems with RAC, Oracle strongly advises to disable the use of Transparent HugePages. In addition, Transparent Hugepages may cause problems even in a single-instance database environment with unexpected performance problems or delays. As such, Oracle recommends disabling Transparent HugePages on all Database servers running Oracle.

Note:  This ONLY applies to the new feature Transparent HugePages, Oracle highly recommends the use of standard HugePages that were recommended for previous releases of Linux.  See Document 361323.1 for additional information on HugePages.

Note: on UEK2 and above, check the existence of the /sys/kernel/mm/transparent_hugepage/ directory.  If this directory does not exist, then the transparent hugepage is removed from the kernel, so there is no need to disable the transparent hugepage

OCCURRENCE

 Starting with SLES11, RHEL6, OL6, and UEK2 Kernels, Transparent HugePages are introduced and enabled by default, and this can cause node reboots and performance problems.

On UEK2 and above, the transparent hugepages is already removed if /sys/kernel/mm/transparent_hugepage/ directory directory does not exist.  In that case, there is no further need to disable the transparent hugepages.

SYMPTOMS

Node reboots.

The ocssd.log may show some of the threads are blocked (but this does not show all the time):

2013-05-01 14:30:45.255: [    CSSD][224204544]clssscMonitorThreads clssnmvKillBlockThread not scheduled for 7500 msecs
2013-05-01 14:30:46.945: [    CSSD][224204544]clssscMonitorThreads clssnmvWorkerThread not scheduled for 8030 msecs

WORKAROUND

The information provided here for Redhat and SLES needs to be confirmed by Redhat and SLES support because Redhat and SLES support may provide a different ways to disable the transparent hugepages.
Contact Redhat and SLES support on how to disable the transparent hugepages on their respective platform.

The workaround provided here is for Redhat 6 and lower, Oracle Linux 6 and lower, and SLES 11 and lower
 

For Oracle Linux 7, Oracle published MOS document 2066217.1
Oracle Linux 7 - How to disable Transparent HugePages for RHCK kernel?

Also, add the following line to /etc/default/grub file:
transparent_hugepage=never

For Redhat 7, Redhat published the following note: how to disable transparent hugepages in Redhat 7.
How to disable transparent hugepages (THP) on Red Hat Enterprise Linux ? - Red Hat Customer Portal

For SLES 12, conatct SLES support for the instruction on how to disable the transparent hugepages

The following instruction are for Redhat 6 or lower, Oracle Linux 6 or lower, and SLES 11 or lower:

To check if the Transparent HugePages are enabled in your server execute the following:

Default/Enabled setting is  [always]:

# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] never

Note 1: For RHEL kernel, the file path is different from above:

# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] never

  
Please modify this file accordingly.

Note 2: For UEK2 kernel, as of 2.6.39-400.116.0 Transparent Hugepages has been removed from the kernel.  If it is not compiled into the kernel then /sys/kernel/mm/transparent_hugepage will not exist.

 Disabled setting is [never]:

# cat /sys/kernel/mm/transparent_hugepage/enabled
always [never]

If "enabled" is NOT set to "[never]", the Transparent HugePages are being used.

You can also issue:

# grep AnonHugePages /proc/meminfo

If the output contains a line like "AnonHugepages: xxxx kB", with a value > 0kB, the kernel is using Transparent HugePages.

Because the kernel currently uses Transparent HugePages only for the anonymous memory blocks like stack and heap, the value of AnonHugepages in /proc/meminfo is the current amount of Transparent HugePages that the kernel is using.

To disable Transparent HugePages boot time either one of the following 2 methods may be used:

Add the following to the kernel boot line in /etc/grub.conf (a symlink to /boot/grub/grub.conf) and reboot the server (this is the preferred method):

transparent_hugepage=never

Once modified the line will read similar to the following example:

title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/ transparent_hugepage=never
        initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img
 

OR

Add the following lines in /etc/rc.local and reboot the server (this still can be done on Redhat 7 although rc.local is being deprecated):

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

 Please change the file path for RHEL kernel to /sys/kernel/mm/redhat_transparent_hugepage/ accordingly. 

===========================================================================

The following comment was received about SLES11 SP3.  Please note that this is not tested, so this is provided only as a talking point to your sys admin who knows SLES11 SP3 and who should also contact SLES for correct information.

in SLES11 SP3 with YAST/GRUB boot loader /etc/grub.conf overrided usind YAST. parameter transparent_hugepage=never must be set in YAST-bootloader- Edit settings - in line: optional kernel parameter [before showopts].

GOAL

The purpose of this document is to detail the steps to disable Transparent HugePages on OL7 running RHCK kernel using tuned service. 

SOLUTION

To keep THP disabled across a reboot, Please perform any one of the following action plans

Use a systemd(1) service to disable transparent hugepages

1. Create the systemd(1) service definition file /etc/systemd/system/disable-thp.service similar to this:  [Unit]
   Description=Disable Transparent Huge Pages (THP)

   [Service]
   Type=simple
   ExecStart=/bin/sh -c "echo 'never' >/sys/kernel/mm/transparent_hugepage/enabled && echo 'never' >/sys/kernel/mm/transparent_hugepage/defrag"

   [Install]
   WantedBy=multi-user.target

2. Enable the new service to run automatically:  

   # systemctl daemon-reload
   # systemctl start disable-thp
   # systemctl enable disable-thp
   # systemctl status disable-thp

Disable transparent hugepages via the Linux kernel command line

1. Add the clause "transparent_hugepage=never" to the GRUB_CMDLINE_LINUX entry of /etc/default/grub, like this:  

   # /etc/default/grub
   GRUB_TIMEOUT=5
   GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
   GRUB_DEFAULT=saved
   GRUB_DISABLE_SUBMENU=true
   GRUB_TERMINAL_OUTPUT="console"
   GRUB_CMDLINE_LINUX="elevator=deadline audit=1 crashkernel=auto rd.lvm.lv=rootvg/rootlv rd.lvm.lv=rootvg/swaplv rd.lvm.lv=rootvg/usrlv transparent_hugepage=never"
   GRUB_DISABLE_RECOVERY="true"

2. Backup the GRUB configuration file and generate a new file to incorporate the change:  

   # cp -pv /boot/grub2/grub.cfg /boot/grub2/grub.cfg-bkp
   # grub2-mkconfig -o /boot/grub2/grub.cfg

    

   For UEFI based machine run the below command for grub configuration change.

   # cp -pv /boot/efi/EFI/redhat/grub.cfg /boot/efi/EFI/redhat/grub.cfg-bkp
   # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

3. Reboot the server to activate the change.  

   # reboot

4. Verify the change is persistent:  

   # cat /sys/kernel/mm/transparent_hugepage/enabled
   always madvise [never]

NOTE: echo never > /sys/kernel/mm/transparent_hugepage/enabled will only help for new THP creations but the one which got wrongly allocated to Oracle DB will still cause issues hence need to have a planned reboot.

 Disable Transparent Hugepages via tuned service

"tuned.service" on OL7 set the transparent_hugepage to always by default, even if it is disabled in grub kernel command line as above.

# grep transparent_hugepage /boot/grub2/grub.cfg
linux16 /vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/vgsystem-root ro  rd.lvm.lv=vgsystem/swap rd.lvm.lv=vgsystem/root rhgb quiet numa=off transparent_hugepage=never                                              
 

Verify the THP (Transparent hugepage) mode

# uname -r
3.10.0-229.el7.x86_64

# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never                                 ← THP is enabled 

Disable it globally on throughput-performance

1. Take a backup of "/usr/lib/tuned/throughput-performance/tuned.conf", then change "transparent_hugepages=always" to "transparent_hugepages=never". 

   #cp /usr/lib/tuned/throughput-performance/tuned.conf /usr/lib/tuned/throughput-performance/tuned.conf.bkp_original
   #vi /usr/lib/tuned/throughput-performance/tuned.conf
    [vm]
    transparent_hugepages=always
      to 
    [vm]
    transparent_hugepages=never 

2.  Reboot the server and verify the outcome.   

   #cat /sys/kernel/mm/transparent_hugepage/enabled
    always madvise [never]                          ← THP is disabled

Disable it on active tuned profile

1.  First Identify which profile is active.     

   # tuned-adm active
   Current active profile: virtual-guest            ← Virtual-guest is the active profile

2. Now edit "/usr/lib/tuned/virtual-guest/tuned.conf" file and append "transparent_hugepages=never" in vm section to disable THP.  

   #cp /usr/lib/tuned/virtual-guest/tuned.conf /usr/lib/tuned/virtual-guest/tuned.conf.bkp_original
   #vi /usr/lib/tuned/virtual-guest/tuned.conf
   [main]
   include=throughput-performance
   [vm]
   transparent_hugepages=never                       

3. Reboot the server and verify the outcome.

   #cat /sys/kernel/mm/transparent_hugepage/enabled
   always madvise [never]                            ← THP Disabled  

Transparent Huge Pages cannot be enabled or disabled on a running machine and requires a reboot.

 THP status in other UEK kernel versions

• UEK3 disabled by default .

  [root@ol6 ~]# grep -i CONFIG_TRANSPARENT_HUGEPAGE /boot/config-3.8.13-118.6.2.el6uek.x86_64
  # CONFIG_TRANSPARENT_HUGEPAGE is not set  

• UEK4 enabled by default.

  [root@ol7 ~]# grep -i CONFIG_TRANSPARENT_HUGEPAGE /boot/config-4.1.12-37.2.2.el7uek.x86_64
  CONFIG_TRANSPARENT_HUGEPAGE=y
  CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y