Hugepages and Large Pages
If you run a Oracle Database on a Linux Server with more than 16 GB physical memory and your System Global Area (SGA) is greater than 8 GB, you should configure HugePages. Oracle promises more performance by doing this. A HugePages configuration means, that the linux kernel can handle „large pages“, like Oracle generally calls them. Instead of standardly 4 KB on x86 and x86_64 or 16 KB on IA64 systems – 4 MB on x86, 2 MB on x86_64 and 256 MB on IA64 system. Bigger pages means, that the system uses less page tables, manages less mappings and by that reduce the effort for their management and access.
However their is a limitation by Oracle, because Automatic Memory Management (AMM) does not support HugePages. If you already use AMM and MEMORY_TARGET is set you have to disable it and switch back to Automatic Shared Memory Management (ASMM). That means set SGA_TARGET and PGA_AGGREGATE_TARGET. But there is another innovation called Transparent Hugpages (THP) which should be disabled as well. The feature will be delivered since Red Hat Linux 6 or a according derivate. Oracle as well as Red Hat recommend to disable Transparent Hugepages. Explanation in point 5 – Change Server configuration. So lets get started and come to the 7 steps:
1. Check Physical Memory
First we should check our „physical“ available Memory. In the example we have about 128 GB of RAM. SGA_TARGET and PGA_AGGREGATE_TARGET together, should not be more than the availabel memory. Besides should be enough space for OS processes itself:
grep MemTotal /proc/meminfo MemTotal: 132151496 kB
2. Check Database Parameter
Second check your database parameter. Initially: AMM disabled? MEMORY_TARGET and MEMORY_MAX_TARGET should be set to 0:
SQL> select value from v$parameter where name = 'memory_target'; VALUE --------------------------- 0
How big is our SGA? In this example about 40 GB. Important: In the following query we directly convert into kB (value/1024). With that we can continue to calculate directly:
SQL> select value/1024 from v$parameter where name = 'sga_target'; VALUE --------------------------- 41943040
Finally as per default the parameter use_large_pages should be enabled:
SQL> select value from v$parameter where name = 'use_large_pages'; VALUE --------------------------- TRUE
3. Check Hugepagesize
In our example we use a x86_64 Red Hat Enterprise Linux Server. So by default hugepagesize should be set to 2 MB:
grep Hugepagesize /proc/meminfo Hugepagesize: 2048 kB
4. Calculate Hugepages
For the calculation of the number of hugepages there is a easy way:
SGA / Hugepagesize = Number Hugepages
Following our example:
41943040 / 2048 = 20480
If you run more than one database on your server, you should include the SGA of all of your instances into the calculation:
( SGA 1. Instance + SGA 2. Instance + … etc. ) / Hugepagesize = Number Hugepages
In My Oracle Support you can find a script (Doc ID 401749.1) called hugepages_settings.sh, which does the calculation. This also includes a check of your kernel version and the actually used shared memory area by the SGA. Please consider that this calculation observes only the actual use of SGA and their use. If your second instance is down it will be not in the account. That means to adjust your SGA and restart your database first. Than you can run the script. Result should be the following line. Maybe you can make your own calculation and than check it with the script:
Recommended setting: vm.nr_hugepages = 20480
5. Change Server Configuration
The next step is to enter the number of hugepages in the server config file. For that you need root permissions. On Red Hat Linux 6 /etc/sysctl.conf.
vi /etc/sysctl.conf vm.nr_hugepages=20480
Correctly inserted, following result should show up:
grep vm.nr_hugepages /etc/sysctl.conf vm.nr_hugepages=20480
The next parameter is hard and soft memlock in /etc/security/limits.conf for our oracle user. This value should be smaller than our available memory but minor to our SGA. Our hugepages should fit into that by 100 percent. For that following calculation:
Number Hugepages * Hugepagesize = minimum Memlock
Following our example:
20480 * 2048 = 41943040
vi /etc/security/limits.conf oracle soft memlock 41943040 oracle hard memlock 41943040
Correctly inserted, following result should show up:
grep oracle /etc/security/limits.conf ... oracle soft memlock 41943040 oracle hard memlock 41943040
As mentioned before we have to disable transparent hugepages from Red Hat Linux version 6 ongoing:
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled [always] madvise never echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag cat /sys/kernel/mm/redhat_transparent_hugepage/enabled always madvise [never]
6. Server Reboot
If all parameter are set, make a complete reboot your server. As an alternative you can reload the parameters with sysctl -p.
7. Check Configuration
Memlock correct?
ulimit -l 41943040
HugePages correctly configured and in use?
grep Huge /proc/meminfo AnonHugePages: 538624 kB HugePages_Total: 20480 HugePages_Free: 12292 HugePages_Rsvd: 8188 HugePages_Surp: 0 Hugepagesize: 2048 kB
Transparent Hugepages disabled?
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled always madvise [never]
Did the database uses HugePages? For that we take a look into the alert log. After „Starting ORACLE instance (normal)“ following entry „Large Pages Information“ gives us advise:
************************ Large Pages Information ******************* Per process system memlock (soft) limit = 100 GB Total Shared Global Region in Large Pages = 40 GB (100%) Large Pages used by this instance: 20481 (40 GB) Large Pages unused system wide = 0 (0 KB) Large Pages configured system wide = 20481 (40 GB) Large Page size = 2048 KB ********************************************************************
If your configuration is incorrect Oracle delivers recommendation here for the right setting. In the following example exactly one Page is missing, so 2048 kB memlock to come to 100% of SGA use of hugepages:
************************ Large Pages Information ******************* ... ... RECOMMENDATION: Total System Global Area size is 40 GB. For optimal performance, prior to the next instance restart: 1. Increase the number of unused large pages by at least 1 (page size 2048 KB, total size 2048 KB) system wide to get 100% of the System Global Area allocated with large pages 2. Large pages are automatically locked into physical memory. Increase the per process memlock (soft) limit to at least 40 GB to lock 100% System Global Area's large pages into physical memory ********************************************************************
Done!
I have an issue of the Huge page.
We have oracle12c AWS RDS on production database and db size around 10gb only.
We are currently facing the high swapping issue with high cpu utilization with below error in alert log.
ORA-00700: soft internal error, arguments: [kskvmstatact: excessive swapping observed], [], [], [], [], [], [], [], [], [], [], []
However when we restart DB at that time we got below recommendations in alert log.
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
Per process system memlock (soft) limit = 64K
Expected per process system memlock (soft) limit to lock
SHARED GLOBAL AREA (SGA) into memory: 12G
Available system pagesizes:
4K, 2048K
Supported system pagesize(s):
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
4K Configured 3047431 3047431 NONE
Reason for not supporting certain system pagesizes:
2048K – Dynamic allocate and free memory regions
RECOMMENDATION:
1. Increase per process memlock (soft) limit to at least 12GB
to lock 100% of SHARED GLOBAL AREA (SGA) pages into physical memory.
We have AMM is auto.
Please help to resolve high cpu utilization.
Hi Vaibhav,
my first intention is to try switching from AMM to ASMM and setting sga_max_size and pga_aggregate_target. Oracle disables AMM for Instances higher 4GB memory with 12.2 – so please try to re-configure that first.
Hi Winkler,
Thanks for prompt response.
We have 16gb memory allocated with 2 cpu and 2 core configuration.
select value from v$parameter where name = ‘memory_target’; and its o/p is 12482248704 means we have configured AMM.
As per your recommendations we have to disable AMM and enable ASMM where above 4GB memory AMM is not working properly and we have to convert it into ASMM correct!.
Correct. Just a first suggestion.
Regards,
Sebastian
Hi what if im using Oracle Linux Enterprise 6.5?
Do i still need to to this?
As mentioned before we have to disable transparent hugepages from Red Hat Linux version 6 ongoing:
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] madvise never
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always madvise [never]
Your article says: Hugepagesize: 2048 kB and formula for calculating as follows: SGA / Hugepagesize = Number Hugepages
Your initial calculation (41943040 / 2048 = 20480) is wrong in using the 2048 value… this is 2048 kB or 2097152 bytes (2048 kB = 2097152)
Your example system needs only 20 hugepages: 41943040 / 2097152 = 20
Here is the information on two machine that we have configured and running flawlessly:
First system with 12 GB RAM: 4731172824/2097152=2255
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
2048K 2300 2258 2258 NONE
Second system with 16 GB RAM: 10737000000/2097152=5120 PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
2048K 5200 5122 5122 NONE
The article was well written, but the values used in your examples is flawed. Also, why are your fonts for light? Very hard to read.
Thanks for the feedback: I will check the fonts and the figures of course
Your numbers are correct, Johannes.
For 40GB SGA….
SQL> select value/1024 from V$parameter where name=’sga_target’;
VALUE
————-
41943040 <—– This is in KB
When you divide 41943040 kb / 2048kb, you get 20480!
Your numbers are correct!
MGA Issue – and it is fixed with Oracle 19.8.0 and newer
Bug 30851951 FILE-MAX Limit To Exceed And Crashes Instance Caused By Open FDS To /dev/shm/*ksipc_mga*.dat
This above bug fix in 19.8.0 version of database
I’ve noticed you calculate the hard and soft limits for memlock to be the minimum required to hold the entire SGA in memory. In the oracle 19c docs though, they say to set these to at lease 90% of system RAM. Thoughts?
“at least 90 percent of the current RAM when HugePages memory is enabled and at least 3145728 KB (3 GB) when HugePages memory is disabled”
https://docs.oracle.com/en/database/oracle/oracle-database/19/cwsol/checking-resource-limits-for-oracle-software-installation-user.html#GUID-293874BD-8069-470F-BEBF-A77C06618D5A
Hi Jim,
actually the hard and soft limits do not allocate additional resources. Due to that the upper limit could even be higher than the available amount of memory.
Please mention the values e.g. sga_target=40g ,sga_max_size=40g ,pga_aggregate_target=16g,pga_aggregate_limit=40g values in this case with 128g RAM with 40 g SGA.
is this value ok?
If you only run one database on that server you could increase the sga_target to arround 80 GB (and hugepages of course). But 40 GB was sufficient for the application running on that database.
very well described. Thanks for sharing!
Hi , Do you have the reference for requirement of 8 Gb SGA for implementing huge pages ?
I’m not 100% sure because it’s 8 years ago where my former colleague wrote that blog but I found a hint that redhat recommended to not use huge pages below a total RAM of 16 GB. With the assumption of having an SGA with 50% of total memory you end up with 8 GB.
Here’s a link to a talk in the Oracle community about that: https://community.oracle.com/tech/apps-infra/discussion/2614309/hugepages-with-smaller-ram