Performance tuning: Huge Pages in Linux. By Riyaj Shamsudeen

Please download to get full document.

View again

of 16
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information Report
Category:

Self-Help

Published:

Views: 0 | Pages: 16

Extension: PDF | Download: 0

Share
Related documents
Description
Performance tuning: Huge Pages in Linux By Riyaj Shamsudeen Who am I? 16 years using Oracle products Over 15 years as Oracle DBA Certified DBA versions 7.0,7.3,8,8i &9i Specializes in performance tuning,
Transcript
Performance tuning: Huge Pages in Linux By Riyaj Shamsudeen Who am I? 16 years using Oracle products Over 15 years as Oracle DBA Certified DBA versions 7.0,7.3,8,8i &9i Specializes in performance tuning, Internals and E-business suite Currently working for The Pythian Group OakTable member rshamsud at gmail.com Blog : orainternals.wordpress.com 3 Disclaimer These slides and materials represent the work and opinions of the author and do not constitute official positions of my current or past employer or any other organization. This material has been peer reviewed, but author assume no responsibility whatsoever for the test cases. If you corrupt your databases by running my scripts, you are solely responsible for that. This material should not should not be reproduced or used without the authors' written permission. 4 Problem description Database was intermittently freezing and business was severely affected. This is a central database and client had already worked with vendor support and problem was still unresolved. Symptom: High Kernel mode CPU usage intermittently in a 4 CPU dual core, hyper threading enabled Linux server. CPU usage 07:20:01 AM CPU %user %nice %system %iowait %idle 07:30:01 AM all :40:01 AM all :50:01 AM all :00:01 AM all :10:01 AM all :20:01 AM all :32:50 AM all :40:02 AM all :50:01 AM all :00:01 AM all :10:01 AM all :20:01 AM all :30:02 AM all :40:01 AM all :50:01 AM all :00:01 AM all :10:01 AM all :20:01 AM all :30:01 AM all :41:54 AM all :50:01 AM all :00:01 AM all CPU usage CPU usage spikes in kernel mode intermittently %SYS CPU usage :40 07:50 08:00 08:10 08:20 08:32 08:40 08:50 09:00 09:10 09:20 09:30 09:40 09:50 10:00 10:10 10:20 Tools? Client was using few tools, but none of the tools were helpful in this case. Well problem is that those tools are averaging out over a longer period of time and doesn't show any issues. We decided to look at all statistics at 8:40AM and 10:10AM Sar -r Watch free memory closely around the time of incident Fortunately, sar data was handy. Looking at free memory, something is odd. 07:40:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused 07:50:01 AM :00:01 AM :10:01 AM :20:01 AM :32:50 AM :40:02 AM :50:01 AM :00:01 AM :10:01 AM :20:01 AM :30:02 AM :41:54 AM :50:01 AM :00:01 AM :10:01 AM Data analysis It is evident that free memory dropped to a smaller value. Then, free memory went up by couple of Mbs. We can derive two things: Enormous memory pressure at 8:32 Memory was released back at 8:40 Of course, there is paging and swapping going on. But, can that be justify high CPU usage in kernel mode? What about database freeze? Memory breakup! Server has 20GB of memory. SGA uses approximately 10GB. Database uses direct I/O and so UNIX buffer usage must be minimal. PGA target is 2GB and maximum ever allocated is 800MB. No other application running in that database. Connection count is 500. So, where is remaining 9GB is used? Client argument is that there shouldn't be any paging or swapping. Memory breakup! 5GB allocated for PageTables alone! cat /proc/meminfo MemTotal: kb MemFree: kb Buffers: kb Cached: kb SwapCached: kb Active: kb Inactive: kb HighTotal: 0 kb HighFree: 0 kb LowTotal: kb LowFree: kb SwapTotal: kb SwapFree: kb Dirty: 556 kb Writeback: 0 kb Mapped: kb Slab: kb CommitLimit: kb Committed_AS: kb PageTables: kb VmallocTotal: kb VmallocUsed: kb VmallocChunk: kb HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kb Memory breakup! HugePages are not being used. cat /proc/meminfo MemTotal: kb MemFree: kb Buffers: kb Cached: kb SwapCached: kb Active: kb Inactive: kb HighTotal: 0 kb HighFree: 0 kb LowTotal: kb LowFree: kb SwapTotal: kb SwapFree: kb Dirty: 556 kb Writeback: 0 kb Mapped: kb Slab: kb CommitLimit: kb Committed_AS: kb PageTables: kb VmallocTotal: kb VmallocUsed: kb VmallocChunk: kb HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kb Page size OS page size is 4KB. 20GB /4KB is 5.2 million OS pages. SGA is part of these 5.2 million OS pages. Just like any other memory page, SGA buffers also can be paged out. If there is a free memory need, then kscand/kswapd scans 5.2 million OS pages looking to free memory. That's why we had all CPUs used by kernel trying to free memory during memory starvation. Solution Fix is easy enough. We need to reduce pagetables size and reduce pages needed to be scanned by kscand/kswapd. Enter hugepages. We setup SGA to use hugepages. SGA using hugepages is locked in memory and not page dout. Pagetables size went down to 200MB or so. Database performance was within acceptable level. References Oracle support site. Metalink.oracle.com. Various documents Internal s guru Steve Adam s website Jonathan Lewis website Julian Dyke s website Oracle8i Internal Services for Waits, Latches, Locks, and Memory by Steve Adams Tom Kyte s website Asktom.oracle.com Blog: 16
Recommended
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks