Applies to:
Solaris Operating System – Version: 8 and later [Release: and later ]
All Platforms
Goal
Memory allocation by Intimate Shared Memory (ISM) is well understood.
Dynamic Intimate Shared Memory(DISM) works slightly differently, and this needs to be accounted for in the system configuration. Otherwise, DISM may not function as expected and it’s usefulness will be limited.
Solution
When shared memory is allocated via shmget(), its size is subtracted from the
available virtual swap space.
When a shared memory segment allocated via shmget() is attached to a process,
it is declared as ISM or DISM by the flags to the shmat() call:
#define SHM_SHARE_MMU 040000 /* share VM resources such as page table */
#define SHM_PAGEABLE 0100000 /* pageable ISM */
The SHM_SHARE_MMU flag makes the shared memory segment an ISM segment, while the
SHM_PAGEABLE flag makes it a DISM segment. It should be noted that a segment
cannot be both, and an attempt to set both flags will fail.
A DISM segment can have portions of the segment selectively locked into memory
by a (root-privileged) process, using the mlock() call. This allows the program
to more selectively manage which portions of the shared memory need to be locked
in at a given time.
The down side to this is that the DISM segment will subtract the size of the
segment from the available memory a second time – once for its in-memory space,
and a second, for its possible swap usage – effectively doubling the virtual
memory requirement for any DISM segments. This second allocation occurs during
the mlock() call.
Thus, a system using DISM needs to have twice as much virtual swap available as the
total DISM segments in use. If the system exhausts available swap, DISM segments
will not be mlock()’able. In Solaris:
Virtual swap = Physical swap (disk) + physical memory
Reservation is done against the physical swap (disk) first. If that is exhausted or not configured then reservation is done against the physical memory. If both are exhausted then shmat() or mlock() may fail. To avoid such failures due to lack of virtual swap configure large size of physical swap in the form of disk or file.
If the system is intended to have little or no disk swap, DISM is not
appropriate and should not be used.
© 2010, www.oracledatabase12g.com. 版权所有.文章允许转载,但必须以链接方式注明源地址,否则追究法律责任.
相关文章 | Related posts:
- Over Allocation Of Resources By Setting High Values For PGA And SGA.
- Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
- Oracle In-Memory Database Cache Oracle TimesTen In-Memory Database
- CI Locks Causing System Hangs
- FAQ Memory Corruption
- Automatic PGA Memory Management
- RMAN Performance Tuning Using Buffer Memory Parameters
- How To Troubleshoot Memory Leaks on UNIX
- Script: Listing Memory Used By All Sessions
- DIAG: Alter System Dump Undo – Quick Reference
Sun Fire[TM] Servers : Dynamic Reconfiguration and Intimate Shared Memory.
Description
To predict the effect of activities such as Dynamic Reconfiguration (DR) on a given production environment, it is useful to know how Intimate Shared Memory (ISM) is mapped to physical address (PA) spaces, and how the PA spaces can be mapped to boards. This Technical Instruction provides details.
The aim of this document is to assist in analysis, and to help avoid DR failures in the following cases, where the cfgadm(1M) command reports :
* Memory operation refused,
* No available memory target,
* VM viability test failed.
Steps to Follow
Dynamic Reconfiguration and Intimate Shared Memory
Contents:
1. Finding ISM segments,
2. Using pmap(1M) to determine if segment is ISM or DISM,
3. Using SUNWscat to determine ISM memory allocations,
3.a. Mapping ISM segments to PA space,
3.b. Mapping PA space to a board,
4. A simple example,
5. References.
——-
1. Finding ISM segments :
The best way to find an ISM segment, is to use the ipcs(1M) command.
As per its definition, the ipcs(1M) utility prints information about active inter-process communication facilities.
By using the “-Am” or “-im” options, it will show if ISM segments are in use in a given environment.
To differentiate an ISM segment from a shared memory(SHM) segment, look at the ISMATTACH value.
ISMATTACH reports the number of ISM attaches to the associated shared memory segments.
If ISMATTACH reports a non null value, then it is an ISM/DISM segment.
DISM – Dynamic Intimate Shared Memory.
SHM segments are reported with ISMATTCH = 0
ISM/DISM segments are reported with ISMATTCH > 0
Example : The following outputs report an ISM/DISM segment(Id 100) and a SHM segment(Id 101).
# ipcs -im
T ID KEY MODE OWNER GROUP ISMATTCH
Shared Memory:
m 100 0x1bb –rw-rw-rw- root other 4
m 101 0×25600000 –rw-rw-rw- root other 0
# ipcs -Am
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME ISMATTCH
Shared Memory:
m 100 0x1bb –rw-rw-rw- root other root other 4 128000000 658 661 1:50:15 no-entry 1:50:11 4
m 101 0×25600000 –rw-rw-rw- root other root other 0 256000 717 0 no-entry no-entry 3:33:28 0
Note :
It is possible to have ISM/DISM segments reported with ISMATTCH = 0, when the processes using the segments have been killed w/o properly removing the segment (ipcrm(1M)) …
Note :
It is not possible to distinguish an ISM segment from a DISM segment, using the ipcs(1M) command.
2. Using pmap(1M) to determine if segment is ISM or DISM :
When the ISMATTCH value from the ipcs(1M) command is non null, there is no way to know if it is an ISM or DISM segment. It is possible though, to use pmap(1M) to get this information.
In the following example, ‘ipcs -im’ command output shows 2 segments; ID’s 100 and 200.
# ipcs -im
T ID KEY MODE OWNER GROUP ISMATTCH
Shared Memory:
m 100 0x1b813514 –rw-r—– smcorau smcorag 8
m 200 0x1bb –rw-rw-rw- smcorau smcorag 4
Both segments are owned by a user called “smcorau”.
# ps -eaf | grep smcorau
smcorau 6731 1 0 Dec 13 ? 0:01 ora_lgwr_SunMC
smcorau 6739 1 0 Dec 13 ? 0:17 ora_dism
Running the pmap(1M) command on the processes that are using the segments, it is possible to determine if they are ISM or DISM segments.
# pmap 6731
6731: ora_lgwr_SunMC
00010000 24888K r-x– /opt/SUNWsymon/oracle/product/8.1.7/bin/oracle
0186C000 240K rwx– /opt/SUNWsymon/oracle/product/8.1.7/bin/oracle
018A8000 1576K rwx– [ heap ]
80000000 75064K rwxsR [ ism shmid=0x64 ]
[output omitted] ^^^^^^^^^^^^^^
# pmap 6739
6739: ora_dism
[output omitted]
F7000000 126976K rwxs- [ dism shmid=0xc8 ]
[output omitted] ^^^^^^^^^^^^^^^
The output above, shows that :
Segment ID 100 (0×64) is in ISM mode.
Segment ID 200 (0xc8) is in DISM mode.
If neither ISM nor DISM is mentioned, then this segment is SHM mode.
3. Using SUNWscat to determine ISM memory allocations :
Note :
The Solaris Crash Analysis Tool (scat) is not bundled in Solaris[TM] and is not supported. If you do not feel comfortable to install SUNWscat and use it on the live production system then it is possible to collect a save a crash dump of the live running Solaris system. Then provide Sun Services personel with the gathered core file for an analysis via SUNWscat. See Document Document: 1004803.1 for more details for how to collect the crash dump. But keep in mind that doing the same analysis from a live core may not be possible.
Note :
Due to some changes in the anon_map and anon_hdr structures in Solaris{TM} 8 and 9 OS (CR#4928612 ), SCAT 4.1 is only applicable for Sun Fire systems running Solaris 8 OS + KJP 117000-05 and later , or Solaris 9 OS KJP 117171-04 and later .
Note :
The following commands can consume resources and, on a large busy system, may take some time to complete, and may lead to a degradation in performance.
3.a. Mapping ISM segments to PA space :
It is possible to map the ISM/DISM segments to PA space and determine which pages a segment is using.
This can be done using the Solaris Crash Analysis Tool and the ‘ipc -M’ command.
Example :
SolarisCAT(live)> ipc -M
IPC status from
[...]
*shmid_ds T ID KEY MODE OWNER GROUPCREATOR CGROUP NATTCH ISMATT SEGSZ LCKSZ CPID LPID
0x30016c5e000 ISM m 0 0x0000029c –rw-rw-rw- user 1 user 1 2 2 536870912 556 557
pid basevaddr lock cmd
ISM 557 0xf7000000 ./pgm1
ISM 556 0xf7000000 ./pgm2
PA range size
=========================== ================================
0×6000000000-0x61ffffffff (0:8K 0:64K 0:512K 118:4M) 472M
0×12000000000-0x1203fffffff (0:8K 0:64K 0:512K 10:4M) 40M
This output can be read :
1st board has 0* 8K + 0*64K + 0*512K + 118*4M pages,
2nd board has 0* 8K + 0*64K + 0*512K + 10*4M pages.
These comprise the 512M ISM segment used by the processes pgm1 and pgm2.
Reminder :
When possible, ISM are composed of large pages (4MB of contiguous memory).
Note :
It is possible to identify the type of segment from ‘ipc -M’ via the “T” column : SM or ISM or DISM.
3.b. Mapping PA space to a board :
Once you have the PA ranges, you can map them to the PA’s assigned to each board. The PA’s can be seen in the ‘cfgadm’ output.
# cfgadm -a -s”cols=ap_id:info,select=type(memory)”
Ap_Id Information
SB3::memory base address 0×6000000000, 1048576 KBytes total
SB9::memory base address 0×12000000000, 8388608 KBytes total, 712640 KBytes permanent
Then, in the previous example, all the pages in the range
0×6000000000-0x61ffffffff come from SB3, and all the pages in the range
0×12000000000-0x1203fffffff come from from SB9.
The scat ‘seg phys’ command maps the PA ranges on all boards in the domain.
SolarisCAT(live)> seg phys
[...]
phys_install list:
PA range size
=========================== =====================
0×6000000000-0x61ffffffff 0×200000000 (8G)
0×12000000000-0x1203fffffff 0×40000000 (1G)
[...]
Note :
On the Sun Fire[TM] 12K/15K/E20K/E25K server, it is possible to view the platform memory slice map from within redx using the ‘pcd’ command;
Keep in mind that it is not recommended to use redx on a live system.
redx> pcd p
PCD Target Clock Freq = 75.000 ==> Target Interconnect Freq = 150.000 MHz.
mem_slice_map[17:0]= 17 15 16 14 13 12 11 10 09 08 07 06 05 04 03 02 01 00
redx>
This map shows that no DR operations have resulted in copy-rename operations.
See Problem Resolution Document: 1001683.1 for details about “Sun Fire[TM] 12K/15K/E20K/E25K:
Location and Relocation of Kernel for DR Operations” and Technical Instruction Document: 1012349.1
“Kernel Cage Splitting Overview” for additional information kernel memory
location in Solaris[TM] 9 with KU patch 118558-05 or higher and platmod patch
117124-07 or higher .
Also, if a domain has IO board 2, and this is the least significantly numbered I/O board in the domain, then it will contain the ‘golden’ IOSRAM. You can find the IO containing the GIOSRAM in the POST log files.
postYYMMDD.HHMM.SS.log:Golden sram is on Slot IO2.
This SRAM can be queried to show which slices have which physical address space, and how big each one is:
redx> bbgdcd 2 m
GDCD Memory chunk list. 2 chunks: 128GB
Chunk PA Size(MB) Slice
0 0060.00000000 00000400 1024 3
1 0120.00000000 00002000 8192 9
Total: 00002400 9216
Note :
On a Sun Fire server domain, using “across-boards” for the interleave-scope setting, there is no way to locate the segments. ‘ipc -M’ reports the PA as a whole.
Example from a 2 * 16GB domain :
[...]
PA range size
=========================== ================================
0-0x7ffffffff (7:8K 4:64K 2:512K 18:4M) 73.0M
[...]
4. A simple example :
Are there any Shared Memory Segments in use on my Sun Fire 15K domain ?
root@15k # ipcs -im as of Thu Jan 6 14:14:36 GMT 2005
IPC status from
T ID KEY MODE OWNER GROUP ISMATTCH
Shared Memory:
m 0 0x1b813514 –rw-r—– smcorau smcorag 8
Which are the processes using this segment ?
root@15k # ps -eaf | grep smcorau
smcorau 2414 1 0 Jan 03 ? 0:00 ora_smon_SunMC
smcorau 2406 1 0 Jan 03 ? 0:00 ora_pmon_SunMC
smcorau 2412 1 0 Jan 03 ? 0:16 ora_ckpt_SunMC
smcorau 2408 1 0 Jan 03 ? 0:00 ora_dbw0_SunMC
smcorau 2410 1 0 Jan 03 ? 0:00 ora_lgwr_SunMC
smcorau 2380 1 0 Jan 03 ? 0:00 /opt/SUNWsymon/oracle/product/8.1.7/bin/tnslsnr smcdblistener -inherit
smcorau 2416 1 0 Jan 03 ? 0:00 ora_reco_SunMC
smcorau 2418 1 0 Jan 03 ? 0:00 ora_s000_SunMC
smcorau 2420 1 0 Jan 03 ? 0:00 ora_d000_SunMC
What is shmid 0 ?
root@15k # pmap 2414
2414: ora_smon_SunMC
00010000 26592K read/exec /opt/SUNWsymon/oracle/product/8.1.7/bin/oracle
01A16000 264K read/write/exec /opt/SUNWsymon/oracle/product/8.1.7/bin/oracle
01A58000 96K read/write/exec [ heap ]
80000000 877312K read/write/exec/shared [ ism shmid=0x0 ]
[...]
root@15k # pmap 2406
2406: ora_pmon_SunMC
00010000 26592K read/exec /opt/SUNWsymon/oracle/product/8.1.7/bin/oracle
01A16000 264K read/write/exec /opt/SUNWsymon/oracle/product/8.1.7/bin/oracle
01A58000 824K read/write/exec [ heap ]
80000000 877312K read/write/exec/shared [ ism shmid=0x0 ]
[...]
and so on …
Both processes in the previous example are attached to shmid 0 which is in ISM mode.
Let’s try to locate the segment on the physical boards of this 2 boards f15k domain.
SolarisCAT(live)> ipc -M
Shared Memory:
[...]
*shmid_ds T ID KEY MODE OWNER GROUPCREATOR CGROUP NATTCH ISMATT SEGSZ LCKSZ CPID LPID
0x300174ba000 ISM m 0 0x1b813514 –rw-r—– 157521 3003 157521 3003 8 8 898367488 2404 25357
pid basevaddr lock cmd
ISM 2420 0×80000000 ora_d000_SunMC
ISM 2418 0×80000000 ora_s000_SunMC
ISM 2416 0×80000000 ora_reco_SunMC
ISM 2414 0×80000000 ora_smon_SunMC
ISM 2412 0×80000000 ora_ckpt_SunMC
ISM 2410 0×80000000 ora_lgwr_SunMC
ISM 2408 0×80000000 ora_dbw0_SunMC
ISM 2406 0×80000000 ora_pmon_SunMC
PA range size
=========================== ================================
0×2000000000-0x21ffffffff (96:8K 0:64K 0:512K 169:4M) 676M
0×20000000000-0x201ffffffff ( 0:8K 0:64K 0:512K 45:4M) 180M
So, ISM segment ID 0 is composed of pages from the PA ranges
0×2000000000-0x21ffffffff and 0×20000000000-0x201ffffffff.
To try to map the PA ranges to the physical boards :
root@15k # cfgadm -a -s”cols=ap_id:info,select=type(memory)”
Ap_Id Information
SB16::memory base address 0×2000000000, 8388608 KBytes total
SB17::memory base address 0×20000000000, 8388608 KBytes total, 997200 KBytes permanent
ISM segment ID 0 is located on board SB16 and SB17.
We can easily check why SB17 has the highest base address and SB16 has the lowest base address.
SolarisCAT(live)> seg phys
[...]
phys_install list:
PA range size
=========================== =====================
0×2000000000-0x21ffffffff 0×200000000 (8G)
0×20000000000-0x201ffffffff 0×200000000 (8G)
From memory slice mapping, we can see there was a copy/rename operation :
15k:sms-svc:7> grep -i golden post05*
post050103.1547.11.log:Golden sram is on Slot IO16.
post050103.1557.05.log:Golden sram is on Slot IO16.
redx> pcd p
mem_slice_map[17:0]= 16 01 15 14 13 12 11 10 09 07 08 06 05 02 03 17 04 00
redx> bbgdcd 16 m
GDCD Memory chunk list. 2 chunks: 128GB
Chunk PA Size(MB) Slice
0 0020.00000000 00002000 8192 1
1 0200.00000000 00002000 8192 16
Total: 00004000 16384
This was a very simple example. In most of the cases, pages that are constituent of the segments are spread on the system.
The same analysis can be done offline from a corefile gathered from the live system but keep in mind that this “picture” from the live system (‘savecore -L’) is a moving target and may not report the information needed for this procedure. As a summary, doing the same analysis from a live core may not be possible.
DISM Troubleshooting For Oracle9i and Later Releases
Description
DISM Troubleshooting For Oracle9i and Later Releases
Outline
1.
Incident Characteristics
2.
Background to Dynamic Intimate Shared Memory(DISM)
3.
Oracle’s implementation of DISM in Oracle9i
4.
What can go wrong
5.
How to diagnose it
6.
How to fix it
7.
Software release details
8.
Further information
Steps to Follow
Incident Characteristics
Symptoms
Extremely high System CPU %, as reported by vmstat or sar; extremely poor performance of Oracle9i, and later end-user applications.
Cause
A known Solaris[TM] 8 Operating System(OS) bug (4966813 Click Here) can cause this problem, and the following Solaris patch should be installed: Patch 117000-05 Click Here. This patch resolves the problem. However, DISM has worked adequately only for very small Oracle SGA configurations under Solaris 8 OS. The system has been observed spending a lot of time in the segspt driver allocating backing store for DISM. DISM should not be used on Solaris 8 OS. DISM should not be used on Solaris 9 OS prior to Update 2 (the 12/02 release).
On Solaris[TM] 9 OS Update 2 and later (and on Solaris 8 OS, once the future fix is in place), this behavior indicates that Oracle System Global Area(SGA) pages are not locked, forcing the kernel to use a slower locking mechanism (softlocks). This results in significantly increased system time and extremely poor performance.
Solution
Ensure that an appropriate Solaris release is being used, and that all necessary patches have been installed. Once you have confirmed that a supported Solaris release is in place, ensure that the oradism binary is configured correctly and that the SGA has been set to an appropriate size, allowing all active pages to be locked.
If you know you will never use Oracle’s dynamic SGA features, it may be simpler not to configure sga_max_size.
The rest of this document provides detailed explanations of DISM and its implementation in Oracle as well as the symptoms, cause, and suggested solution to possible problems related to DISM.
Background to DISM
What is DISM
Solaris Dynamic Intimate Shared Memory (DISM) provides shared memory that is dynamically resizable. DISM means that applications can respond to changes in memory availability by dynamically increasing or decreasing the size of optimized shared memory segments. The first major application to support DISM was Oracle9i – Oracle9i uses DISM for its dynamic System Global Area (SGA) capability.
The Origins of DISM
Many applications, and especially databases, use shared memory to cache frequently-used data (the buffer cache) and for interprocess communication. Solaris provides an optimized shared memory capability known as Intimate Shared Memory (ISM), and all major databases take advantage of it.
ISM benefits
ISM offers a number of benefits over standard System V shared memory:
1.
ISM shared memory is automatically locked by the kernel when the segment is created. This not only ensures that the memory cannot be paged out, it also allows the kernel to use a fast locking mechanism when doing I/O into or out of the shared memory segment, thereby saving significant CPU time.
2.
Kernel virtual-to-physical memory address translation structures, are shared between processes that attach to the shared memory, saving kernel memory and CPU time.
3.
Large pages, supported by the UltraSPARC Memory Management Unit (MMU), are automatically allocated for ISM segments (as of Solaris 2.6 OS). Large pages can reduce the number of memory pointers by a factor of 512. This reduction in complexity translates into noticeable performance improvements, especially on systems with large amounts of memory.
4.
Since memory is locked, no swap space is needed to back it, thereby saving disk space.
Unfortunately though, ISM segments cannot be resized. To change the size of an ISM database buffer cache, the database must be shutdown and restarted, affecting system availability. For example, removing memory by Dynamic Reconfiguration may require shutting down database instances.
DISM overcomes this limitation. A large DISM segment can be created when the database boots, with sections of it selectively locked or unlocked as memory requirements change. Instead of the kernel automatically locking DISM memory, though, locking and unlocking is done by the application (e.g. Oracle).
DISM Performance
When DISM was first released in Solaris 8 Update 3 (1/01), it inherited a number of the benefits of ISM. In particular:
1.
Memory is locked (by the application), preventing paging and allowing I/O to use fast kernel locking mechanisms.
2.
Kernel virtual to physical memory address translation structures are shared between processes that attach to the DISM segment, saving kernel memory and CPU time.
In this first release, large MMU pages were not supported. For Solaris 8 systems with 8GB of memory or less, it is reasonable to expect a performance degradation of up to 10% compared to ISM, due to the lack of large page support in DISM. The actual performance impact will vary, though, according to the amount of shared memory and the frequency of access to it. Sun recommends avoiding DISM on Solaris 8 either where SGAs are greater than 8 Gbytes in size, or on systems with a typical CPU utilization of 70% or more. In general, where performance is critical, DISM should be avoided on Solaris 8. As we will see, Solaris 9 Update 2 (the 12/02 release) is the appropriate choice for using DISM with systems of this type.
Solaris 9 Update 1 (the 9/02 release) introduced large page support for DISM segments. Tests have shown that, as of this release, DISM and ISM performance are equivalent. This enhancement is significant – the availability benefits of DISM can be enjoyed without compromising performance.
Solaris 9 Update 2 (the 12/02 release) further enhances DISM – unlocked memory can be more efficiently made available to other applications by proactively returning it to the free list, rather than waiting for the page daemon to locate it. This release also introduced a number of necessary DISM-related bug fixes, making it the minimum Solaris 9 release for running DISM.
Finally, since DISM memory is not automatically locked, swap space must be allocated for the whole segment. If the swap defined is not big enough, the system will reserve memory pages in RAM, which can cause a memory shortage. Memory reservation in RAM does not decreases freemem, but decreases availrmem. This remaining difference between ISM and DISM is unlikely to be a major issue, though, given the capacity of modern disk drives.
For more information on this extra swap reservation incurred when DISM is used, see Technical Instruction Document: 1010818.1 (Solaris[TM] Operating System: DISM double allocation of memory) “Solaris[TM] Operating System: DISM double allocation of memory”.
Oracle’s Implementation of DISM in Oracle9i
Oracle took advantage of DISM in Oracle9i (and subsequent releases, such as Oracle10g) to implement dynamic SGA resizing. Dynamic SGA resizing allows a database administrator (DBA) to respond to changing needs by increasing or decreasing SGA memory without a database reboot. Without DISM this feature would not be available on Solaris.
Since DISM requires the application to lock memory, and since memory locking can only be carried out by applications with superuser privileges, Oracle implemented a daemon that runs with root privileges. Since Oracle does not normally run with superuser privileges, the Role Based Access Control (RBAC) features introduced in Solaris 8 were used to provide that access in the first release of Oracle9i. RBAC allows a nominated binary to run with an effective uid of root. During the Oracle install procedure, the installer is asked to run a script as root; this script established the necessary RBAC permissions. In particular, entries were added to the /etc/user_attr and /etc/security/exec_attr files. Later Oracle releases simply made the $ORACLE_HOME/bin/oradism binary setuid root to achieve the same end.
The $ORACLE_HOME/bin/oradism binary implements the new root daemon. Look for it in ps with the description ora_dism_$ORACLE_SID.
For example:
serv1% ps -aef | grep dism
root 747 1 0 13:57:26 19:42 ora_dism_custdb
oradba1 18456 18391 0 23:37:08 pts/6 0:00 grep dism
Oracle9i introduced a new init.ora parameter, sga_max_size, to activate DISM. This variable establishes the maximum size to which the SGA can grow; it can only be modified statically (in other words an Oracle reboot is required before any change to sga_max_size takes effect). Oracle will use DISM instead of ISM if sga_max_size is set larger than the total of the database buffers (in particular, db_cache_size dynamic SGA resizing is not supported with the older db_block_buffers parameter), the shared pool, the redo buffers, the large pool, the Java pool, and the SGA fixed size (representing Oracle’s internal requirements).
Once DISM has been invoked, Oracle automatically locks an amount of memory determined by the total of the elements described above (database cache, shared pool, etc). Subsequently, the DBA can alter the size of the database buffers (db_cache_size) and the shared pool (shared_pool_size) with Oracle’s alter system command. For example:
alter system set db_cache_size =
Depending on the command, Oracle then locks additional memory, subject to the upper limit imposed by sga_max_size, or releases memory for use elsewhere by the operating system. The results of alter system actions are shown in the alert log file, typically located in $ORACLE_HOME/rdbms/log/alert_$ORACLE_SID.log. An example is shown below:
CKPT: Current size = 37904 MB, Target size = 3760 MB
Fri Jun 21 21:29:50 2002
Completed checkpoint up to RBA [0x7e.2.10], SCN: 0×0000.25c3f387
CKPT: Resize completed for buffer pool DEFAULT for blocksize 2048
Fri Jun 21 21:29:54 2002
ALTER SYSTEM SET db_cache_size=’3932160000′ SCOPE=MEMORCKPT: Begin
resize of buffer pool 3 (DEFAULT for block size 2048)Y;
What can go wrong
Oradism permissions not setup correctly
System administrators do not always install Oracle using the install script. Instead, the relevant directories may be transferred to another system using tar, cpio or a similar utility. If the /etc/user_attr and /etc/security/exec_attr files on the target system are not modified accordingly, the oradism program will not run with the correct permissions.
For later Oracle releases that made the oradism program setuid root, a transfer of the Oracle binaries to another system may not have preserved these permissions (for example, tar may have been run as the oracle user rather than as root). Also, if binaries are located on another system and mounted on the server with NFS, it is possible that the mount options needed to permit root access have not been set, with the result that the oradism setuid permissions will not be honored.
Oradism dies
If the Oracle9i oradism process dies for some reason (unlikely unless a system administrator accidentally kills it), all locked memory will be automatically unlocked. Performance will suffer accordingly.
Note, however, that oradism can be restarted automatically by Oracle10g if it dies.
SGA memory is only partially locked
If the DBA sets the database buffers, shared pool etc too large in the init.ora file, or later increases them too much with an alter system command, Oracle may not be able to successfully lock the whole of the requested memory. In this case, when Oracle uses that portion of the memory that is not locked, poor performance will result. Note that memory is not locked and unlocked in a single chunk lock and unlock operations are carried out on individual memory ‘granules’, typically 16Mbytes in size.
How to diagnose it
Solaris release
To determine your Solaris release, refer to the /etc/release file:
serv1% cat /etc/release
Solaris 9 12/02 s9s_u2wos_10 SPARC
Copyright 2002 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 05 November 2002
The system in the example above is running Solaris 9 Update 2 (as indicated by u2 in the s9s_u2wos_10 string).
SGA not locked
Systems with active use of unlocked SGA memory can be identified with either the lockstat or statit utilities. The lockstat utility ships with both Solaris 8 and 9; the statit utility is a widely-used although unsupported utility that can be downloaded from http://www.solarisdatabases.com.
The lockstat utility shows mutex contention. As root, look for mutex contention in the segspt_softunlock or spt_anon_getpages functions. Two examples are included below showing the first few lines of a lockstat profile from a system running an active Oracle instance with unlocked SGA memory. The following command can be used to generate these profiles.
serv1# /usr/sbin/lockstat -A -n 200000 sleep 10
Here are the two sample profiles:
Adaptive mutex spin: 22908 events in 10.006 seconds (2290 events/sec)
Count indv cuml rcnt spin Lock Caller
—————————————————————————
4896 21% 21% 1.00 27532 0x3001314ec88 spt_anon_getpages+0×54
81 0% 22% 1.00 11 0x300001a1e98 sc_flush+0x1c4
78 0% 22% 1.00 2 pse_mutex+0xb0 page_unlock+0x1c
77 0% 22% 1.00 2 pse_mutex+0×368 page_unlock+0x1c
Adaptive mutex spin: 561050 events in 10.017 seconds (56011 events/sec)
Count indv cuml rcnt spin Lock Caller
—————————————————————————
549729 98% 98% 1.00 146 0x300085a2580 segspt_softunlock+0xc4
834 0% 98% 1.00 10685 0x300085a2578 spt_anon_getpages+0×54
715 0% 98% 1.00 12 0x300085a2580 spt_anon_getpages+0×64
The presence of such mutexes indicates that SGA memory is not locked.
In statit, look for faults due to s/w lcking req. A non-zero result on a system with an active Oracle instance indicates that some or all SGA memory is not locked. The following example illustrates how to check for this behavior:
serv1% statit sleep 30 | grep lcking
31.62 faults due to s/w lcking req 0.00 kernel as as_flt()s
The example above shows softlocks, indicating a problem with SGA memory locking.
Permissions problems
Examine the alert log for each instance. The following entry indicates that Oracle was unable to start the oradism process with superuser privileges:
Wed Jul 16 14:03:39 2003
WARNING: ——————————-
WARNING: oradism not set up correctly.
Dynamic ISM can not be locked. Please
setup oradism, or unset sga_max_size.
[diagnostic 0, 5, 0]
—————————————-
Performance will be very significantly degraded for this instance, since SGA memory is not locked.
To check whether a running oradism process has the appropriate permissions, use the pcred program (available on both Solaris 8 and 9). Consider the following example:
serv1% ps -aef | grep dism
root 747 1 0 13:57:26 19:42 ora_dism_custdb
oradba1 18456 18391 0 23:37:08 pts/6 0:00 grep dism
serv1% sudo pcred 747
747: e/r/suid=0 e/r/sgid=5432
In this example, the oradism program is running with superuser privileges (suid=0).
To check if RBAC is configured appropriately, look for the appropriate entries in the /etc/user_attr and /etc/security/exec_attr files.
serv1% grep -i dism /etc/security/exec_attr
Oracle DISM
mgmt:suser:cmd:::/export/home/oracle/bin/oradism:euid=serv1% grep -i
dism /etc/user_attr
oracle::::type=normal;profiles=Oracle DISM mgmt
In this case both files have been setup. Instances using binaries located in /export/home/oracle/bin will be able to run oradism with root privileges.
Check, too, if the setuid bit has been set on the oradism program. For example:
serv1% ls -l $ORACLE_HOME/bin/oradism
-rwsr-sr-x 1 root dba 9280 Apr 9 2002 /export/home/oracle/bin/oradism*
In the above example the oradism program is owned by root, and the setuid bit is set (as indicated by the first s in rws). The program should therefore be able to start with appropriate permissions.
Oradism dies
Check for the presence of the oradism process with the ps program.
Unfortunately, in the unlikely event that this process dies, in Oracle9i it will not be noted in the alert log file until the next time alter system is used to grow or shrink memory.
Oracle10g has the ability to restart oradism if it dies.
SGA memory only partially locked
In Oracle10g, a new table (x$ksmge) can be examined to determine the state of locks for each granule of SGA memory (each granule is typically 16Mbytes in size).
For Oracle9i, an interpose library is from the Veritas, Oracle, Sun Joint Escalation Center (VOSJEC). Untar the file into a temporary directory, and install the library according to the instructions in the README file. When oradism is started, a /tmp/mlock.log file will be created, and all lock and unlock operations will be logged to the file along with their return status (non-zero return status indicates failure). An example is given below of the information that will be logged if locks attempts are successful:
Mon Nov 10 13:57:26 2003: Detected mlock(380000000, 400000)
Mon Nov 10 13:57:26 2003: mlock return status=0
Mon Nov 10 13:57:26 2003: Detected mlock(1013000000, 1400000)
Mon Nov 10 13:57:26 2003: mlock return status=0
Mon Nov 10 13:57:26 2003: Detected mlock(381000000, 1000000)
Mon Nov 10 13:57:26 2003: mlock return status=0
An example of unsuccessful lock attempts is shown below:
Sat May 25 02:45:22 2002: Detected mlock(e61000000, 1000000)
Sat May 25 02:45:22 2002: mlock return status=-1, errno=11 <————-
Sat May 25 02:45:22 2002: Detected mlock(e62000000, 1000000)
Sat May 25 02:45:23 2002: mlock return status=-1, errno=11 <————-
How to fix it
Solaris OS release issues
As previously noted, a known Solaris 8 OS bug (4966813 Click Here) can cause this problem. The solution is to install Solaris Patch 117000-05 Click Here. DISM should only be used on Solaris 8 OS, if patch 117000-05 or later is installed . DISM should not be used on Solaris 9 OS prior to Update 2 (the 9/02 release).
Permissions problems
The simplest way of ensuring appropriate permissions is to make the oradism program setuid root, obviating the need for RBAC. This can be achieved as follows:
serv1% chown root $ORACLE_HOME/bin/oradism
serv1% chmod 4755 $ORACLE_HOME/bin/oradism
serv1% ls -l $ORACLE_HOME/bin/oradism
-rwsr-xr-x 1 root dba 9280 Apr 9 2002 /export/home/oracle/bin/oradism*
Oradism dies
If the oradism process dies in Oracle10g, no action is necessary; Oracle is able restart it. In Oracle9i, though, the only practical solution is to shutdown and restart the instance.
SGA memory only partially locked
The main solution to this problem is to ensure that it doesn’t happen. By installing the interpose library, it is possible to monitor the success of locking operations. Attempts to lock more memory than the system has available should be avoided.
General Guidelines
The following guidelines are offered in conclusion:
1.
Ensure you are using an appropriate release of Solaris OS(refer to the next section of this document).
2.
If you don’t need to resize the SGA dynamically, you probably don’t need to set sga_max_size.
3.
If you use sga_max_size, check that oradism is working correctly as described above.
4.
On Solaris 8 OS, DISM should only be used if Solaris Patch 117000-05 is installed. Even once the patch is installed, DISM should be avoided where performance is critical. Use of DISM can cost up to 10% in performance compared to ISM for SGAs up to 8 Gbytes in size (although your mileage may vary depending on your circumstances!). Sun recommends avoiding DISM on Solaris 8 where the SGA is larger than 8 Gbytes, or where CPU utilization is typically greater than 70%.
5.
On Solaris 9 as of Update 2, the performance of DISM is equivalent to ISM. DISM should not be used with releases of Solaris 9 prior to Update 2.
Software Release Details
Solaris 8 OS
For correct operation of DISM, Solaris Patch 117000-05 must be installed. To install 117000-05 the system should already have Solaris Patch 108528-29 installed.
Solaris 9 OS
DISM should only be used on Solaris 9 Update 2 or later. Earlier releases of Solaris 9
OS do not contain all necessary patches.
Oracle
Oracle9i supports DISM. For the first release only (Oracle9.0.1.0), an Oracle patch is required.
When Will DISM Start On Oracle Database?
Applies to:
Oracle Server – Enterprise Edition – Version: 9.2.0.1 to 11.1.0.8
Sun Solaris x86 (32-bit)
Sun Solaris SPARC (64-bit)
Sun Solaris x86-64 (64-bit)
Sun Solaris SPARC (32-bit)
Goal
When DISM will start on Oracle Database?
Solution
Dynamic intimate shared memory (DISM) is a unique feature of Solaris. It achieves dynamic memory management. On other OSs, When you specify SGA_MAX_SIZE, then the complete size of SGA_MAX_SIZE is reserved from your memory (reserved from Swap not from physical RAM) at instance startup. Then you can dynamically increase SGA_TARGET size until you reach SGA_MAX_SIZE. On Solaris, however, and due to DISM feature, SGA_MAX_SIZE is not reserved from the memory at instance startup but it is allocated when needed. This presents a great value since you do not reserve memory you are not using.
The use of DISM requires that the ora_dism process starts. The process will not start unless SGA_MAX_SIZE is greater than sum total of all SGA components. This means that in case you use SGA_TARGET. you specify size of SGA_MAX_SIZE greater than SGA_TARGET.
**For 11g only**
Oracle Database also uses DISM if it is available on the system and MEMORY_TARGET or MEMORY_MAX_TARGET is set by the user.
By default, SGA_MAX_SIZE is not set during database installation and hence it takes the same value as SGA_TARGET. The use of SGA_TARGET is not a necessity to use DISM. Oracle will use DISM instead of ISM if SGA_MAX_SIZE is set larger than the total of the database buffers (in particular, db_cache_size dynamic SGA resizing is not supported with the older db_block_buffers parameter), the shared pool, the redo buffers, the large pool, the Java pool, the fixed size. In this case you will need to check the total of following parameters:
* db_cache_size
* java_pool_size
* large_pool_size
* log_buffer
* shared_pool_size
* streams_pool_size
* db_keep_cache_size
* db_recycle_cache_size
The difference between the total and the SGA_MAX_SIZE is so small that it can be approximated in calculations related to Granule size, so DISM should not be started, otherwise the ora_dism process will be started. .
Please note that to use DISM, you must have ISM enabled (which is the default case on Oracle 10g/11g). You will need always to check parameter:
*._use_ism= TRUE
WARNING: Oradism did not start up correctly — Reported in the Alert.Log
Symptoms
The alert log reports the following errors:
WARNING: oradism did not start up correctly.
Dynamic ISM can not be locked.
oradism creation failed for unknown reasons 0 8 303
Cause
Mount points set incorrectly for Oracle.
Solution
Correct the mount points.
You need to have setuid on the Oracle mount points or setuid won’t work.
Here’s the error from the system error log:
Mar 15 13:38:22 test test: [ID 809163 kern.info] NOTICE: oradism, uid 303: setuid execution not allowed, dev=13b00001770
Mount point shows:
/app on /dev/vx/dsk/orabindg/appvol read/write/nosetuid/nodevices/delaylog/largefiles/ioerror=mwdisable/dev=4ec1770
Solaris[TM] 10 Operating System allows capture of entire text segment in corefiles
Description
This technical instruction outlines how the Solaris[TM] 10 Operating System allows capture of entire text segment in corefiles
Steps to Follow
In analysing coredumps from applications in user space, before looking at the
core with a debugger, it’s generally necessary to match the libraries and
binaries.
In Solaris[TM] 10 Operating system(OS), the /usr/bin/coreadm command can now be
configured to capture the entire text segment of the process, thereby removing
the requirement to match the libraries and binary.
Coreadm can now be given a parameter:
‘-G’ (global)
or
‘-P’ (per process)
to include various content.
‘coreadm -I content’ is equivalent to ‘coreadm -P content 1′ with the exception
that it is persistent across reboots.
As you would expect, coreadm will only let you modify the coredump parameters of
processes for which you have access rights. So, to use the ‘-G’ option you will
need to have sys_config privilege. The ‘-I’ option requires either sys_config or
proc_owner privilege. To apply ‘-P’ to processes that you don’t own, you will
need proc_owner privilege.
The root user has these privileges by default.
From the man page for coreadm
A core file content description is specified using a series
of tokens to identify parts of a process’s binary image:
anon Anonymous private mappings, including thread stacks
that are not main thread stacks
data Writable private file mappings
dism DISM mappings
heap Process heap
ism ISM mappings
rodata
Read-only private file mappings
shanon
Anonymous shared mappings
shfile
Shared mappings that are backed by files
shm System V shared memory
stack Process stack
text Readable and executable private file mappings
ctf CTF type information sections for loaded
In addition, you can use the token all to indicate that core
files should include all of these parts of the process’s
binary image. You can use the token none to indicate that no
mappings are to be included. The default token indicates
inclusion of the system default content
(stack+heap+shm+ism+dism+text+data+rodata+anon+shanon+ctf).
The /proc file system data structures are always present in
core files regardless of the mapping content.
You can use + and – to concatenate tokens. For example, the
core file content default-ism would produce a core file with
the default set of mappings without any intimate shared
memory mappings.
If you can reproduce the problem, then it will be worthwhile configuring for
content ‘all’. If the process can be identified while running, then you can
apply the ‘-P all’ option to the pid. Failing that, you can force all coredumps
system wide to ‘all’ using the option ‘-G all’.
Examples
coreadm -P all 12345
Forces pid 12345 to dump all contents into a core file if
it core dumps.
coreadm -G all
Makes the same change system wide.
Note
As shipped, captured core dumps will contain the text segment. No action needs
to be taken to enable this functionality.
Do you know if all of this is true and applies to Oracle 11gR2/Solaris10 sparc?
True! But up to your 11gR2 setting!