Memory Matters in 2011

IBM System z Technical University – Vienna , Austria – May 2-6

zZS29 Memory Matters in 2011
Martin Packer

1
© 2011 IBM Corporation


Notices
This information was developed for products and services offered in the U.S.A.

Note to U.S. Government Users Restricted Rights — Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently
available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally
equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation
of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can
send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.
The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION
PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions,
therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the
publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites
are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and
cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the
suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies,
brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these
sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface
for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability,
serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing,
or distributing application programs conforming to IBM's application programming interfaces.



Trademarks

 This presentation contains trade-marked IBM products and technologies. Refer to the
following Web site:

http://www.ibm.com/legal/copytrade.shtml



Abstract

Memory - hardware, z/OS and associated middleware - continues to be an
exciting subject, with enhancements coming thick and fast. Meanwhile its
usage and management is often poorly understood.
This presentation brings performance practitioners and capacity planners
up to date with topics on both real and virtual memory, with DB2 discussed
as a key memory-consuming product.
There will also be an opportunity to discuss what instrumentation and
capabilities are needed for the future. (Martin is in active discussions with
hardware and software development organisations on both the function and
instrumentation fronts.)



Topics

 System-Level
– Paging Subsystem Design
– Dump Space
– 1 MB (Large) Pages
– z/OS R.10 64-Bit Common
– ADDR64
– System Area Above 2GB
 Coupling Facility
 DB2
 Conclusion



System-Level



Systems That The “New” z/OS R.8 Algorithm Might Impact

 Design point is for “zero paging” considerations
– Most production systems
 Systems likely to page:
– Small production systems
– Development and Test systems
• Is “Development” really so low priority these days?

 A potential mitigator for paging systems:
– Frequently-referenced pages generally “more important”
 Pragmatic choice:
– Whether to allow some memory constraint
– Even more pragmatic choice:
• Whether to configure for some unusual peak
– (See later foils on dumping)



Paging Subsystem Design
 Zero paging is not a reason to skimp on your paging subsystem
– Even more memory to dump when you have to
• At least be able to dump your big address spaces eg DBM1
• It’s preferable to be able to dump the entire system
 Rule of Thumb:
– Have your free paging space be at least 1.5 times your real storage
• Gives you a chance at containing "big dump" cases
 Another Rule of Thumb:
– Don't let your page data sets get above 30% full
• Contiguous Slot Algorithm degrades above that point
 These Two Rules of Thumb complement each other
– But consider them a starting point in your design discussions
 Consider the value of PAV now that ASM supports it
– But still consider the importance of proper I/O design



Paging Subsystem Design …
 It’s possible to have two page data sets on the same volume
– Optimism that performance is OK with PAVS enabled
• IOSQ observed to be 0
• Disc observed to be non-trivial but probably the case with 1
– Paging I/O not cached
– Motivation: To use large volumes exclusively for paging
 APAR OA20749 allows larger page data sets
– Limit up from 4 GB to 44.9 GB
• 64K cylinders
– Available with z/OS R.8 onwards
– 1 large page data set can have 2 PAVs
• 2 smaller ones can have 4
 In a not-so-recent customer case 2107 greatly outperformed 2105
– And ASM was observed to route paging I/O towards the 2107 data sets
 HyperPAVs work properly for page data sets as of R.10



Dump Space
 Dumping managed at system level by CHNGDUMP (CD) operator
command
– CD SDUMP,BUFFERS=nnnM
• Target value for real frames
– CD SDUMP,MAXSPACE=nnnnnnnnM
• Maximum concurrent virtual space
– Defaults to 500MB
• If you run out dumps can’t be taken
• Partial dumps a possibility
– Likely to impede diagnostics
 Dumping much faster if all in memory than a combination of paging
space and memory
– A trade off between availability and memory provisioning
 Subsystems may be smart in what they dump and in what order
– To maximise the likelihood of getting adequate diagnostics



Dumping Enhancements in z/OS R.12

 Batch page-in I/O operations during SDUMP capture eliminates much of
the I/O delay
 Data captured will no longer look recently referenced
– This data will be paged-out before other potentially more important
data
 Certain components now exploit a more efficient capture method in their
SDUMP exits
– For example GRS for SDATA GRSQ data, IOS for CTRACE data, and
configuration dataspaces



1MB (Large) Pages
●
Issue: Translation Lookaside Buffer (TLB) Coverage shrinking as % of memory size
●
1 MB Pages allow for a single TLB entry to fulfill many more address translations
than 4KB pages
●
1 MB Pages will provide exploiters with better TLB coverage
●
Only fixed above the bar virtual can be backed by 1 MB pages
●
No paging
●
Mixed 4 KB and 1 MB running the rule
●
OA25485: NEW FUNCTION - FACILITY CLASS FOR RSM LARGE PAGE ACCESS
●
Allows non-Authorised applications to use IARV64 REQUEST=GETSTOR
PAGEFRAMESIZE=1MEG or MAX
●
Facility Class is IARRSM.LRGPAGES
●
Exploitation by Websphere Application Server V7
●
Java heap
●
Java 6 SR 1 or later with -X1p1M flag
●
Buffer Pools in DB2 10
●
OA32001: LFAREA PARAMETER INCONSISTENT WITH VALIDITY CHECKS
●
Corrects calculation if LFAREA=nn% used to be % of >2GB real


z/OS R.12 Large Page Support

 Nucleus
– Note: Nucleus is in 31-bit storage
– Expected to improve performance for z/OS itself
 Large Page Coalesce support
– During page storage shortage situations available large pages
converted to 4KB pages
– Later 4KB pages will be coalesced to form 1MB pages
– OA31116: VARIOUS LARGE PAGE COALESCE AND RECOVERY
FIXES
• Implements this and fixes some other problems
• For Release 10 and 11 also



z/OS R.10 64-Bit Common
 IARV64 REQUEST=GETCOMMON
 Allows sharing above the bar across every address space
– In a much simpler fashion than Shared 64-Bit
 Controlled by HVCOMMON
 Any address space can access storage without explicitly having
to request access
 To limit overlays only allowed if:
– Authorized code
– System keys 0-7
 Storage can be DREF, Fixed
– Unlike Shared
 Storage needs to be explicitly freed
 Requires explicit exploitation
– To achieve 31-Bit Common VSCR


RMF Support For 64-Bit Common

 Type 71:
– Number of 64-bit common memory objects allocated
– 64-bit common memory frames backed in real storage
– 64-bit fixed common memory frames
– 64-bit common memory auxiliary storage slots

– Additional information on shared memory objects:
• Number of 64-bit shared memory objects allocated
• High virtual shared memory frames backed in real storage



RMODE 64 For Non-Executable Modules

 New with z/OS R.11
 The LOAD macro supports new keywords to identify an area above 2G in virtual
for a directed load:
– eg ADDR64 keyword as the analog of ADDR
 The system does not verify that the module truly can be relocated above 2G
– For example, it may have a 4-byte address constant to another part of itself
– This is consistent with current LOAD with ADDR behavior
 The system does not verify that the module is truly non-executable.
– If attempt is made to transfer flow of execution to the module, results are
unpredictable
• Also true if any executable code is placed above 2G by any mechanism



USEZOSV1R9RULES

 Setting to NO changes GETMAIN / STORAGE OBTAIN behaviour
– Generally a safe thing to do
– However may result in ABEND0Cx, overlay of storage, or other
problems
• Documented in APAR OA27291
 Setting to NO has been observed to improve DB2 startup behaviour:
– Particularly the opening of a large number of VSAM data sets
– Note also DB2 APAR PM17542 and z/OS R.12 MEMDSENQMGMT:
• Uses in-memory structures to decrease ENQ time for large
numbers of data sets
 USEZOSV1R9RULES introduced with z/OS Release 10



System Area Above 2GB

 A new system area will be created in the address
space 64-bit map
– Analogue of (E)LSQA
 Memory objects allocated in the System Area will
start at x’8_00000000’
 An authorized program can issue IARV64
REQUEST=GETSTOR, LOCALSYSAREA=NO|YES
to indicate that the memory object should be allocated
from the System Area above 2GB
 In fork processing, during the 64-bit copy phase,
memory objects that are allocated in the system area
will not be copied to the child space



Coupling Facility



Coupling Facility Memory

 Much more static usage than most z/OS LPARs
– Though varying traffic suggests different demands for memory
 “White space” considerations apply
– Duplexing obviously lowers the requirement
 Main categories:
– Dump
– Structures
• Structure sizing and management is an important topic
– Available
 Instrumentation:
– Type 74 Subtype 4
– (Type 70 Subtype 1 just gives memory at ICF LPAR level)
– Exploiter-level statistics



Coupling Facility Memory - Structures
 Each structure type has different memory management considerations
– e.g “False Lock Contention” avoidance in lock structures
– Not just structure type but exploiter
• e.g how VSAM RLS Cache works vs DB2 Group Buffer Pools

 Structures occasionally need to increase in size when upgrading to a
later CFLEVEL
 Use CFSIZER website to size CF structures
– http://www.ibm.com/systems/z/cfsizer/
– Most IBM Product structures catered for
• Given a product name it tells you which structures you want
– And suggests a size
• Produces an “initial” configuration
– For example DB2 structures are likely to need tuning
> In fact CFSIZER probably suggests to you too few GBPs
• Has good Help



Coupling Facility Memory – Structures ...
 4 Potential sizes:
– Current size - R744SSIZ
– Maximum size (MAXSIZE) - R744SMAS
– Initial size (INITSIZE) - R744QSIZ
– Minimum Size (MINSIZE) - R744SMIS
 AUTOALTER can change current size
– For all structure types
– e.g Directory Entry / Data Element ratio for cache structures
• Increases the size of the portion that's constrained
 AUTOALTER is NOT perceived as a “workload spike” handler
– It's really for gradual workload growth
 Current usage and limits for contents of structure in 74-4
– Also Lock Structure Contention / False Contention numbers
– True understanding requires exploiter instrumentation
• e.g DB2 Statistics and Accounting traces



Subsystems and Applications



DB2



DB2
 DBM1 Address Space is biggest user of real memory
– IRLM, MSTR and DIST address spaces generally smaller
• Rare to see virtual storage issues with these 3
 Main DBM1 virtual storage usage in 2 areas:
– Buffer Pools
• In 64-Bit Virtual in V8
• WLM Automatic Buffer Pool Management in V9
– Thread-Related Storage
• Used for callers of DB2 services i.e. applications
• In 31-Bit Virtual in V8
 Other areas generally smaller
– But not always
• e.g. EDM Pool
– Sizes usually related to applications
• But not linear with concurrent threads
– e.g. Prepared Statement Cache, RID Pool, Sort Pool



DB2 Memory Effectiveness Instrumentation

 SMF 100 Records
– Give virtual memory view of buffer pools
• Virtual pools, dataspace pools (and hiperpools)
– And thresholds
• All these are variable using ALTER BUFFERPOOL command
• Effectiveness of buffer pools
• Group Buffer Pools
– Reinforced by SMF 101 application-level reporting
– Some other areas
• e.g Logging, EDM Pool



DB2 Virtual Storage Instrumentation
 IFCID 225 breaks down into eg
– Long-Term used
• eg Buffer pools
– Used unrelated to threads
– Thread related storage
• All threads
– Answers general questions about scalability
– Actual groupings are GETMAINed, Variable, Stack and Fixed
– Also gives REAL memory usage
 IFCID 225 is enhanced significantly in DB2 Versions 8, 9 and 10
– For example to show real memory usage
• Very recent APAR in V10 to show real memory above the Bar
 IFCID 217 allows you to break down thread-related storage into
– eg Blue, Red and Green threads
– Potentially answers questions about scalability based on detailed thread growth "by
colour"
 Lots of tuning knobs for DB2 virtual storage



Client H DB2 Virtual Storage
SYSA Subsystem DB2A July 14, 2004

1200

1000

800
MB

600

400

200

0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Hour
Virtual Pools EDM Pool Free EDM Pool Used GBP Castout Comp Dicts Other GETMAINed
Agt Local Non-Sys Ag Local Sys Pipe Mgr RDS Ops RID Pool PSC Control Blocks
PSC Statements Trace Tables Other Variable Fixed Stack



Client H DB2 DBM1 Thread Memory - Headline Numbers
SYSA Subsystem DB2A

700 800
600 700

500 600
500

Threads
400
MB

400
300
300
200 200
100 100
0 0
Hour

Thread Storage Threads



DB2 Version 8 Virtual Storage



64-Bit Virtual Storage Constraint Relief in DB2 Version 8
 Most of thread storage stays below the 2GB bar
– Implies you still need to manage threads
– Experiences suggest some growth in thread-related storage
 IRLM also is 64 bit
– Allows many more concurrent locks
– PC=NO is no longer supported
• ECSA usage is no longer possible
• PC instruction used instead to access IRLM address space
 Expect to see significant growth in real memory usage
– As subsystems are now able to grow
• Larger buffer pools
• Larger areas such as Prepared Statement Cache
• Perhaps more threads
– As long-term page fixing is available as an option
• Pain if even a single page is not backed by real memory
– As DB2 Version 8 requires more real memory anyhow



64-Bit Virtual Storage Constraint Relief in DB2 Version 9
 EDM Pool:
– SKPT and SKCT sections moved above the bar
– CT and PT sections split between above and below the bar
– Hash tables and fixed pools moved above the bar
– BUT what remains is not subject to LRU algorithm
• So EDM Pool MUST be sized for the peak usage and then some
 Dynamic Statement Cache:
– Considerable movement of control blocks to above the bar
 DBM1 Internal Monitor:
– Reports with console messages when thresholds reached
 DDF:
– Uses 128GB Memory Object instead of ECSA
• Always created at DB2 startup
• All DB2 address spaces are registered to use it
• Eliminates most data moves
• Reduces total storage usage as only 1 copy
• Reduces ECSA usage
• Eases use of larger communications buffers
 Utilities use Shared Memory Objects
– Avoids the CPU to move rows between the Batch Utility and DBM1 Address Spaces



DB2 Long-Term Page Fixing
 For each I/O the buffers must be page-fixed and –freed
– Expensive in CPU terms
 V8 allows LONG TERM page fixing by pool
– PGFIX(YES)
– Can save significant CPU
• Most especially for high I/O rate buffer pools
• At the other end of the scale – 100% hits – maybe LRU algorithm good
 In V9 Group Buffer Pool castout buffers are ALWAYS long term page fixed
– Separate area from buffer pools
– If high castout I/O rate could be a significant saving
 IMPORTANT: Long term page fixing MUST be considered in the light of system
conditions:
– For example, don’t page fix to the point of causing other memory users to
page to death
• Especially important for z/OS Release 8 and beyond



DB2 10
 Vast majority of thread storage moved above The Bar
–Expectation of 5 to 10 times as many threads supportable
–Might allow coalescing of DB2 subsystems
• But caution on the other reasons for maintaining the current number
–Might allow other things
• Such as RELEASE(DEALLOCATE) to cut CPU
–Some of working storage (some of stack, xproc) remains below The Bar

=> Focus shifts to REAL memory
 Reworking of Dynamic SQL Prepared Statement Cache could alter the basis for its
sizing
–Two statements which are identical but with DIFFERENT literal values are now
treated as the same
• One copy in the cache
• More hits per cached copy
–Might allow a reduced cache
• Equally, might make Prepared Statement Cache more worth doing
–Still doesn't obviate the benefit of “keeping prepared statements beyond Commit”
NOTE: MAXKEEPD could be bigger because area moved above The Bar


DB2 10 ...

 In-Memory Pagesets
– Data read into buffer pool at startup
– Useful for small lookup tables
 1 MB Page Size
– Requires long-term page-fixing the buffer pools



Conclusion
 Virtual and Real memory and the paging subsystem are very much worth managing
– For capacity planning
– For resource optimisation
– Because installations can still face constraints
– Because running out of space can cause an outage
• Whether real or virtual memory or paging space
 Consider your stance on keeping memory free for e.g. dump capture
 There’s lots of instrumentation to help you
 Product capabilities are continuing to evolve
– 2007 saw DB2 Version 9, IMS Version 10 and CICS TS 3.2
– 2008 has seen System z10 EC, z/OS Release 10, WAS V7 and MQ V7
– 2009 saw IMS Version 11 and CICS TS 4.1 and z/OS Release 11
– 2010 saw DB2 10 and z/OS Release 12
 It’s important to look at the system, the subsystem & the application all together
– And at the machine level wouldn’t be a bad idea, either
– Applications drive subsystem memory demand
• But supply comes from the system

Memory Matters in 2011

Recommended

Recommended

More Related Content

What's hot

What's hot (10)

Similar to Memory Matters in 2011

Similar to Memory Matters in 2011 (20)

More from Martin Packer

More from Martin Packer (15)

Memory Matters in 2011