Memory Analysis in OS/2

Author: Greg Shah
Date: April 18, 2000

What is Theseus?

Theseus is a tool provided by IBM for the detailed investigation and analysis of memory usage in the OS/2 operating system.  This tool is THE premier tool for memory analysis available.  There are no substitutes.

As documented in the "Introduction" section of THESEUS3.INF:

 THESEUS3* is a memory analysis program for the OS/2 WARP* 3.x, 4.x, and e-Business server systems to:

    1. Determine the amount of memory consumed by each active process.
         a) Memory leak detection.
         b) Working set analysis.
         c) Memory utilization.
    2. Look at application memory (in hexadecimal notation)
         a) In hexadecimal byte notation.
         b) In hexadecimal word notation (2 bytes).
         c) In hexadecimal double word notation ( 4 bytes ).
         d) In unassembled notation.
    3. Look at selected OS/2 system control blocks (formatted)
    4. Look at selected machine registers and control blocks (formatted)

This tool is invaluable to application developers and system integrators alike.  It can be used for problem determination, system tuning and many other purposes.

Obtaining Theseus

As of April 5, 2000, Theseus 3.002 is the current version and is available from IBM's FTP site.

Cost

Theseus is provided by IBM at no charge.

How Does It Work?

OS/2 leverages the Intel CPU protection model to ensure system stability, integrity and security.  This model is based on a 4 level categorization scheme where each increased level of authorization is a superset of the next level or "ring".  There are 4 rings (0-3).

Ring 3 is the traditional application level and in this level each process is highly protected from other processes of the same level.  In addition, ring 3 processes are limited in their ability to directly access hardware.  All access must be made through established interfaces (APIs) which allow a ring transition to code at a more priviledged ring to handle the request on behalf of the ring 3 process.

Ring 0 is the kernel or supervisor level and at this level processes have full access to the entire system and all of its resources.  This is where kernel "plug-ins" or device drivers can be added which provide new services to the system.

Theseus is comprised of a ring 3 executible (THESEUS3.EXE) that is a PM program that reads, interprets and displays the most essential kernel data structures related to processes/modules and memory management.   Theseus provides a device driver (THESEUS5.SYS) that utilizes the access capability of a ring 0 process to provide the EXE with this information. In addition, this device driver provides read-only access to every byte of physical memory.

Installation/Running

Installation is quite simple and well documented in the associated README.
  1. create a directory in which to store Theseus
  2. download and unzip the self-extracting ZIP THES3002.EXE to the newly created directory
  3. DEVICE=X:\THESEUS\THESEUS5.SYS is added to CONFIG.SYS (where X: is the drive on which Theseus is installed)
  4. run X:\THESEUS\THESEUS3.EXE to launch the ring 3 PM application

Viewing System Information

Process Hierarchy - this is the main window of Theseus

System -> General System -> General System Information
System -> General System -> Global Info Seg
System -> General System -> Device Drivers

System -> General System -> Open Files

System -> RAM Usage by Process

System -> Swapper

Viewing Advanced System Information

System -> Page Contents

System -> General System -> GDT

Registers -> Current IDT

Viewing Process Information

Process -> General Process Information

Note that a great deal of information is available on a per process basis, including information and the state of every thread.  In addition, all open files, the environment space and even the command line for the executible can be seen.

Process -> Page Table

Then highlight a linear address and use "Functions -> Show Memory at Linear Address in Hex...", highlight "Show the memory as disassembled instructions" and choose 32-bit or 16-bit.  The following is a 16-bit disassembly of the linear address 00010000 of CMD.EXE:

To look at the kernel or device drivers, it is necessary to look at physical memory.  One can find a "starting point" by:

  1. looking at "System -> General System -> Device Drivers", highlighting the strategy entry point for the module in question and then using "Functions -> Show Memory at Virtual Address in Hex..."
  2. looking at "System -> General System -> Modules", hyperlinking to MTE entry for the module in question and then highlighting the linear address in the Object Table for the object of interest and using "Functions -> Show Memory at Linear Address in Hex..."
  3. looking at the Registers -> Current IDT and tracing from the virtual address specified by the selector and offset for the specific IDT entry of interest
  4. looking at "System -> General System -> System Anchor Segment", hyperlinking to MTE entry for "kernel MTE records" and then highlighting the linear address in the Object Table for the object of interest and using "Functions -> Show Memory at Linear Address in Hex..."

Snooping Around

Memory at any location in the system can be viewed in a HEX dump format.  It is important to know the type of address before trying to display the contents of memory.  A more detailed discussion of the possible types of memory is outside the scope of this presentation.

One good example of this feature is to display the SMBIOS (System Management BIOS) data area which is usually located in the page of memory that starts at the physical address 000F0000.  This data area can be identified by the 4 byte character string _SM_. On my system, this string can be found at 000F6CE0 and it extends for roughly 2375 bytes (0x947).  See this link for a display of the data that is available.

Memory Leak Analysis

A memory leak occurs when an application or system module allocates an ever increasing amount of memory without ever deallocating this memory.  In other words, it occurs when a program loses track of or miscalculates its memory allocations.  Normally programs will deallocate memory at known times, to free this often constrained resource for the system.  With a memory leak, more memory is used than required and in extreme cases, this can result in consuming all of the available RAM of the system.  When the system gets into this state, it is called "overcommitted" and the system's performance and reliability will suffer.

This is especially troublesome when the memory leak is in a process which is "long running", like a server process.  The only way to "clear" this condition is to exit or kill the process in question.  If the module is running at ring 0, then a reboot is necessary since these modules can't be stopped otherwise.  When a process exits, OS/2 deallocates all memory associated with that process, whether the process asked for this explicitly or not.

For the purposes of this presentation, a very simplistic C program called HOG.EXE has been created.  Here is the C source code.  This program will allocate memory on a periodic basis until a CTRL-C is used to kill the program.  The EXE can be obtained via FTP.

HOG 4096 1

This command line will allocate a 4Kb page of memory every second.  Once this is started, one can use the following Theseus reports/features to identify this "leaking" process.

To look at the system as a whole, use "System -> Memory Leak Detection" then turn on "Functions -> Start Periodic Leak Detection".  This will generate a report every 30 seconds, of the incremental pages of memory per process that have been allocated or deallocated since the leak detection was started.  The following is some sample output from this report while running HOG.EXE:

Notice that HOG.EXE is the only application making allocations of memory of any size.  This is a VERY SIMPLISTIC example and if there are memory intesive applications on this machine, they may be alloacting many more pages of RAM than shown here.  The idea with this analysis is to watch the system over a long period of time and discern significant changes in the usage of memory.

Once a possible application has been isolated as a problem, the process can be selected in the main "Process Hierarchy" window and the Process -> Memory Leak Detection can be used to watch this process with greater precision/detail.  These basic tools are useful for detecting leaks in ring 3 processes, however if the leak is in a shared module (a DLL used by many processes) or if it is in a module that is running at ring 0, then a leak will be harder to isolate.  In this case, other reports will need to be viewed over a period of time, with the intention of isolating a process that is always using an ever increasing amount of memory.

Working Set Analysis

The "working set" of a machine is an estimate of the "high water mark" in terms of amount of memory required for this machine to function adequately.  This means that all of the application and system function which this machine is to support needs to be executed while Theseus is peridoically measuring the total memory requirements.  This will generate a worst case scenario of memory required and can be used to tune or design an optimal environment.

There are several reports which can provide a high level overview of an application or system's requirements, however these reports are "point in time" snapshots of the current results, rather than a measurement of the maximum values encountered.  These reports include:

  1. System -> RAM by Usage by Process
  2. System -> Swapper
  3. System -> Nonswappable Memory Analysis
However, Theseus does provide a working set measurement facility (System -> Working Set).    After opening this window, use "Functions -> Working Set Start" to begin the measurement interval.  Then operate the system as expected under a production scenario.  It is most important to simulate the most memory intensive operations and functions to ensure that the analysis is as accurate as possible.  The following is an example report:

If only a single application's working set is in question OR if the working set of the machine is too large for the machine's resources, then the "Process -> Working Set" analysis feature can be used.  In this case, the user will need to manually aggregate the results if a systemwide working set is in question.

Known Problems

These are some problems we have encountered, it is in no way an "official list" of bugs.
  1. "Detail" windows don't display in the "Window List"
  2. This version does not work on Warp v3 with Fixpak 38.

References

  1. The Design of OS/2 by H.M. Deitel and M.S. Kogan, Addison-Wesley 1992, ISBN 0-201-54889-5
  2. The OS/2 Debugging Handbook - Volume IV, System Diagnostic Reference by IBM International Technical Support Organization (ITSO), February 1996, IBM pub SG24-4643
  3. Using THESEUS3, Theseus product documentation included with the distribution as THESEUS3.INF.