This chapter helps you estimate disk and memory requirements. This chapter contains the following sections:
Note: If you are migrating from an earlier version of Essbase, see the Essbase Installation Guide for additional information about estimating space requirements.
This chapter uses a worksheet approach to help you keep track of the many components that you calculate. If you are using the printed version of this book, you can photocopy the worksheets. Otherwise, you can simulate the worksheets on your own paper. Labels, such as DA and MA help you keep track of the various calculated disk and memory component values,
You need to understand the units of storage that Essbase uses in order to size a database. This discussion assumes that you are familiar with the following basic concepts before you continue:
An Essbase database consists of many different components. In addition to an outline file and a data file, Essbase uses several types of files and memory structures to manage data storage, calculation, and retrieval operations.
Table 93 describes the major components that you must consider when you estimate the disk and memory requirements of a database. "Yes" means the type of storage indicated is relevant, "No" means the type of storage is not relevant.
Essbase uses disk space for its server software and for each database. Before estimating disk storage requirements for a database, you must know how many dimensions the database includes, the sparsity and density of the dimensions, the number of members in each dimension, and how many of the members are stored members.
To calculate the disk space required for a database:
Note: The database sizing calculations in this chapter assume an ideal scenario with an optimum database design and unlimited disk space. The amount of space required is difficult to determine precisely because most multidimensional applications are sparse.
Before estimating disk space requirements for a database, you must calculate the factors to be used in calculating the estimate. Later in the chapter you will use these values to calculate the components of a database. For each database, you will then add together the sizes of its components.
Table 94 lists the sections that provide instructions to calculate these factors. Go to the section indicated, perform the calculation, then write the calculated value in the Value column.
Database Sizing Factor |
Label |
Value |
---|---|---|
The potential number of data blocks is the maximum number of data blocks possible in the database.
If the database is already loaded, you can see the potential number of blocks on the Statistics tab of the Database Information dialog box of Application Manager or on the Statistics tab of the Database Properties dialog box of Essbase Administration Services.
If the database is not already loaded, you must calculate the value.
To determine the potential number of data blocks, assume that data values exist for all combinations of stored members.
The following types of members are not stored members:
a * b * c * d * e * f * g (and so on) = potential number of blocks
Enter Sparse Dimension Name |
Enter Number of Stored Members |
|
---|---|---|
The Sample Basic database contains the following sparse dimensions:
Therefore, there are 19 * 25 = 475 potential data blocks.
Number of Existing Data Blocks
As compared with the potential number of blocks, the term existing blocks refers to those data blocks that Essbase actually creates. For Essbase to create a block, at least one value must exist for a combination of stored members from sparse dimensions. Because many combinations can be missing, the number of existing data blocks is usually much less than the potential number of data blocks.
If the database is already loaded, you can see the number of existing blocks on the Statistics tab of the Database Information dialog box of Essbase Application Manager or on the Statistics tab of the Database Properties dialog box of Essbase Administration Services. Write the value in the cell labeled DB in Table 94.
If the database is not already loaded, you must estimate a value.
To estimate the number of existing data blocks:
number of existing blocks = estimated density * potential number of blocks
The following three examples show different levels of sparsity and assume 100,000,000 potential data blocks:
.05 (estimated density) * 100,000,000 (potential blocks) = 5,000,000 existing blocks
.15 (estimated density) * 100,000,000 (potential blocks) = 15,000,000 existing blocks
.50 (estimated density) * 100,000,000 (potential blocks) = 50,000,000 existing blocks
The potential, expanded (uncompressed) size of each data block is based on the number of cells in a block and the number of bytes used for each cell. The number of cells in a block is based on the number of stored members in the dense dimensions. Essbase uses eight bytes to store each intersecting value in a block.
If the database is already loaded, you can see the size of an expanded data block on the Statistics tab of the Database Information dialog box of Application Manager or on the Statistics tab of the Database Properties dialog box of Essbase Administration Services.
If the database is not already loaded, you must estimate the value.
To determine the size of an expanded data block:
The following types of members are not stored members:
a * b * c * d * e * f * g (and so on) = the total number of cells
(Total number of cells) * 8 bytes per cell = expanded block size
Enter Dense Dimension Name |
Number of Stored Members |
|
---|---|---|
The Sample Basic database contains the following dense dimensions:
Perform the following calculations to determine the potential size of a data block in Sample Basic:
12 * 8 * 2 = 192 data cells 192 data cells * 8 bytes = 1,536 bytes (potential data block size)
Compression affects the actual disk space used by a data file. The two types of compression, bitmap and run-length encoding (RLE), affect disk space differently. For information about data compression unrelated to estimating size requirements, see Data Compression.
If you are not using compression or if you have enabled RLE compression, skip this calculation and proceed to Compressed Data Files.
Note: Due to sparsity also existing in the block, actual (compressed) block density varies widely from block to block. The calculations in this discussion are only for estimation purposes.
To calculate an average compressed block size when bitmap compression is enabled:
expanded block size * block density = compressed block size
Assume an expanded block size of 1,536 bytes and a block density of 25%:
1,536 bytes * .25 = 384 bytes (compressed block size)
To estimate the disk-space requirement for a database, make a copy of Table 97 or use a separate sheet of paper as a worksheet for a single database. If multiple databases are on a server, repeat this process for each database. Write the name of the database on the worksheet.
Each row of this worksheet refers to a section that describes how to size that component. Perform each calculation and write the results in the appropriate cell in the Size column. The calculations use the factors that you wrote in Table 94.
Database Name: |
||
---|---|---|
Database Component |
Size |
|
Work Areas (sum of DE through DI) |
||
Linked Reporting Objects Considerations, if needed |
||
Total disk space required for the database. |
After writing all the sizes in the Size column, add them together to determine the disk space requirement for the database. Add the database name and size to the list in Table 99. Table 99 is a worksheet for determining the disk space requirement for all databases on the server.
Repeat this exercise for each database on the server. After estimating disk space for all databases on the server, proceed to Estimating the Total Server Disk Space Requirement.
The following sections describe the calculations to use to estimate components that affect the disk-space requirements of a database.
The calculation for the space required to store the compressed data files (essxxxxx.pag) uses the following factors from Table 94:
To estimate the space required for the compressed data files, multiply the compressed block size by the number of data blocks and write the size of the compressed data files cell labeled DE in Table 97.
Note: If compression is not used, substitute the expanded block size for the compressed block size in this formula.
384 (compressed block size) * 15,000,000 (number of data blocks) = 5,760,000,000 bytes (size of compressed data files)
The following subtopics show how to calculate fixed-size overhead. Use one of two methods of calculation, depending whether the database uses bitmap compression, run-length encoding (RLE), or no compression.
Fixed Size Overhead Using Bitmap Compression
Calculations for fixed-size overhead using bitmap compression use the following factors and constants:
The compression bitmap uses one bit for each cell in a block. Dividing the expanded block size by 8 provides the number of cells, which equals the number of bits in the bitmap. Dividing this value again by 8 determines the number of bytes; therefore the following procedure divides the expanded block size by 64 to obtain the fixed-size overhead for each block.
To calculate the fixed-size overhead when bitmap compression is enabled:
((expanded block size in bytes/64)+72) = temporary value
The result of this calculation is the fixed-size overhead per block.
fixed-size overhead per block * number of existing blocks = fixed-size overhead for the database
Assume bitmap compression and an expanded block size of 4,802 bytes with a total of 15,000,000 existing blocks.
(4,802 / 64) + 72 = 147.03 bytes
152 bytes * 15,000,000 blocks = 2,280,000,000 bytes (database overhead)
Calculation for Rounding Up |
Result |
---|---|
Fixed Size Overhead Using Run-Length Encoding (RLE) or No Compression
Calculations for fixed-size overhead for databases not using bitmap compression use the following factors and constants:
To calculate the fixed-size overhead for a database that uses RLE or no compression:
72 * Number of existing blocks = Fixed-size overhead for the database
Assume a total of 15,000,000 existing blocks.
72 bytes * 15,000,000 blocks = 1,080,000,000 bytes
The calculation for the space required to store the index files (essxxxxx.ind) uses the following factors:
To calculate the total size of a database index, including all index files. perform the following calculation. Write the size of the compressed data files to the cell labeled DG in Table 97.
number of existing blocks * 112 bytes = the size of database index
Assume a database with 15,000,000 blocks.
15,000,000 blocks * 112 = 1,680,000,000 bytes
Note: If the database is already loaded, select Database > Information in Application Manager and look at the Files tab for the size of the index file. If you are using Essbase Administration Services, click the Storage tab on the Database Properties window.
If you are using bitmap or RLE compression, a certain amount of fragmentation occurs. The amount of fragmentation is based on individual database and operating system configurations and cannot be precisely predicted.
As a rough estimate, calculate 20% of the compressed database size (value DE from Table 97) and write the result to the cell labeled DH in the same table.
Assume a compressed database size of 5,769,000,000 bytes.
5,769,000,000 bytes * .2 = 1,153,800,000 bytes
The space required by an outline can have two components.
To estimate the size of the outline:
If the database includes few aliases or very short aliases and short member names, use a smaller number within this range. If you know that the names or aliases are very long, use a larger number within this range.
Because the name-length factor is an estimated average, the following formula provides only a rough estimate of the main area of the outline.
number of members * name-length factor = size of main area of outlineNote: See Limits, for the maximum sizes for member names and aliases.
For memory space requirements calculated later in this chapter, use the size of the main area of the outline.
Note: Within the count of members, do not include Label Only members and shared members.
(number of base-dimension members * sum of count of attribute-dimension members)/8 = size of attribute association area for a base dimension
main area of outline + total attribute association area = total disk space required for the outline
Assume the outline has the following characteristics:
400 bytes (name-length factor) x 26,000 members = 10,400,000 bytes
(23,000 * (20 + 50)) bits / 8 bits per byte = 201,250 bytes
(2,500 * 12) bits / 8 bits per byte = 3,750 bytes
201,250 bytes + 3,750 bytes = 205,000 bytes
10,400,000 bytes + 205,000 bytes = 10,605,000 bytes (outline disk space requirement)
Note: Do not use this procedure to calculate outline memory space requirements. See The Outline Size Used in Memory.
Three different processes create temporary work areas on the disk:
To create these temporary work areas, Essbase may require disk space equal to the size of the entire database. Restructuring and migration need additional work space the size of the outline. Because none of these activities occur at the same time, a single allocation can represent all three requirements.
To calculate the size of a work area used for restructuring, migration, and recovery, calculate the sum of the sizes of the following database components from Table 97:
work area = size of compressed data files + fixed-size overhead + size of index files + fragmentation allowance + outline size
Write the result of this calculation to the cell labeled DJ in Table 97.
You can use the Linked Reporting Objects (LROs) feature to associate objects with data cells. The objects can be flat files, HTML files, graphics files, and cell notes. For information about linked reporting objects, see Linking Objects to Essbase Data.
Two aspects of LROs affect disk space:
Note: You can set a limit on the size of a linked object, if the linked object is a file (as opposed to a cell note). For information, see Limiting LRO File Sizes for Storage Conservation.
To estimate the disk space requirements for linked reporting objects:
sum of LRO sizes + size of LRO catalog = LRO disk space requirement
Assume the database uses 1500 LROs which are composed composed of:
512,000 bytes + 12,288,000 bytes = 12,800,000 bytes total LRO disk space requirement
The earlier calculations in this chapter estimate the data storage requirement for a single database. Often, more than one database resides on the server.
In addition to the data storage required for each database, the total Essbase data storage requirement on a server includes Essbase software. Allow approximately 80 to 202 MB (84,451,328 to 211,953,664 bytes) for the base installation of Essbase software and sample applications. The allowance varies by platform and file management system. For details, see the Essbase Installation Guide.
To estimate the total server disk space requirement:
List of Databases (From Table 97) |
Size |
|
---|---|---|
Total Essbase server disk requirement in megabytes (MB): DN divided by 1,048,576 bytes |
The minimum memory requirement for running Essbase is 64 MB. On UNIX systems, the minimum requirement is 128 MB. Based on the number of applications and databases and the database operations on the server, the amount of memory you require may be more.
To estimate the memory required on the server:
Each open application has the following memory requirement at startup:
Multiply the number of applications that will be running simultaneously on the server by the appropriate startup requirement and write the resulting value to the cell labeled ML in Table 103.
To estimate the memory requirement for a database, make a copy of Table 100 or use a separate sheet of paper as a worksheet for a single database. If multiple databases are on a server, repeat this process for each database. Write the name of the database on the worksheet.
Each row links to information that describes how to size that component. Perform each calculation and note the results in the appropriate cell in the Size column. Some calculations use the factors that you wrote in Table 101. After filling in all the sizes in the Size column, add them together to determine the memory requirement for that database.
After estimating disk space for all databases on the server, proceed to Estimating the Total Server Disk Space Requirement.
Database Name: |
||
---|---|---|
Memory Requirement |
Size |
|
Database outline. See The Outline Size Used in Memory. |
||
Index cache. See Sizing the Index Cache. |
||
Cache-related overhead. See Cache-Related Overhead. |
||
Area for data structures. See Memory Area for Data Structures. |
||
Memory used for data retrievals. See Estimating Additional Memory Requirements for Data Retrievals |
||
Memory used for calculations. See Estimating Additional Memory Requirements for Calculations |
||
Summarize the size values from MB through MF for an estimate of the total memory required for a database. |
||
Divide the value from MG. by 1,048,576 bytes for the total database memory requirement in megabytes (MB). |
In Table 103, enter the name of the database and the total memory requirement in megabytes, MH.
Before you start the estimate, calculate factors to be used in calculating the estimate.
Table 101 lists sizing factors with references to sections in this and other chapters that provide information to determine these sizes. Go to the section indicated, perform the calculation, then return to Table 101 and write the size, in bytes, in the Value column of this table.
Later in this chapter, you can refer to Table 101 for values to use in various calculations.
Database Sizing Factor |
Value |
|
---|---|---|
The number of cells in a logical block. See The Number of Cells in a Logical Block. |
||
The number of threads allocated through the ESSCMD, SERVERTHREADS. See the Technical Reference in the docs directory. |
||
Potential stored-block size. See Size of Expanded Data Block. |
The calculations in this chapter do not account for other factors that affect how much memory is used. The following factors have complex implications and their affects on memory size cannot be calculated:
The attribute association area included in disk space calculations is not a sizing factor for memory. Calculate only the main area of the outline.
For memory size requirements, outline size is calculated using the following factors:
To calculate the outline memory requirement, multiply the number of members by a name-length factor between 350 and 450 bytes and write the result to the cell labeled MA in Table 100.
If the database includes few aliases or very short aliases and short member names, use a smaller number within the 350-450 byte range. If you know that the names or aliases are very long, use a larger number within this range.
Because the name-length factor is an estimated average, the following formula provides only a rough estimate of the main area of the outline:
memory size of outline = number of members * name-length factor
Note: See Limits, for the maximum sizes for member names and aliases.
Assuming the outline has 26,000 members and a median name-length, use the following calculation to estimate the outline size used in memory:
26,000 members * 400 bytes per member = 10,400,000 bytes
At startup, Essbase sets aside memory for the index cache, the size of which can be user-specified. To determine the size of the index cache, see Sizing the Index Cache and write the size in the cell labeled MB in Table 100.
Essbase uses additional memory while it works with the caches.
The calculation for this cache-related overhead uses the following factors:
To calculate the cache-related overhead at startup:
index cache size * .5 = index cache-related overhead
((# of server threads allocated to the Essbase server process * 3) * 256) + 5242880 bytes = additional cache overhead
cache-related overhead = index cache-related overhead + additional cache overhead
The term logical block applies to an expanded block in memory.
To determine the cell count of a logical block, multiply together all members of each dense dimension (including Dynamic Calc and Dynamic Calc And Store members but excluding Label Only and shared members).
a * b * c * d * e * f * g = the total number of cells
Enter Dense Dimension Name |
Number of Members |
|
---|---|---|
Excluding Label Only and shared members, the dense dimensions in Sample Basic contain 17 (Year), 14 (Measures), and 4 (Scenario) members. The calculation for the cell count of a logical block in Sample Basic is:
17 * 14 * 4 = 952 cells
At application startup time, Essbase sets aside an area of memory based on the following factors:
To calculate the data structure area in memory:
Number of threads * ((Number of members in the outline * 26 bytes) + (Logical block cell count * 36 bytes))
Assuming 20 threads for the Sample Basic database, the startup area in memory required for data structures is calculated as follows:
20 threads * ((79 members * 26 bytes) + (952 cells * 36 bytes)) = 726,520 bytes 726,520 bytes / 1,048,576 bytes = .7 MB
In addition to startup memory requirements, operations such as queries and calculations require additional memory. Because of many variables, the only way to estimate memory requirements of operations is to run sample operations and monitor the amount of memory used during these operations.
Essbase processes requests for database information (queries) from a variety of sources. For example, Essbase processes queries from the Spreadsheet Add-in and from Report Writer. Essbase uses additional memory when it retrieves the data for these queries, especially when Essbase must perform dynamic calculations to retrieve the data. This section describes Essbase memory requirements for query processing.
Essbase is a multithreaded application in which queries get assigned to threads. Threads are automatically created when Essbase is started. In general, a thread exists until you shut down OLAP Server (for more information, see Running Essbase Servers, Applications, and Databases).
As Essbase processes queries, it cycles through the available threads. For example, assume 20 threads are available at startup. As each query is processed, Essbase assigns each succeeding query to the next sequential thread. After it has assigned the 20th thread, Essbase cycles back to the beginning, assigning the 21st query to the first thread.
While processing a query, a thread allocates some memory, and then releases most of it when the query is completed. Some of the memory is released to the operating system and some of it is released to the dynamic calculator cache for the database being used. However, the thread holds on to a portion of the memory for possible use in processing subsequent queries. As a result, after a thread has processed its first query, the memory held by the thread is greater than it was when Essbase first started.
Essbase uses the maximum amount of memory for query processing when both of these conditions are true:
In the example where 20 threads are available at startup, the maximum amount of memory is used for queries when at least 20 queries have been processed and the maximum number of simultaneous queries are in process.
Calculating the Maximum Amount of Additional Memory Required
To estimate query memory requirements by observing actual queries:
Use the following variables when you calculate the formula in Estimating the Maximum Memory Usage for A Query Before and After Processing:
Determining the Total Number of Threads
The potential number of threads available is based on the number of licensed ports that are purchased. The actual number of threads available depends on settings you define for the Agent or the server. Use the number of threads on the system as the value for Total#Threads in later calculations.
Estimating the Maximum Number of Concurrent Queries
Determine the maximum number of concurrent queries and use this value for Max#ConcQueries in later calculations. This value cannot exceed the value for Total#Threads.
Estimating the Maximum Memory Usage for A Query Before and After Processing
The memory usage of individual queries depends on the size of each query and the number of data blocks that Essbase needs to access to process each query. To estimate the memory usage, calculate the additional memory Essbase uses during processing and after processing each query.
Decide on several queries that you expect to use the most memory. Consider queries that must process large numbers of members; for example, queries that perform range or rank processing.
To estimate the memory usage of a query:
Use this value for MemBeforeP.
Use this value for MemDuringP.
AdditionalMemDuringP = MemDuringP - MemBeforeP
The amount of additional memory required for data retrievals will not exceed:
Max#ConcQueries * MAXAdditionalMemDuringP + (Total#Threads - Max#ConcQueries) * MAXAdditionalMemAfterP
Write the result of this calculation, in bytes, to the cell labeled ME in Table 100.
Because this calculation method assumes that all of the concurrent queries are maximum-sized queries, the result may exceed your actual requirement. It is difficult to estimate the actual types of queries that will be run concurrently.
To adjust the memory used during queries, you can set values for the retrieval buffer and the retrieval sort buffer. For information, see Setting the Retrieval Buffer Size and Setting the Retrieval Sort Buffer Size.
If you cannot perform this test with actual queries, you can calculate a very rough estimate for the operational requirement of a query by summarizing the following values and multiplying the sum by the maximum number of possible concurrent queries:
To estimate the maximum memory needed for concurrent queries, assume the following values:
100 blocks * 7616 bytes = 761,600 bytes
Estimated memory for retrievals:
20 * (10,240 bytes + 10,240 bytes + 761,600 bytes) = 15,641,600 bytes
For existing calculation scripts, you can use the memory monitoring tools provided for the operating system on the server to observe memory usage. Run the most complex calculation and take note of the memory usage both before and while running the calculation. Calculate the difference and use that figure as the additional memory requirement for the calculation script.
To understand calculation performance, see Optimizing Calculations.
If you cannot perform a test with a calculation script, you can calculate a very rough estimate for the operational requirement of a calculation by adding together the following values:
For the total calculation requirement, summarize the amount of memory needed for all calculations that will be run simultaneously and write that total to the cell labeled MF in Table 100.
Note: The size and complexity of the calculation scripts affect the amount of memory required. The effects are difficult to estimate.
You can use Table 103 as a worksheet on which to calculate an estimate of the total memory required on the server.
Component |
Memory Required, in Megabytes (MB) |
|
---|---|---|
Sum of application startup memory requirements (see Startup Memory Requirement for an Application) |
||
In rows a through g below, list concurrent databases (from copies of Table 100) and enter their respective memory requirements (MH) in the column to the right. |
||
To estimate the total Essbase memory requirement on a server:
If cache memory locking is enabled, the total memory requirement should not exceed two-thirds of available RAM; otherwise, system performance can be severely degraded. If cache memory locking is disabled, the total memory requirement should not exceed available RAM.
If there is insufficient memory available, you can redefine your cache settings and recalculate the memory requirements. This can be an iterative process. For guidelines, see Fine Tuning Cache Settings. In some cases, you may need to purchase additional RAM.
![]() © 2002 Hyperion Solutions Corporation. All rights reserved. http://www.hyperion.com |