| OSF DCE SIG | R. Friedrich (HP) | |
| Request For Comments: 32.0 | June 1993 |
Managing applications in a distributed environment is a complex task. The DCE Serviceability SIG has published a requirements document that focuses on isolating, diagnosing and correcting error conditions [Hubb]. The DME Managed Objects SIG has specified the requirements for managed objects [Sear]. However, performance related tasks such as configuration planning, application tuning, bottleneck analysis, and capacity planning require supplementary data. These performance measurement capabilities are a necessary component of any commercially viable computer technology.
This RFC represents work done by the DCE Performance SIG. It complements the DCE and DME management working groups by proposing the requirements and capabilities to monitor the performance of a DCE cell. To accomplish this we discuss the requirements for performance monitoring, the metrics that are of interest to performance analysts, and the instrumentation necessary to collect performance data.
The guiding philosophy is to define a set of standardized performance instrumentation that can be consistently collected, reported and interpreted in a heterogeneous environment. Furthermore, the proposed measurement system should be compiled into the core DCE services for use at customer sites. This requires strict limits on acceptable overhead.
A companion RFC, RFC 33.0, discusses a proposed measurement system architecture and design guidelines for the collection, analysis and presentation of performance data [Frie].
This proposal focuses on the DCE RPC and CDS services. These have been determined to be the most important minimal subset for performance monitoring. (The security service is also a critical component, but due to its evolution, we have delayed addressing it.) A future work group proposal will extend performance monitoring capability requirements to all DCE services.
We also propose a generic server performance interface (GSPI) that application developers can use to automatically compile performance instrumentation into their client and server source.
The working group's goal is to establish requirements so that OSF can begin deployment of core instrumentation with the DCE Release 1.1.* and then roll out additional instrumentation in later releases.
The following list summarizes the minimum content for DCE release 1.1.*:
Develop rpc_msmt routines that include data on response times,
service requirements, queue lengths, bytes transferred, messages rejected
(errors), etc.
Develop cds_msmt routines that include data on response time, service
requirements, queue lengths, type of entry accessed (group, profile), etc.
Develop/extend IDL and run time libraries to support automatic performance instrumentation in application servers. See section 5.7 for details.
Develop a consistent interface for collecting key host OS performance metrics (such as node-wide CPU utilization, disk I/O rates, network traffic) so that it can be reported through the same presentation services as the DCE metrics.
Create a user guide for collecting and reporting performance data. Create an application developer guide on instrumenting application servers.
To ensure consistent meaning the following terms are defined for use in this document.
Metrics specify those measurable quantities that form the criteria to evaluate the performance of a system under study. They can be raw information (such as events) or derived quantities such as statistical measures.
Instrumentation are specialized software components incorporated into programs to provide mechanisms for measuring events and calculating the relevant performance metrics. The basic measurement techniques are counting, sampling and tracing. The objective of instrumentation is to provide measures of resource utilization (such as CPU, memory, I/O, network, etc.) and processing time (such as service time, queuing time, etc.) that can be analyzed as frequency and time histograms, or delivered as statistical measures.
The type of instrumentation that we propose utilize high-level procedure call statements and macros that can be incorporated into the specific DCE services at strategic locations. This is supplemented with IDL enhancements to capture client and generic server performance data through automatically created performance interfaces.
A process that provides on-going collection and reporting of performance data for evaluation by system manager, application designers and capacity planners.
A software process/thread executing on a particular network node. A DCE client is usually a custom developed application that issues RPC requests to a DCE server but can also be a core DCE service making a request of another DCE server.
A software process/thread executing on a particular network node. A DCE
server can be a core DCE service (such as the rpcd) or a
custom developed
service. For the purpose of this paper the hardware server will be
referred to as a network node.
Performance instrumentation collects and displays performance information to the cell administrator, system administrator, application designer, and the system designer.
Sufficient capabilities must exist on any viable commercial computer system to answer the many performance related questions that arise during operation:
Performance instrumentation and monitors are necessary to avoid, identify and resolve performance bottlenecks. Users cannot succeed with the deployment of a new technology without the ability to monitor and manage their new applications. Furthermore, it should be emphasized that:
The following summarizes the motivation for performance instrumentation:
Without this data it will be difficult to deploy and manage DCE-based applications.
Our initial work will focus on identifying required metrics and instrumentation for RPC and CDS. Future activities will expand to include identifying where instrumentation should be placed (this will likely require close support from OSF and technology suppliers) and the design of the actual instrumentation (algorithms and data structures) [Frie].
As we investigated the need and requirements for DCE performance instrumentation we discovered that there exists several related activities. How we incorporated their requirements are discussed for each:
The following items are not specifically addressed by this requirements document.
We have identified the following requirements for the successful design of standardized instrumentation and a measurement system.
The following general requirements should be satisfied by the measurement system.
This section describes the users and uses of performance data to ensure that the metrics and architecture are satisfactory. The users and uses of performance data are summarized in the following list(s). (Though presented here in list format, this data can best be conceptualized as a table or matrix, with columns tagged with the five named user types and rows tagged with the the five named use types.)
The five major users of performance data are: (a) the platform vendor who provides the DCE infrastructure on a particular platform; (b) the third party/in-house application developer; (c) the cell system administrator; (d) the cell capacity planner; (e) the information technology decision maker. For each of these users we have summarized the major attributes of performance data: (i) the user function; (ii) estimated factors; (iii) measurable factors; (iv) actions that can be taken based on the data; (v) the degrees of freedom that the particular user has in using this data to improve the productivity of the computing environment.
Isolating performance problems in a distributed environment is much more difficult than in a centralized environment. It is important that the measurement system provide data from the client's perspective to approximate the end users perception of system performance. This provides insights into why a client's transaction or operation is not meeting performance expectations.
From the server's perspective requests appear as an incoming stream and replies signify completions (throughput). Monitoring from a server's perspective also provides node utilization.
Having information from both the client and server perspectives can also provide an indication about the performance of the interconnect.
To maximize the value of the measurement system it is required that application developers be provided with the capability to easily create generic performance management interfaces for application servers. Predefined performance metrics should be available by including the appropriate header file and custom, extensible application defined metrics should also be supported. These interfaces should be accessible from the standard measurement system and data for these application servers should be combined and displayed through the measurement system presentation services.
Requirements for different data capture granularities and subsets requires that the maximum amount of data is collected with the minimum amount of overhead. Consequently, we have defined varying data collection intensity levels to provide increasing detail in the collected data.
Under the best scenario there should be no overhead incurred by the measurement system when there are no observations required. This can be approached practically if the design includes a zero intensity level.
Increasing the observation intensity level allows more detailed data to be collected which results in increasing complexity of metrics and collected data. In general, increasing intensity levels provide a superset of statistical data and metrics.
The following list specifies the five data collection intensity levels and their characteristics.
Level 4 functions should be provided by hooking into the tracing services defined by the DCE serviceability group [Hubb].
Support concurrent requests of the measurement system with different configurations and sampling intervals.
Enable/disable this instrumentation on a running system without disrupting an active application environment.
Provide a closed feedback loop so that changes to system configuration parameters are reflected in the data collected by the measurement system.
A performance measurement system, although not a system management service in and of itself, is an important aspect of any system management capability. Therefore, the measurement system should wherever possible:
Provide configurable data collection overhead to minimize impact on system resources. Both the measurement interval and the granularity of the collected data should be configurable. The CPU overhead goals of the measurement system are listed below based on the amount of data that has been requested (specific memory, disk and network resource consumption goals have not been identified):
The measurement system should work within the framework of the DCE security system.
Establish a validation suite so that the integrity of the measurement system can be confirmed and maintained for each platform and release.
This section describes and establishes a set of prioritized metrics that will guide the design and placement of performance instrumentation. Performance metrics are provided for a client perspective (end user) and for a server perspective.
This section also describes the statistics that the measurement system must produce.
The following metrics define the quantities and the notation that will be used throughout the remainder of the document. These metrics have been derived from [Lazo] and [Jain]. The notation is derived from [Lazo].
In general, a metric with an annotation of k is for a particular resource k. Non-annotated metrics are for the system as a whole. The above non-annotated metrics can also be defined for a particular resource. For example, \*(lL\*(kK is the arrival rate of requests at resource k.
The following metrics should be collected or derived from a client perspective:
The following metrics should be collected or derived from a server perspective:
The typical server operation completes normally and that is the expected view when we collect performance information. Three categories of metrics are associated with the outcomes of a requested operation. These categories are termed speed, reliability, and availability. This RFC discusses only those metrics related to speed. The Management SIG might wish to pursue the reliability and availability categories.
The instrumentation must provide analysis software with the data required to compute the following statistical quantities:
These quantities are described in detail in [Alle] and [Jain].
The following sections describe the metrics for the RPC and CDS DCE services. It also provides some guidance to technology providers in how to interpret the metrics so that they can be accurately and consistently implemented. Specific instrumentation design guidelines can be found in [Frie].
For each DCE core service a prioritized list of performance metrics is provided. The metrics were developed and prioritized by the working group and are subject to change based on general DCE SIG comment. The DCE Performance SIG expects that all high priority items will be incorporated into the next available DCE release. Instrumentation for other DCE services, as well as medium and low priority metrics presented here, should be incorporated into a future release based on SIG input.
Based on the above client and server views the following metrics should be collected for each client\*(f!
This implies that clients can be located by the measurement system. The current DCE technology does not provide a mechanism to locate clients -- only servers that register in the CDS.thread and for each server RPC interface. The threads and interfaces must be uniquely identifiable.
For each metric the minimum, maximum, sum, mean, variance, 90th percentile, skew and histograms should be collected based on the collection intensity level.
This measures the total elapsed time, including server processing time and delay/queueing, for a client routine that invokes a particular DCE server.
Measure the elapsed time per RPC call from the time the client's runtime initiates the call until the last packet has been received and unmarshalled. This should include nested RPC call elapsed times if other DCE servers, such as the security service, must be invoked (the nested RPC call should optionally be broken out). Note that this time will not include client application or user interface response time since they are outside the scope of the DCE services.
For a given sampling interval measure the Elapsed Time from the execution
of rpc_call_start to the execution of rpc_call_end
as measured in the client process.
This measures the total elapsed time, including server processing time and delay/queueing, required for the server to satisfy a client request.
Measure the elapsed time per RPC call from the time the server runtime receives the call until the last packet has been marshalled and sent. This should include nested RPC call elapsed times if other DCE servers, such as the security service, must be invoked (the nested RPC call times should optionally be broken out).
For a given sampling interval collect the mean elapsed time from the
execution of rpc_call_start to the execution of
rpc_call_end as measured in the server process.
This measures the service requirement at the server, including operating system and network software processing time, required to satisfy a client's request. This request may consist of multiple RPC packets but only one RPC call. This requires that the host operating system support a performance measurement system and that DCE servers use it to gather CPU service time.
This measures the server's total throughput as determined by the number of successful completions of client requests per unit time.
This measures the frequency of client requests and should be available for each server RPC interface.
For a given sampling interval:
rpc_call_start is executed by the client (initiates
an RPC call).
rpc_call_end is executed by the client (client has
received results).
These two metrics measure the number of packets sent by the client and the number of packets received by the client.
For connectionless RPC count the number of times rpc__dg_xmit_pkt is
called by the client.
For connection oriented RPC count the number of time
rpc_cn_assoc_send_frag is called by the client.
These two metrics measure the number of packets sent by the server and the number of packets received by the server for all clients.
For connectionless RPC count the number of times rpc__dg_xmit_pkt is
called by the server.
For connection oriented RPC count the number of time
rpc_cn_assoc_send_frag is called by the server.
This metric provides information about the size of the data transferred from the client to the server.
This metric provides information about the size of the data transferred from the server to the client.
This information, although not a performance metric, provides insight into the operational environment and whether error conditions may be causing performance problems.
The following metrics were assigned medium priority and are briefly described.
The following metrics were assigned low priority and are briefly described.
In the following example we illustrate the use of these performance metrics. Relevant performance metrics are indicated at the bottom of the figure (client request arrival rate and response time; service demand on the server).
FIGURE 1. Client View of Performance
+----------+ Request-A +------------+
| Client_A | ----------------> | |
| | <---------------- | |
+----------+ Reply | |
| |
| Server_C |
| |
+----------+ Request-B | | Request +----------+
| Client_B | ----------------> | | ---------> | Remote |
| | <---------------- | | <--------- | Server_D |
+----------+ Reply +------------+ Reply +----------+
Metrics:
Arrival Rate Service Demand
Response Time
Two clients, Client_A and Client_B, issue RPC requests to a server, Server_C. From a client's perspective RPC requests are made to a specific server. After some amount of processing and queueing time the server replies to the request. The simple case involving one client and one server is shown as Request-A in Figure 1 where Client_A requests an operation of Server_C.
The actual communication can be more complicated as shown for Request-B. In this example Client_B makes a request of Server_C who in turn must initiate a client request to a remote Server_D in order to complete Request-B. Note that from the client's perspective there is no change in the performance metrics but the measurement system must be able to include the processing and queueing time of the remote server in the total processing and queuing time for Request-B. The measurement system is further complicated if Server_C is multi-threaded because two different threads may process the initial request from Client_B and the completion from Server_D.
The following table provides an example performance report from the client's view. The performance measurement system should be able to provide this summary. The detailed description of each metric is in section 5.5.1.
TABLE 1. Sample Performance Output -- Client View
+---------------------------+----------------+---------------+
| Metric | Client_A | Client_B |
+===========================+================+===============+
| RPC Calls/sec | | |
| Ave | 35.4 | 12.6 |
| Std Dev | 14.2 | 7.5 |
+---------------------------+----------------+---------------+
| RPC Response Time (msec) | | |
| Min / Max | 2.5 / 175.3 | 2.4 / 256.3 |
| Ave / Std Dev | 14.5 / 22.8 | 22.5 / 28.6 |
+---------------------------+----------------+---------------+
| Packets/sec | | |
| Sent | 23.0 | 78.0 |
| Received | 45.0 | 12.0 |
+---------------------------+----------------+---------------+
| KBytes/sec | | |
| Sent | 2.5 | 15.0 |
| Received | 35.0 | 10.0 |
+---------------------------+----------------+---------------+
| tab(@) center box; |
| lb | rb | rb |
| l | r | r. |
| Metric@Client_A@Client_B |
| = |
| RPC Calls/sec@@ |
| Ave@35.4@12.6 |
| Std Dev@14.2@\07.5 |
| _ |
| RPC Response Time (msec)@@ |
| Min / Max@ 02.5 / 175.3@\02.4 / 256.3 |
| Ave / Std Dev@14.5 / \022.8@22.5 / \028.6 |
| _ |
| Packets/sec@@ |
| Sent@23.0@78.0 |
| Received@45.0@12.0 |
| _ |
| KBytes/sec@@ |
| Sent@\02.5@15.0 |
| Received@35.0@10.0 |
From a server's perspective an incoming stream of RPC requests from a pool of clients must be processed. After some amount of queueing and processing time the server completes the processing and replies to the request. This is shown in Figure 2 as a server with an input queue. The actual communication situation can become more complicated if the requested server must initiate a client request to a remote server in order to complete the incoming request. The measurement system must be able to include the processing and queueing time of the remote server in the total processing and queuing time of the original incoming request. Again, the relevant performance metrics are indicated at the bottom of the figure (message arrival rate and interarrival time; length of queue; number of servers, residence time and utilization).
FIGURE 2. Server View of Performance
+----------+ Reply
| | <------------------------------------------------------+
| | +------------+ |
| | Queue /| /| |
| Client | ---------+ +------------+| |
| Pool | Request | | | || RPC || |
| | -----------------> | | | ----> || Server(s) || -----+
| | | | | |+-----------|+
| | ---------+ |/ |/
+----------+ +------------+
Metrics:
Arrival Rate Queue Length Number of Servers
Interarrival Time Residence Time
Utilization of Servers
The following table provides an example performance report from the server's view. The performance measurement system should be able to provide this summary. The detailed description of each metric is in section 5.5.1.
TABLE 2. Sample Performance Output -- Server View
+---------------------------+----------------+---------------+
| Metric | Server_C | Server_D |
+===========================+================+===============+
| Arrival Rate | | |
| Ave | 48.0 | 8.6 |
| Std Dev | 10.6 | 2.5 |
+---------------------------+----------------+---------------+
| Throughput | 48.0 | 8.6 |
+---------------------------+----------------+---------------+
| Residence Time (sec) | | |
| Min / Max | 1.5 / 246.3 | 1.5 / 350.3 |
| Ave / Std Dev | 8.5 / 18.8 | 12.5 / 24.8 |
+---------------------------+----------------+---------------+
| Utilization (%) | 78.0 | 35.0 |
+---------------------------+----------------+---------------+
| Queue Length | 1.4 | 0.5 |
+---------------------------+----------------+---------------+
| Packets/sec | | |
| Sent | 57.0 | 22.0 |
| Received | 101.0 | 45.0 |
+---------------------------+----------------+---------------+
| KBytes/sec | | |
| Sent | 45.0 | 8.0 |
| Received | 17.5 | 5.0 |
+---------------------------+----------------+---------------+
| tab(@) center box; |
| lb | rb | rb |
| l | r | r. |
| Metric@Server_C@Server_D |
| = |
| Arrival Rate@@ |
| Ave@48.0@8.6 |
| Std Dev@10.6@2.5 |
| _ |
| Throughput@48.0@8.6 |
| _ |
| Residence Time (sec)@@ |
| Min / Max@1.5 / 246.3@\01.5 / 350.3 |
| Ave / Std Dev@8.5 / \018.8@12.5 / \024.8 |
| _ |
| Utilization (%)@78.0@35.0 |
| _ |
| Queue Length@1.4@0.5 |
| _ |
| Packets/sec@@ |
| Sent@57.0@22.0 |
| Received@101.0@45.0 |
| _ |
| KBytes/sec@@ |
| Sent@45.0@8.0 |
| Received@17.5@5.0 |
The following sections describe the metrics for the CDS services. It also provides some guidance to technology providers in how to interpret the metrics so that they can be accurately and consistently implemented. Specific instrumentation design guidelines can be found in [Frie].
The CDS performance metrics have been subdivided into client (clerk) and server (cdsd) metrics. This section describes client metrics and the next section describes server metrics.
The following are collected for each CDS client (cdsclerk\*(f!). Each
The cdsclerk is a transitory process that will terminate after a specified interval of time. This will present challenges for instrumentation initialization and storage.client must be uniquely identifiable.
For each metric the minimum, maximum, sum, mean, variance, 90th percentile, skew and histograms should be collected based on the collection intensity level.
For a given sampling interval:
The CDS architecture tries to provide a cache as close to the client as possible. This metric, when combined with the next, provides a means for assessing the efficiency of the CDS caching mechanism.
It appears that the counter DNS_CLERK_CACHE_HITS_COUNT in
routine CAADSVCR
collects this metric. If so, all that is required is to hook this up to
the standardized collection mechanism.
The CDS architecture tries to provide a cache as close to the client as possible. This metric, when combined with the previous, provides a means for assessing the efficiency of the CDS caching mechanism.
It appears that the counter DNS_CLERK_CACHE_BYPASS_COUNT in routine
CAADSVCR collects this metric. If so, all that is required is
to hook this up to the standardized collection mechanism.
This metric, when combined with the next, provides a means for assessing the workload demand on the CDS caching mechanism.
It appears that the counter DNS_CLERK_READ_OPS_COUNT in
routine CAADSVCR
collects this metric. If so, all that is required is to hook this up to
the standardized collection mechanism.
This metric, when combined with the previous, provides a means for assessing the workload demand on the CDS caching mechanism.
It appears that the counter DNS_CLERK_WRITE_OPS_COUNT in
routine CAADSVCR
collects this metric. If so, all that is required is to hook this up to
the standardized collection mechanism.
Summarize total CDS request demand by summing the following:
DNS_CLERK_READ_OPS_COUNT.
DNS_CLERK_WRITE_OPS_COUNT.
DNS_CLERK_MISC_OPS_COUNT.
This metric counts the number of requests that are not local to the cell and hence may experience significant delays due to inter-cell communication.
The following metrics were assigned medium priority and are briefly described.
This metric counts the number of requests that are not satisfied by the local cdsclerk and hence may experience additional delays.
This metric describes the length of time that an entry resides in the cache and can provide insight into cache efficiency.
The following metrics were assigned low priority and are briefly described.
The following metrics are collected per CDS server (clearinghouse) for a given sampling interval. Each server must be uniquely identifiable.
Unless otherwise stated all CDS server metrics are measured from the RPC NSI interface level and not the CDSPI level.
For each metric the minimum, maximum, sum, mean, variance, 90th percentile, skew and histograms should be collected based on the collection intensity level.
This measures the service requirement, including operating system and network software processing time, required to satisfy a client's request of a particular CDS server. This request may consist of multiple RPC packets but only one RPC NSI call. This requires that the host operating system support a performance measurement system and that DCE servers use it to gather CPU service time.
This measures the CDS server's total throughput as determined by the number of successful completions of all client requests.
This metric describes the service time of the CDS server for RPC binding handle imports or XDS read attributes requests.
This metric describes the service time of the CDS server for RPC binding handle exports or XDS modify attributes requests.
This metric describes the workload demand placed on the CDS server for RPC binding handle imports or XDS read attributes requests.
This metric describes the workload demand placed on the CDS server for RPC binding handle exports or XDS modify attributes requests.
This metric describes the number of disk writes that the CDS server performs during a modify attribute request to maintain physical consistency of the name space. A calling process blocks until the I/O completes.
This metric describes the amount of data written to disk when the CDS server performs a logging I/O.
This metric describes the number of checkpoints that the CDS server performs during its normal operation.
This metric describes the number of skulks that the CDS server performs during its normal operation.
Application developers who create DCE-based servers need consistent and integrated performance instrumentation. This custom performance instrumentation should be controlled, collected, and presented similar to and in conjunction with the DCE core services.
A mechanism to create a generic server performance interface (GSPI) for a server needs to be provided by DCE. This server interface should be accessible by the performance measurement monitor. This mechanism needs to provide the following metrics for the server as a whole and for each server interface:
This performance interface should be easy to compile into a DCE server and customizable if necessary.
Furthermore, this instrumentation should be designed around a framework in which the set of observable metrics is at a minimum statically extensible (compile time) and preferably dynamically loadable. It is important that more metrics can be added to the initial set and that the data gathered by them can be fed into higher level collection and analysis tools without having to rewrite those tools. This can be done in several ways but any extensible scheme probably requires that the metrics are named (ideally with globally unique names and probably assigned a UUID) and associated with a self-describing data structure. This information would not be transmitted with actual data samples but would be used by the data collectors and analyzers to interpret the data and possibly exercise control. Metrics should probably be grouped into families, for example RPC metrics or CPU metrics, and described at that level rather than individually.
The instrumentation framework needs to provide a standardized metric description language (perhaps IDL although it may not be a very good match). It would be nice to support the following concepts as part of the description language:
This makes adding new metrics easier and limits the variation which is likely to occur if application developers work with lower level primitives.
Although each metric family could be its own RPC process and register its own interface it is preferable to have a single data collection process (daemon) on each measured system that gathers the data and presents it through a common interface. This would limit the proliferation of RPC interfaces and improve manageability of the instrumentation system. It would need to support methods for registering, activating, and listing the state of available metrics on an individual system. If the metrics themselves are well-described then it is relatively simple to develop either a DCE-specific collector or to interface to existing monitoring packages.
More sophisticated performance diagnosis and modeling requires that the resources consumed by a particular transaction (or application) be traceable as various network resources are invoked. The result of this instrumentation is that it should provide a dynamic call graph of all services invoked and the callee of those services. Also, for each call the corresponding service time should be collected.
For example, invoking a server interface may require that the server become a client and invoke other servers. This instrumentation provides tracing of the parent call and all subsequent cascading calls.
To maximize the information and effectiveness of presenting performance data for distributed applications it is necessary to integrate critical non-DCE performance data into a cell presentation screen. This non-DCE performance data includes node CPU utilization, disk I/O activity and network activity.
Specifically, each DCE-host operating system should optionally\*(f!
If a platform vendor does not support this standard interface then the performance presentation screens should report the data as Unavailable.provide a standard interface for the following node-wide metrics:
This interface is described in [Frie].
It can be argued that this interface is an unnecessary requirement and should be left to third party suppliers. This overlooks two problems. First, critical performance metrics can only be captured by the host operating system. Second, third party tools, if not available for every node in a heterogeneous environment are of limited use to a cell administrator. Consequently, it is imperative to integrate performance data from various sources into a seamless whole to make performance management possible in a distributed environment.
This section provides an overview of a performance measurement architecture to support efficient collection of the required performance instrumentation. Complete details are in [Frie]. We will also briefly describe several potential data collection mechanisms.
Monitoring a distributed system is more difficult than monitoring a centralized system. The data is distributed and must be assembled from various disparate locations before analysis can start.
A modular approach is necessary to design a monitoring system that is flexible, generalized and efficient in implementation. This section describes our proposed approach. Significant parts of this section have been derived from [Jain] and [Ferr].
The layers of the distributed system monitor are shown in the following figure and briefly described below. Note that our proposed architecture is focused on both DCE core services and custom designed servers developed by application writers.
FIGURE 3. Layers in a Distributed Performance Monitoring System
+-----------------+
| Observation | -- DCE Server
+-----------------+
|
+-----------------+
| Collection |
+-----------------+
|
+-----------------+
| Analysis |
+-----------------+
|
+-----------------+
| Presentation |
+-----------------+
|
+-----------------+ \e
| Interpretation | \e
+-----------------+ \e
| \e
+-----------------+ Can also be automated but often done by
| Console/Control | humans. Can also be integrated with a
+-----------------+ larger distributed management scheme.
| /
+-----------------+ /
| Management | /
+-----------------+ /
A monitor may consist of multiple components from each of the layers listed above. There is a many-to-many relationship between successive layers. A single collector may gather data from multiple observers. Many analysis layers may use data gathered by a single collector. Several collectors may be acting with a single observer, etc.
Thus another view of these layers that may exist for a simple distributed system are shown in the following figure. As indicated by the figure, the interaction among layers in the measurement architecture is flexible. Multiple collectors can communicate with a single or multiple observers. Also, the analysis, presentation and higher levels of the architecture can be distributed throughout the cell.
FIGURE 4. DCE Measurement System Architecture
+---------------+ +-----------+ +-----------+ +-----------+
| DCE SUBSYSTEM | | STUB CODE | | STUB CODE | | SUBSYSTEM |
+---------------+ +-----------+ +-----------+ +-----------+
\e / \e \e /
+-----------------+ +-----------------+ +-----------------+
| Observation | | Observation | | Observation |
+-----------------+ +-----------------+ +-----------------+
\e / |
+-----------------+ +-----------------+
| Collection | | Collection |
+-----------------+ +-----------------+
/ \e |
+-----------------+ +-----------------+ +-----------------+
| Analysis | | Analysis | | Analysis |
+-----------------+ +-----------------+ +-----------------+
| | |
+-----------------+ +-----------------+ +-----------------+
| Presentation | | Presentation | | Presentation |
+-----------------+ +-----------------+ +-----------------+
| |
+-----------------+ +-----------------+
| Interpretation | | Interpretation |
+-----------------+ +-----------------+
| |
+-----------------+ +-----------------+
| Console/Control | | Console/Control |
+-----------------+ +-----------------+
| |
+-----------------+ +-----------------+
| Management | | Management |
+-----------------+ +-----------------+
We have identified three instrumentation implementation strategies. The first is to instrument the RPC run time libraries. Second, the automatic IDL definition of performance interfaces for each server in a cell. Third, the performance interfaces to the internal DCE services (such as CDS, DFS, etc.). These are described in detail in [Frie].
This document is the result of many individuals from the DCE Performance SIG who contributed their time and expertise. Principal among these are the individuals listed below.
Judi Bank (IBM), Peter Bates (Bellcore), Sarr Blumson (CITI at University of Michigan), Don Bolinger (OSF), Art Gaylord (Project Pilgrim at University of Massachusetts), Russ Heinselman (DEC), Bruce Keith (DEC), Joe Martinka (HP), Peter Oleinick (Transarc), Jerome Rolia (Carleton University), Chuck Royalty (Boeing), Barry Searle (IBM), Tracy Sienknecht (HP), Jim Van Sciver (OSF).
Current instrumentation in the RPC and CDS services is described below. The list may not be complete. It should be noted that the current instrumentation was deemed unsatisfactory because it was incomplete and inconsistent. Some of it, however, may be leveraged into the recommended measurement system.
This section briefly documents the existing instrumentation that we have found in the DCE 1.0 release of RPC.
The DCE Application Development Reference manual intro(3rpc) page describes the statistics vector data structure. The RPC runtime maintains statistics on a per address space basis.
The following metrics are collected:
rpc_c_stats_calls_in -- The number of RPCs received by runtime.
rpc_c_stats_calls_out -- The number of RPCs initiated by runtime.
rpc_c_stats_pkts_in -- The number of network packets
received by the runtime.
rpc_c_stats_pkts_out -- The number of network packets sent by the
runtime.
These values can be obtained by using the rpc_mgmt_inq_stats routine.
The data structure and additional information can be found in the Application Reference Guide.
Additionally, we found a set of undocumented logging utilities by examining the RPC 1.0 source.
In the RPC runtime library source the files rpclog.h and
rpclog.c
contain reference to an ULTRIX only logging function implemented by way
of macros. One of the more interesting ones is the TIMESTAMP(code)
macro (where the code is one of 256 possibilities delineated in the
rpclog.h). This debugging trace capability could have some performance
usage.
A cursory investigation revealed that these macros write to a single location in main memory and that specialized hardware is used to store the data to another machine for processing. It also appears that this special hardware works only on Ultrix/VAX and not Ultrix/MIPs. Also, this instrumentation has been done only for the connection RPC. It appears that this mechanism would only be useful on machines with custom built hardware monitors (something that is not likely to be pervasive across DCE platforms).
The following counters are kept by routine clerk_counter,
which updates
the counts. They are available through the show_clerk command, which
can be invoked via an RPC call. The counters are also written to the
client caches.
DNS_CLERK_PROTOCOL_ERROR_COUNT.
DNS_CLERK_AUTHENTICATION_COUNT.
DNS_CLERK_READ_OPS_COUNT.
DNS_CLERK_CACHE_HITS_COUNTs.
DNS_CLERK_CACHE_BYPASS_COUNT.
DNS_CLERK_WRITE_OPS_COUNT.
DNS_CLERK_MISC_OPS_COUNT.
REMOTE_CH_READ_OPS_COUNT.
REMOTE_CH_WRITE_OPS_COUNT.
adver_service_rpc.c counts the number of clearinghouses in the cache
(results accumulated in variable clh_count).
CDS also keeps information on the cache size. The size is initialized
by routine casys.c. If the size is explicitly specified, and is less
than the storage configured, the specification is used. Otherwise, it
is set to 0.5% of physical memory.
| Rich Friedrich | Internet email: richf@nsa.hp.com | |
| Hewlett-Packard Company | Telephone: +1-408-447-2425 | |
| 19410 Homestead Road | ||
| Mailstop 43UA | ||
| Cuptertino, CA 95014 | ||
| USA |