Monitoring OpenAM Services

You can configure OpenAM to allow you to access a web based view of OpenAM MBeans on port 8082 where the core server runs, such as http://openam-ter.example.com:8082/. Either use the console, or use the ssoadm command:

The default authentication file allows you to authenticate over HTTP as user demo, password changeit. The user name and password are kept in the file specified, with the password encrypted:

Or:

You can encrypt a new password using the ampassword command. After changing the authentication file, you must restart OpenAM for the changes to take effect.

You can configure OpenAM to allow you to listen for Java Management eXtension (JMX) clients, by default on port 9999. Either use the OpenAM console page under Configure > Global Services > System > Monitoring and make sure both Monitoring Status and Monitoring RMI interface status are both set to Enabled, or use the ssoadm command:

A number of tools support JMX, including jvisualvm and jconsole. When you use jconsole to browse OpenAM MBeans for example, the default URL for the OpenAM running on the local system is service:jmx:rmi:///jndi/rmi://localhost:9999/server.

You can also browse the MBeans by connecting to your web application container, and browsing to the OpenAM MBeans. By default, JMX monitoring for your container is likely to be accessible only locally, using the process ID.

Also see Monitoring and Management Using JMX for instructions on how to connect remotely, how to use SSL, and so forth.

You can configure OpenAM to allow you to listen on port 8085 for SNMP monitoring. Either use the console, or use the ssoadm command:

Once activated, SNMP monitoring works over UDP by default. You may want to install one of many available network monitoring tools. For the purpose of this section, basic SNMP service and monitoring tools have been installed on a GNU/Linux system. The same commands should work on a Mac OS X system.

SNMP depends on labels known as Object Identifiers (OIDs). These are uniquely defined labels, organized in tree format. For OpenAM, they are configured in a .mib file named FORGEROCK-OPENAM-CTS.mib, found inside the /path/to/tomcat/webapps/openam/WEB-INF/lib/openam-mib-schema-13.5.2-15.jar file of the OpenAM deployment.

For detailed information on configured OIDs, see "Core Token Service (CTS) Object Identifiers" in the Reference.

With the OIDs in hand, you can set up an SNMP server to collect the data. You would also need SNMP utility commands with associated OIDs to measure the current state of a component. First, to verify the operation of SNMP on a GNU/Linux system, over port 8085, using SNMP version 2c, run the following command:

The output should normally specify communications over UDP. If you get a timeout message, the SNMP service may not be running.

You can get the value for a specific OID. For example, the following command would retrieve the cumulative count for CTS create operations, over port 8085:

If your version of the tool does not support the enterprises OID string, use 1.3.6.1.4.1 instead, as in 1.3.6.1.4.1.36733.1.2.3.3.1.1.1.

For one view of the tree of OIDs, you can use the snmpwalk command. For example, the following command lists all OIDs related to CTS:

A number of CTS OIDs are listed with a Counter64 value. As defined in RFC 2578 an OID so configured has a maximum value of 2^64 - 1.

You can monitor policy evaluation performance over SNMP. OpenAM records statistics for up to a number of recent policy evaluation requests. (You can configure the number in OpenAM console under Configuration > System > Monitoring. For details, see the system configuration reference section, "Monitoring" in the Reference.)

Interface Stability: Evolving

As described in "CTS SNMP Monitoring", SNMP uses OIDs defined in the .mib file, FORGEROCK-OPENAM-POLICY.mib, found inside the /path/to/tomcat/webapps/openam/WEB-INF/lib/openam-mib-schema-13.5.2-15.jar file of the OpenAM deployment. This file specifies the statistics OpenAM keeps for policy evaluation operations. Adapt the examples in "CTS SNMP Monitoring" to read monitoring statistics about policy evaluation on the command line.

When monitoring is active, OpenAM records statistics about both the numbers and rates of policy evaluations performed, and also the times taken to process policy evaluations.

The statistics are all read-only. The base OID for policy evaluation statistics is enterprises.36733.1.2.2.1. The following table describes the values that you can read:

You can monitor stateful session statistics over SNMP. OpenAM records statistics for up to a configurable number of recent sessions. (You can configure the number in OpenAM console under Configuration > System > Monitoring. For details, see the system configuration reference section, "Monitoring" in the Reference.)

SNMP monitoring is not available for stateless sessions.

Interface Stability: Evolving

As described in "CTS SNMP Monitoring", SNMP uses OIDs defined in a .mib file that specifies the statistics OpenAM keeps for policy evaluation operations, the FORGEROCK-OPENAM-SESSION.mib file. This file is found inside the /path/to/tomcat/webapps/openam/WEB-INF/lib/openam-mib-schema-13.5.2-15.jar file of the OpenAM deployment. Adapt the examples in "CTS SNMP Monitoring" to read monitoring statistics about sessions on the command line.

When monitoring is active, OpenAM records statistics about both the numbers of internal, remote, and CTS sessions, and also the times taken to process sessions.

The statistics are all read-only. The base OID for session statistics is enterprises.36733.1.2.1. Times are expressed in nanoseconds rather than milliseconds, as many operations take less than one millisecond. The following table describes the values that you can read:

To adjust the debug level while OpenAM is running, login to the OpenAM console as OpenAM administrator and navigate to Deployment > Servers > Server Name > Debugging. The default level for debug logging is Error. This level is appropriate for normal production operations, in which case no debug log messages are expected.

Setting the debug log level to Warning increases the volume of messages. Setting the debug log level to Message dumps detailed trace messages. Unless told to do so by qualified support personnel, do not use Warning and Message levels in production.

By default, certain components that run in OpenAM’s JVM—for example, embedded OpenDJ configuration stores—do not generate trace-level messages when you configure the debug log level to Message. If you need trace-level messages for these components, navigate to Deployment > Servers > Server Name > Advanced, create a org.forgerock.openam.slf4j.enableTraceInMessage property, and set its value to true.

During development, you might find it useful to log all debug messages to a single file. In order to do so, set Merge Debug Files to on.

OpenAM logs to a single file immediately after you change this property. You do not need to restart OpenAM or the container in which it runs for the change to take effect.

OpenAM lets you capture debug log messages selectively for a specific service. This can be useful when you must turn on debugging in a production system where you want to avoid excessive logging, but must gather messages when you reproduce a problem. Perform these steps to capture debug messages for a specific service:

By default OpenAM does not rotate debug logs. To rotate debug logs, edit WEB-INF/classes/debugconfig.properties where OpenAM is deployed.

Changes to properties in the debugconfig.properties file take effect immediately. You do not need to restart OpenAM or the container in which it runs for the changes to take effect.

Start OpenAM recording with the ssoadm start-recording command. For example:

In the preceding ssoadm start-recording command example, the recording.json file specifies the information to be recorded and under what conditions recording automatically terminates. This file is known as the recording control file. "The Recording Control File" describes the format of recording control files and provides an annotated example.

An active recording event stops when:

A JSON file that is input to the ssoadm start-recording command controls the amount of information OpenAM records, the recording duration, and the location of recording output files.

Information recorded by OpenAM is stored at the path debugFileLocation/record/issueID/referenceID. For example, if the debug file location is /home/openam/debug, the issue ID 103572, and the reference ID policyEvalFails, the path containing recorded information is /home/openam/debug/record/103572/policyEvalFails.

When there are multiple recording events with the same issueID and referenceID, OpenAM appends a timestamp to the referenceID of the earliest paths. For example, multiple recording events for issue ID 103572 and reference ID policyEvalFails might be stored at the following paths:

OpenAM compresses the output from recording events when you set the zipEnable property to true. The output file can be found at the path debugFileLocation/record/issueID/referenceID_timestamp.zip. For example, compressed output for a recording event for issue ID 103572 and reference ID policyEvalFails might be stored at the following path: debugFileLocation/record/103572/policyEvalFails_2015-08-12-12-19-02-683-PDT.zip.

Use the referenceID property value to segregate output from multiple problem recreations associated with the same case. For example, while troubleshooting case 103572, you notice that you only have a problem when evaluating policy for members of the Finance realm. You could trigger two recording events as follows: