Additional Audit Details From OpenIDM 4.5.1-20 onwards, the audit service supports audit event handlers that connect to third-party tools. As described in "Configuring Audit Event Handlers", an audit event handler manages audit events, sends audit output to a defined location, and controls their format. As we do not endorse or support the use of any third-party tools, we present the handlers for such tools in this separate appendix. For the purpose of this appendix, we assume that the third-party tool is configured on the same system as your instance of OpenIDM. We realize that you may prefer to set up a third-party tool on a remote system. If you have configured a third-party tool on a remote system, the reliability of audit data may vary, depending on the reliability of your network connection. However, you can limit the risks with appropriate buffer settings, which can mitigate issues related to your network connection, free space on your system, and related resources such as RAM. (This is not an exhaustive list.) Elasticsearch Audit Event Handler Starting with OpenIDM 4.5.0, you can configure the Elasticsearch audit event handler. It allows you to log OpenIDM events in file formats compatible with the Elasticsearch search server. Installing and Configuring Elasticsearch This appendix assumes that you are installing Elasticsearch on the same system as OpenIDM. For Elasticsearch downloads and installation instructions, see the Elasticsearch Getting Started document. You can set up Elasticsearch Shield with basic authentication to help protect your audit logs. To do so, read the following Elasticsearch document on Getting Started with Shield. Follow up with the following Elasticsearch document on how you can Control Access with Basic Authentication. You can configure SSL for Elasticsearch Shield. For more information, see the following Elasticsearch document: Setting Up SSL/TLS on a Cluster. Import the certificate that you use for Elasticsearch into OpenIDM’s truststore, with the following command: $ keytool \ -import \ -trustcacerts \ -alias elasticsearch \ -file /path/to/cacert.pem \ -keystore /path/to/openidm/security/truststore Once imported, you can activate the useSSL option in the audit.json file. If you created an Elasticsearch Shield username and password, you can also associate that information with the username and password entries in that same audit.json file. Creating an Audit Index for Elasticsearch If you want to create an audit index for Elasticsearch, you must set it up before starting OpenIDM, for the audit event topics described in this section: "OpenIDM Audit Event Topics". To do so, execute the REST call shown in the following audit index file. Note the properties that are not_analyzed. Such fields are not indexed within Elasticsearch. The REST call in the audit index file includes the following URL: http://myUsername:myPassword@localhost:9200/audit That URL assumes that your Elasticsearch deployment is on the localhost system, accessible on default port 9200, configured with an indexName of audit. It also assumes that you have configured basic authentication on Elasticsearch Shield, with a username of myUsername and a password of myPassword. If any part of your Elasticsearch deployment is different, revise the URL accordingly. Do not transmit usernames and passwords over an insecure connection. Enable the useSSL option, as described in "Configuring the Elasticsearch Audit Event Handler". Configuring the Elasticsearch Audit Event Handler "Configuring the Elasticsearch Audit Event Handler via the Admin UI" and "Configuring the Elasticsearch Audit Event Handler in audit.json" illustrate how you can configure the Elasticsearch Audit Event Handler. If you activate the Elasticsearch audit event handler, we recommend that you enable buffering for optimal performance, by setting: "enabled" : true, The buffering settings shown are not recommendations for any specific environment. If performance and audit data integrity are important in your environment, you may need to adjust these numbers. If you choose to protect your Elasticsearch deployment with the plugin known as Shield, and configure the ability to Control Access with Basic Authentication, you can substitute your Elasticsearch Shield admin or power_user credentials for myUsername and myPassword. If you activate the useSSL option, install the SSL certificate that you use for Elasticsearch into the OpenIDM keystore. For more information, see the following section: "Accessing the Security Management Service". Configuring the Elasticsearch Audit Event Handler via the Admin UI To configure this event handler through the Admin UI, click Configure > System Preferences > Audit. Select ElasticsearchAuditEventHandler from the drop-down text box, click Add Event Handler, and configure it in the window that appears. For a list of properties, see "Common Audit Event Handler Property Configuration". Configuring the Elasticsearch Audit Event Handler in audit.json Alternatively, you can configure the Elasticsearch audit event handler in the audit.json file for your project. The following code is an excerpt from the audit.json file, with Elasticsearch configured as the handler for audit queries: { "auditServiceConfig" : { "handlerForQueries" : "elasticsearch", "availableAuditEventHandlers" : [ "org.forgerock.audit.handlers.elasticsearch.ElasticsearchAuditEventHandler", "org.forgerock.audit.handlers.csv.CsvAuditEventHandler", "org.forgerock.openidm.audit.impl.RepositoryAuditEventHandler", "org.forgerock.openidm.audit.impl.RouterAuditEventHandler" ], You should also set up configuration for the Elasticsearch event handler. The entries shown are defaults, and can be configured. In fact, if you have set up Elasticsearch Shield, with or without SSL/TLS, as described in "Installing and Configuring Elasticsearch", you should change some of these defaults. "eventHandlers" : [ { "name" : "elasticsearch" "class" : "org.forgerock.audit.handlers.elasticsearch.ElasticsearchAuditEventHandler", "config" : { "connection" : { "useSSL" : false, "host" : "localhost", "port" : "9200" }, "indexMapping" : { "indexName" : "audit" }, "buffering" : { "enabled" : false, "maxSize" : 20000, "writeInterval" : "1 second", "maxBatchedEvents" : "500" } "topics" : [ "access", "activity", "recon", "sync", "authentication", "config" ] } } ], If you set useSSL to true, add the following properties to the connection code block: "username" : "myUsername", "password" : "myPassword", For more information on the other options shown in audit.json, see "Common Audit Event Handler Property Configuration". Querying and Reading Elasticsearch Audit Events By default, Elasticsearch uses pagination. As noted in the following Elasticsearch document on Pagination, queries are limited to the first 10 results. For example, the following query is limited to the first 10 results: $ curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Content-Type: application/json" \ --request GET \ "http://localhost:8080/openidm/audit/access?_queryFilter=true" To override the limit of 10 results, follow the guidance shown in "Paging and Counting Query Results" for pageSize. To set up a queryFilter that uses a "starts with" sw or "equals" eq comparison expression, you will need to set it up as a not_analyzed string field, as described in the following Elasticsearch document on Term Query.. You should also review the section on "Comparison Expressions". If you haven’t already done so, you may need to modify and rerun the REST call described in "Creating an Audit Index for Elasticsearch". The queryFilter output should include UUIDs as id values for each audit event. To read audit data for that event, include that UUID in the URL. For example, the following REST call specifies an access event, which includes data on the client: $ curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Content-Type: application/json" \ --request GET "http://localhost:8080/openidm/audit/access/75ca07f5-836c-4e7b-beaa-ae968325a529-622" Audit Configuration Schema The following tables depict schema for the six audit event topics used by OpenIDM. Each topic is associated with the following files that you can find in the openidm/audit directory: access.csv: see "Access Event Topic Properties" activity.csv: see "Activity Event Topic Properties" authentication.csv: see "Authentication Event Topic Properties" config.csv: see "Configuration Event Topic Properties" recon.csv: see "Reconciliation Event Topic Properties" sync.csv: see "Synchronization Event Topic Properties" If you open the CSV files from that directory in a spreadsheet application, those files can help you read through the tables shown in this appendix. OpenIDM Specific Audit Event Topics Reconciliation Event Topic Properties Event Property Description _id UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0" transactionId The UUID of the transaction; you may see the same ID in different audit event topics. timestamp The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z" eventName The name of the audit event: recon for this log userId User ID trackingIds A unique value for an object being tracked action Reconciliation action, depicted as a CREST action. For more information, see "Synchronization Actions" exception The stack trace of the exception linkQualifier The link qualifier applied to the action; For more information, see "Mapping a Single Source Object to Multiple Target Objects" mapping The name of the mapping used for the synchronization operation, defined in conf/sync.json. message Description of the synchronization action messageDetail Details from the synchronization run, shown as CREST output situation The synchronization situation described in "Synchronization Situations" sourceObjectId The object ID on the source system, such as managed/user/jdoe status Reconciliation result status, such as SUCCESS or FAILURE targetObjectId The object ID on the target system, such as system/xmlfile/account/bjensen reconciling What OpenIDM is reconciling, source for the first phase, target for the second phase. ambiguousTargetObjectIds When the situation is AMBIGUOUS or UNQUALIFIED, and OpenIDM cannot distinguish between more than one target object, OpenIDM logs the object IDs, to help figure out what was ambiguous. reconAction The reconciliation action, typically recon or null entryType The type of reconciliation log entry, such as start, entry, or summary. reconId UUID for the reconciliation operation Synchronization Event Topic Properties Event Property Description _id UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0" transactionId The UUID of the transaction; you may see the same ID in different audit event topics. timestamp The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z" eventName The name of the audit event: sync for this log userId User ID trackingIds A unique value for an object being tracked action Synchronization action, depicted as a CREST action. For more information, see "Synchronization Actions" exception The stack trace of the exception linkQualifier The link qualifier applied to the action; For more information, see "Mapping a Single Source Object to Multiple Target Objects" mapping The name of the mapping used for the synchronization operation, defined in conf/sync.json. message Description of the synchronization action messageDetail Details from the reconciliation run, shown as CREST output situation The synchronization situation described in "Synchronization Situations" sourceObjectId The object ID on the source system, such as managed/user/jdoe status Reconciliation result status, such as SUCCESS or FAILURE targetObjectId The object ID on the target system, such as uid=jdoe,ou=People,dc=example,dc=com Commons Audit Event Topics Access Event Topic Properties Event Property Description _id UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0" timestamp The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z" eventName The name of the audit event: access for this log transactionId The UUID of the transaction; you may see the same transaction for the same event in different audit event topics userId User ID trackingIds A unique value for an object being tracked server.ip IP address of the OpenIDM server server.port Port number used by the OpenIDM server client.ip Client IP address client.port Client port number request.protocol Protocol for request, typically CREST request.operation Typically a CREST operation request.detail Typically details for an ACTION request http.request.secure Boolean for request security http.request.method HTTP method requested by the client http.request.path Path of the HTTP request http.request.queryParameters Parameters sent in the HTTP request, such as a key/value pair http.request.headers HTTP headers for the request (optional) http.request.cookies HTTP cookies for the request (optional) http.response.headers HTTP response headers (optional) response.status Normally, SUCCESSFUL, FAILED, or null response.statusCode SUCCESS in response.status leads to a null response.statusCode; FAILURE leads to a 400-level error response.detail Message associated with response.statusCode, such as Not Found or Internal Server Error response.elapsedTime Time to execute the access event response.elapsedTimeUnits Units for response time roles OpenIDM roles associated with the request Activity Event Topic Properties Event Property Description _id UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0" timestamp The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z" eventName The name of the audit event: activity for this log transactionId The UUID of the transaction; you may see the same transaction for the same event in different audit event topics. userId User ID trackingIds A unique value for the object being tracked runAs User to run the activity as; may be used in delegated administration objectId Object identifier, such as /managed/user/jdoe operation Typically a CREST operation before JSON representation of the object prior to the activity after JSON representation of the object after the activity changedFields Fields that were changed, based on "Watched Fields: Defining Fields to Monitor" revision Object revision number status Result, such as SUCCESS message Human readable text about the action passwordChanged True/False entry on changes to the password Authentication Event Topic Properties Event Property Description _id UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0" timestamp The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z" eventName The name of the audit event: authentication for this log transactionId The UUID of the transaction; you may see the same transaction for the same event in different audit event topics. userId User ID trackingIds A unique value for an object being tracked result The result of the transaction, either "SUCCESSFUL", or "FAILED" principal An array of the accounts used to authenticate, such as [ "openidm-admin" ] context The complete security context of the authentication operation, including the authenticating ID, the targeted endpoint, the roles applied, and the IP address from which the authentication request was made. entries The JSON representation of the authentication session Configuration Event Topic Properties Event Property Description _id UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0" timestamp The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z" eventName The name of the audit event: config for this log transactionId The UUID of the transaction; you may see the same transaction for the same event in different audit event topics. userId User ID trackingIds A unique value for an object being tracked runAs User to run the activity as; may be used in delegated administration objectId Object identifier, such as ui operation Typically a CREST operation before JSON representation of the object prior to the activity after JSON representation of the object after to the activity changedFields Fields that were changed, based on "Watched Fields: Defining Fields to Monitor" revision Object revision number Audit Event Handler Configuration When you set up an audit event handler, you can configure several properties. Most of the properties in the following table are used by the CSV audit event handler, and may be configured in the audit configuration file for your project: project-dir/conf/audit.json. In several cases, the following table does not include an entry for description, as the UI Label / Text is sufficient. If you’re reviewing this from the OpenIDM Admin UI, click Configure > System Preferences > Audit, and click the edit icon associated with your event handler. The tables shown in this section reflect the order in which properties are shown in the Admin UI. That order differs when you review the properties in your project’s audit.json file. Common Audit Event Handler Property Configuration UI Label / Text audit.json File Label Description Name name config sub-property. Given name of the audit event handler Audit Events topics config sub-property; may include events such as access, activity, and config Use for Queries handlerForQueries Audit Event Handler to use for Queries Enabled enabled config sub-property n/a config The JSON object used to configure the handler; includes several sub-properties Shown only in audit.json class The class name in the Java file(s) used to build the handler Two properties shown only in the audit.json file for your project are: The class name used to build the handler, which may shown as one of the availableAuditEventHandlers, as shown in this excerpt from the audit.json file: "availableAuditEventHandlers" : [ "org.forgerock.audit.handlers.elasticsearch.ElasticsearchAuditEventHandler", "org.forgerock.audit.handlers.csv.CsvAuditEventHandler", "org.forgerock.openidm.audit.impl.RepositoryAuditEventHandler", "org.forgerock.openidm.audit.impl.RouterAuditEventHandler" ], The audit event handler config property, which comes after a second instance of the class name of that audit event handler. For an example, see the following excerpt of an audit.json file: "eventHandlers" : [ { "class" : "org.forgerock.audit.handlers.csv.CsvAuditEventHandler", "config" : { "name" : "csv", "logDirectory" : "&{launcher.working.location}/audit", "topics" : [ The following table includes config properties for the CSV audit event handler. That is different from the audit event topic config property, a category of logging data described in "OpenIDM Audit Event Topics". CSV Audit Event Handler Unique config Properties UI Label / Text audit.json File Label Description File Rotation fileRotation File rotation options rotationEnabled rotationEnabled File rotation: true or false boolean maxFileSize maxFileSize File rotation: Maximum size for an audit file, before rotation is triggered rotationFilePrefix rotationFilePrefix File rotation: Prefix to add to the start of an audit file, after rotation Rotation Times rotationTimes File rotation: Time to trigger, after midnight; may use entries such as 5 seconds, 5 minutes, 5 hours, disabled File Rotation Suffix rotationFileSuffix File rotation: Suffix appended to the end of audit file names Rotation Interval rotationInterval File rotation: Time period between log rotation; may use 5 seconds, 5 minutes, 5 hours, disabled File Retention fileRetention Specifies how long to keep an audit file Maximum Number of Historical Files maxNumberOfHistoryFiles File retention: Maximum number of backup audit files Maximum Disk Space maxDiskSpaceToUse File retention: Maximum disk space for audit files Minimum Free Space Required minFreeSpaceRequired File retention: Minimum disk space required on system with audit files rotationRetentionCheckInterval rotationRetentionCheckInterval Interval for periodically checking file rotation and retention policies Log Directory logDirectory Directory with CSV audit event handler files CSV Output Formatting formatting quoteChar quoteChar Formatting: Character used around a CSV field delimiterChar delimiterChar Formatting: Character between CSV fields End of Line Symbols endOfLineSymbols Formatting: end of line symbol, such as \n or \r Security: CSV Tamper Evident Configuration security Uses keystore-based signatures Enabled enabled CSV Tamper Evident Configuration: true or false Filename filename CSV Tamper Evident Configuration: Path to the Java keystore Password password CSV Tamper Evident Configuration: Password for the Java keystore Keystore Handler keystoreHandlerName CSV Tamper Evident Configuration: Keystore name Signature Interval signatureInterval CSV Tamper Evident Configuration: Signature generation interval. Default = 1 hour. Units described in "Minimum Admin UI CSV Audit Handler Configuration Requirements". Buffering buffering Configuration for optional event buffering enabled enabled Buffering: true or false autoFlush autoFlush Buffering: avoids flushing after each event Except for the common properties shown in "Common Audit Event Handler Property Configuration", the Repository and Router audit event handlers share one unique property: resourcePath: { "class" : "org.forgerock.openidm.audit.impl.RouterAuditEventHandler", "config" : { "name" : "router", "topics" : [ "access", "activity", "recon", "sync", "authentication", "config" ], "resourcePath" : "system/auditdb" } }, Repository / Router Audit Event Handler Unique config Properties UI Label / Text audit.json File Label Description resourcePath resourcePath Path to the repository resource Take care when reading JMS properties in the audit.json file. They include the standard Open Identity Platform audit event topics, along with JMS-unique topics: JMS Audit Event Handler Unique config Properties UI Label / Text audit.json File Label Description Delivery Mode deliveryMode For messages from a JMS provider; may be PERSISTENT or NON_PERSISTENT Session Mode sessionMode Acknowledgement mode, in sessions without transactions. May be AUTO, CLIENT, or DUPS_OK. Batch Configuration Settings batchConfiguration Options when batch messaging is enabled Batch Enabled batchEnabled Boolean for batch delivery of audit events Capacity capacity Maximum event count in the batch queue; additional events are dropped Thread Count threadCount Number of concurrent threads that pull events from the batch queue Maximum Batched Events maxBatchedEvents Maximum number of events per batch Insert Timeout (Seconds) insertTimeoutSec Waiting period (seconds) for available capacity, when a new event enters the queue Polling Timeout (Seconds) pollTimeoutSec Worker thread waiting period (seconds) for the next event, before going idle Shutdown Timeout (Seconds) shutdownTimeoutSec Application waiting period (seconds) for worker thread termination JNDI Configuration jndiConfiguration Java Naming and Directory Interface (JNDI) Configuration Settings JNDI Context Properties contextProperties Settings to populate the JNDI initial context with JNDI Context Factory java.naming.factory.initial Initial JNDI context factory, such as com.tibco.tibjms.naming.TibjmsInitialContextFactory JNDI Provider URL java.naming.provider.url Depends on provider; options include tcp://localhost:61616 and tibjmsnaming://192.168.1.133:7222 JNDI Topic topic.audit Relevant JNDI topic; default=audit JNDI Topic Name topicName JNDI lookup name for the JMS topic Connection Factory connectionFactoryName JNDI lookup name for the JMS connection factory The Elasticsearch audit event handler is relatively complex, with config subcategories for connection, indexMapping, buffering, and topics. Elasticsearch Audit Event Handler Unique config Properties UI Label / Text audit.json File Label Description Connection connection Elasticsearch audit event handler useSSL useSSL Connection: Use SSL/TLS to connect to Elasticsearch host host Connection: Hostname or IP address of Elasticsearch (default: localhost) port port Connection: Port used by Elasticsearch (default: 9200) username username Connection: Username when Basic Authentication is enabled via Elasticsearch Shield password password Connection: Password when Basic Authentication is enabled via Elasticsearch Shield Index Mapping indexMapping Defines how an audit event and its fields are stored and indexed indexName indexName Index Mapping: Index Name (default=audit). Change if 'audit' conflicts with an existing Elasticsearch index Buffering buffering Configuration for buffering events and batch writes (increases write-throughput) enabled enabled Buffering: recommended maxSize maxSize Buffering: Fixed maximum number of events that can be buffered (default: 10000) Write Interval writeInterval Buffering: Interval (default: 1 s) at which buffered events are written to Elasticsearch (units of 'ms' or 's' are recommended) maxBatchedEvents maxBatchedEvents Buffering: Maximum number of events per batch-write to Elasticsearch for each Write Interval (default: 500) Authentication and Session Module Configuration Details Release Levels & Interface Stability