Additional Audit Details

From OpenIDM 4.5.1-20 onwards, the audit service supports audit event handlers that connect to third-party tools. As described in "Configuring Audit Event Handlers", an audit event handler manages audit events, sends audit output to a defined location, and controls their format.

As we do not endorse or support the use of any third-party tools, we present the handlers for such tools in this separate appendix. For the purpose of this appendix, we assume that the third-party tool is configured on the same system as your instance of OpenIDM. We realize that you may prefer to set up a third-party tool on a remote system.

If you have configured a third-party tool on a remote system, the reliability of audit data may vary, depending on the reliability of your network connection. However, you can limit the risks with appropriate buffer settings, which can mitigate issues related to your network connection, free space on your system, and related resources such as RAM. (This is not an exhaustive list.)

Elasticsearch Audit Event Handler

Starting with OpenIDM 4.5.0, you can configure the Elasticsearch audit event handler. It allows you to log OpenIDM events in file formats compatible with the Elasticsearch search server.

Installing and Configuring Elasticsearch

This appendix assumes that you are installing Elasticsearch on the same system as OpenIDM. For Elasticsearch downloads and installation instructions, see the Elasticsearch Getting Started document.

You can set up Elasticsearch Shield with basic authentication to help protect your audit logs. To do so, read the following Elasticsearch document on Getting Started with Shield. Follow up with the following Elasticsearch document on how you can Control Access with Basic Authentication.

You can configure SSL for Elasticsearch Shield. For more information, see the following Elasticsearch document: Setting Up SSL/TLS on a Cluster.

Import the certificate that you use for Elasticsearch into OpenIDM’s truststore, with the following command:

$ keytool \
-import \
-trustcacerts \
-alias elasticsearch \
-file /path/to/cacert.pem \
-keystore /path/to/openidm/security/truststore

Once imported, you can activate the useSSL option in the audit.json file. If you created an Elasticsearch Shield username and password, you can also associate that information with the username and password entries in that same audit.json file.

Creating an Audit Index for Elasticsearch

If you want to create an audit index for Elasticsearch, you must set it up before starting OpenIDM, for the audit event topics described in this section: "OpenIDM Audit Event Topics".

To do so, execute the REST call shown in the following audit index file. Note the properties that are not_analyzed. Such fields are not indexed within Elasticsearch.

The REST call in the audit index file includes the following URL:

http://myUsername:myPassword@localhost:9200/audit

That URL assumes that your Elasticsearch deployment is on the localhost system, accessible on default port 9200, configured with an indexName of audit.

It also assumes that you have configured basic authentication on Elasticsearch Shield, with a username of myUsername and a password of myPassword.

If any part of your Elasticsearch deployment is different, revise the URL accordingly.

Do not transmit usernames and passwords over an insecure connection. Enable the useSSL option, as described in "Configuring the Elasticsearch Audit Event Handler".

Configuring the Elasticsearch Audit Event Handler

If you activate the Elasticsearch audit event handler, we recommend that you enable buffering for optimal performance, by setting:

"enabled" : true,

The buffering settings shown are not recommendations for any specific environment. If performance and audit data integrity are important in your environment, you may need to adjust these numbers.

If you choose to protect your Elasticsearch deployment with the plugin known as Shield, and configure the ability to Control Access with Basic Authentication, you can substitute your Elasticsearch Shield admin or power_user credentials for myUsername and myPassword.

If you activate the useSSL option, install the SSL certificate that you use for Elasticsearch into the OpenIDM keystore. For more information, see the following section: "Accessing the Security Management Service".

Configuring the Elasticsearch Audit Event Handler via the Admin UI

To configure this event handler through the Admin UI, click Configure > System Preferences > Audit. Select ElasticsearchAuditEventHandler from the drop-down text box, click Add Event Handler, and configure it in the window that appears.

elastic audit

Configuring the Elasticsearch Audit Event Handler in audit.json

Alternatively, you can configure the Elasticsearch audit event handler in the audit.json file for your project.

The following code is an excerpt from the audit.json file, with Elasticsearch configured as the handler for audit queries:

{
   "auditServiceConfig" : {
      "handlerForQueries" : "elasticsearch",
      "availableAuditEventHandlers" : [
         "org.forgerock.audit.handlers.elasticsearch.ElasticsearchAuditEventHandler",
         "org.forgerock.audit.handlers.csv.CsvAuditEventHandler",
         "org.forgerock.openidm.audit.impl.RepositoryAuditEventHandler",
         "org.forgerock.openidm.audit.impl.RouterAuditEventHandler"
      ],

You should also set up configuration for the Elasticsearch event handler. The entries shown are defaults, and can be configured. In fact, if you have set up Elasticsearch Shield, with or without SSL/TLS, as described in "Installing and Configuring Elasticsearch", you should change some of these defaults.

"eventHandlers" : [
   {
      "name" : "elasticsearch"
      "class" : "org.forgerock.audit.handlers.elasticsearch.ElasticsearchAuditEventHandler",
      "config" : {
         "connection" : {
            "useSSL" : false,
            "host" : "localhost",
            "port" : "9200"
         },
         "indexMapping" : {
            "indexName" : "audit"
         },
         "buffering" : {
            "enabled" : false,
            "maxSize" : 20000,
            "writeInterval" : "1 second",
            "maxBatchedEvents" : "500"
         }
         "topics" : [
            "access",
            "activity",
            "recon",
            "sync",
            "authentication",
            "config"
         ]
      }
   }
],

If you set useSSL to true, add the following properties to the connection code block:

"username" : "myUsername",
"password" : "myPassword",

For more information on the other options shown in audit.json, see "Common Audit Event Handler Property Configuration".

Querying and Reading Elasticsearch Audit Events

By default, Elasticsearch uses pagination. As noted in the following Elasticsearch document on Pagination, queries are limited to the first 10 results.

For example, the following query is limited to the first 10 results:

$ curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Content-Type: application/json" \
--request GET \
"http://localhost:8080/openidm/audit/access?_queryFilter=true"

To override the limit of 10 results, follow the guidance shown in "Paging and Counting Query Results" for pageSize.

To set up a queryFilter that uses a "starts with" sw or "equals" eq comparison expression, you will need to set it up as a not_analyzed string field, as described in the following Elasticsearch document on Term Query.. You should also review the section on "Comparison Expressions". If you haven’t already done so, you may need to modify and rerun the REST call described in "Creating an Audit Index for Elasticsearch".

The queryFilter output should include UUIDs as id values for each audit event. To read audit data for that event, include that UUID in the URL. For example, the following REST call specifies an access event, which includes data on the client:

$ curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Content-Type: application/json" \
--request GET
"http://localhost:8080/openidm/audit/access/75ca07f5-836c-4e7b-beaa-ae968325a529-622"

Audit Configuration Schema

The following tables depict schema for the six audit event topics used by OpenIDM. Each topic is associated with the following files that you can find in the openidm/audit directory:

If you open the CSV files from that directory in a spreadsheet application, those files can help you read through the tables shown in this appendix.

OpenIDM Specific Audit Event Topics

Reconciliation Event Topic Properties
Event Property Description

_id

UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0"

transactionId

The UUID of the transaction; you may see the same ID in different audit event topics.

timestamp

The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z"

eventName

The name of the audit event: recon for this log

userId

User ID

trackingIds

A unique value for an object being tracked

action

Reconciliation action, depicted as a CREST action. For more information, see "Synchronization Actions"

exception

The stack trace of the exception

linkQualifier

The link qualifier applied to the action; For more information, see "Mapping a Single Source Object to Multiple Target Objects"

mapping

The name of the mapping used for the synchronization operation, defined in conf/sync.json.

message

Description of the synchronization action

messageDetail

Details from the synchronization run, shown as CREST output

situation

The synchronization situation described in "Synchronization Situations"

sourceObjectId

The object ID on the source system, such as managed/user/jdoe

status

Reconciliation result status, such as SUCCESS or FAILURE

targetObjectId

The object ID on the target system, such as system/xmlfile/account/bjensen

reconciling

What OpenIDM is reconciling, source for the first phase, target for the second phase.

ambiguousTargetObjectIds

When the situation is AMBIGUOUS or UNQUALIFIED, and OpenIDM cannot distinguish between more than one target object, OpenIDM logs the object IDs, to help figure out what was ambiguous.

reconAction

The reconciliation action, typically recon or null

entryType

The type of reconciliation log entry, such as start, entry, or summary.

reconId

UUID for the reconciliation operation

Synchronization Event Topic Properties
Event Property Description

_id

UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0"

transactionId

The UUID of the transaction; you may see the same ID in different audit event topics.

timestamp

The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z"

eventName

The name of the audit event: sync for this log

userId

User ID

trackingIds

A unique value for an object being tracked

action

Synchronization action, depicted as a CREST action. For more information, see "Synchronization Actions"

exception

The stack trace of the exception

linkQualifier

The link qualifier applied to the action; For more information, see "Mapping a Single Source Object to Multiple Target Objects"

mapping

The name of the mapping used for the synchronization operation, defined in conf/sync.json.

message

Description of the synchronization action

messageDetail

Details from the reconciliation run, shown as CREST output

situation

The synchronization situation described in "Synchronization Situations"

sourceObjectId

The object ID on the source system, such as managed/user/jdoe

status

Reconciliation result status, such as SUCCESS or FAILURE

targetObjectId

The object ID on the target system, such as uid=jdoe,ou=People,dc=example,dc=com

Commons Audit Event Topics

Access Event Topic Properties
Event Property Description

_id

UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0"

timestamp

The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z"

eventName

The name of the audit event: access for this log

transactionId

The UUID of the transaction; you may see the same transaction for the same event in different audit event topics

userId

User ID

trackingIds

A unique value for an object being tracked

server.ip

IP address of the OpenIDM server

server.port

Port number used by the OpenIDM server

client.ip

Client IP address

client.port

Client port number

request.protocol

Protocol for request, typically CREST

request.operation

Typically a CREST operation

request.detail

Typically details for an ACTION request

http.request.secure

Boolean for request security

http.request.method

HTTP method requested by the client

http.request.path

Path of the HTTP request

http.request.queryParameters

Parameters sent in the HTTP request, such as a key/value pair

http.request.headers

HTTP headers for the request (optional)

http.request.cookies

HTTP cookies for the request (optional)

http.response.headers

HTTP response headers (optional)

response.status

Normally, SUCCESSFUL, FAILED, or null

response.statusCode

SUCCESS in response.status leads to a null response.statusCode; FAILURE leads to a 400-level error

response.detail

Message associated with response.statusCode, such as Not Found or Internal Server Error

response.elapsedTime

Time to execute the access event

response.elapsedTimeUnits

Units for response time

roles

OpenIDM roles associated with the request

Activity Event Topic Properties
Event Property Description

_id

UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0"

timestamp

The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z"

eventName

The name of the audit event: activity for this log

transactionId

The UUID of the transaction; you may see the same transaction for the same event in different audit event topics.

userId

User ID

trackingIds

A unique value for the object being tracked

runAs

User to run the activity as; may be used in delegated administration

objectId

Object identifier, such as /managed/user/jdoe

operation

Typically a CREST operation

before

JSON representation of the object prior to the activity

after

JSON representation of the object after the activity

changedFields

Fields that were changed, based on "Watched Fields: Defining Fields to Monitor"

revision

Object revision number

status

Result, such as SUCCESS

message

Human readable text about the action

passwordChanged

True/False entry on changes to the password

Authentication Event Topic Properties
Event Property Description

_id

UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0"

timestamp

The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z"

eventName

The name of the audit event: authentication for this log

transactionId

The UUID of the transaction; you may see the same transaction for the same event in different audit event topics.

userId

User ID

trackingIds

A unique value for an object being tracked

result

The result of the transaction, either "SUCCESSFUL", or "FAILED"

principal

An array of the accounts used to authenticate, such as [ "openidm-admin" ]

context

The complete security context of the authentication operation, including the authenticating ID, the targeted endpoint, the roles applied, and the IP address from which the authentication request was made.

entries

The JSON representation of the authentication session

Configuration Event Topic Properties
Event Property Description

_id

UUID for the message object, such as "0419d364-1b3d-4e4f-b769-555c3ca098b0"

timestamp

The time that OpenIDM logged the message, in UTC format; for example "2015-05-18T08:48:00.160Z"

eventName

The name of the audit event: config for this log

transactionId

The UUID of the transaction; you may see the same transaction for the same event in different audit event topics.

userId

User ID

trackingIds

A unique value for an object being tracked

runAs

User to run the activity as; may be used in delegated administration

objectId

Object identifier, such as ui

operation

Typically a CREST operation

before

JSON representation of the object prior to the activity

after

JSON representation of the object after to the activity

changedFields

Fields that were changed, based on "Watched Fields: Defining Fields to Monitor"

revision

Object revision number

Audit Event Handler Configuration

When you set up an audit event handler, you can configure several properties. Most of the properties in the following table are used by the CSV audit event handler, and may be configured in the audit configuration file for your project: project-dir/conf/audit.json.

In several cases, the following table does not include an entry for description, as the UI Label / Text is sufficient.

If you’re reviewing this from the OpenIDM Admin UI, click Configure > System Preferences > Audit, and click the edit icon associated with your event handler.

The tables shown in this section reflect the order in which properties are shown in the Admin UI. That order differs when you review the properties in your project’s audit.json file.

Common Audit Event Handler Property Configuration
UI Label / Text audit.json File Label Description

Name

name

config sub-property. Given name of the audit event handler

Audit Events

topics

config sub-property; may include events such as access, activity, and config

Use for Queries

handlerForQueries

Audit Event Handler to use for Queries

Enabled

enabled

config sub-property

n/a

config

The JSON object used to configure the handler; includes several sub-properties

Shown only in audit.json

class

The class name in the Java file(s) used to build the handler

Two properties shown only in the audit.json file for your project are:

  • The class name used to build the handler, which may shown as one of the availableAuditEventHandlers, as shown in this excerpt from the audit.json file:

    "availableAuditEventHandlers" : [
        "org.forgerock.audit.handlers.elasticsearch.ElasticsearchAuditEventHandler",
        "org.forgerock.audit.handlers.csv.CsvAuditEventHandler",
        "org.forgerock.openidm.audit.impl.RepositoryAuditEventHandler",
        "org.forgerock.openidm.audit.impl.RouterAuditEventHandler"
    ],
  • The audit event handler config property, which comes after a second instance of the class name of that audit event handler. For an example, see the following excerpt of an audit.json file:

    "eventHandlers" : [
        {
            "class" : "org.forgerock.audit.handlers.csv.CsvAuditEventHandler",
            "config" : {
                "name" : "csv",
                "logDirectory" : "&{launcher.working.location}/audit",
                "topics" : [

The following table includes config properties for the CSV audit event handler. That is different from the audit event topic config property, a category of logging data described in "OpenIDM Audit Event Topics".

CSV Audit Event Handler Unique config Properties
UI Label / Text audit.json File Label Description

File Rotation

fileRotation

File rotation options

rotationEnabled

rotationEnabled

File rotation: true or false boolean

maxFileSize

maxFileSize

File rotation: Maximum size for an audit file, before rotation is triggered

rotationFilePrefix

rotationFilePrefix

File rotation: Prefix to add to the start of an audit file, after rotation

Rotation Times

rotationTimes

File rotation: Time to trigger, after midnight; may use entries such as 5 seconds, 5 minutes, 5 hours, disabled

File Rotation Suffix

rotationFileSuffix

File rotation: Suffix appended to the end of audit file names

Rotation Interval

rotationInterval

File rotation: Time period between log rotation; may use 5 seconds, 5 minutes, 5 hours, disabled

File Retention

fileRetention

Specifies how long to keep an audit file

Maximum Number of Historical Files

maxNumberOfHistoryFiles

File retention: Maximum number of backup audit files

Maximum Disk Space

maxDiskSpaceToUse

File retention: Maximum disk space for audit files

Minimum Free Space Required

minFreeSpaceRequired

File retention: Minimum disk space required on system with audit files

rotationRetentionCheckInterval

rotationRetentionCheckInterval

Interval for periodically checking file rotation and retention policies

Log Directory

logDirectory

Directory with CSV audit event handler files

CSV Output Formatting

formatting

quoteChar

quoteChar

Formatting: Character used around a CSV field

delimiterChar

delimiterChar

Formatting: Character between CSV fields

End of Line Symbols

endOfLineSymbols

Formatting: end of line symbol, such as \n or \r

Security: CSV Tamper Evident Configuration

security

Uses keystore-based signatures

Enabled

enabled

CSV Tamper Evident Configuration: true or false

Filename

filename

CSV Tamper Evident Configuration: Path to the Java keystore

Password

password

CSV Tamper Evident Configuration: Password for the Java keystore

Keystore Handler

keystoreHandlerName

CSV Tamper Evident Configuration: Keystore name

Signature Interval

signatureInterval

CSV Tamper Evident Configuration: Signature generation interval. Default = 1 hour. Units described in "Minimum Admin UI CSV Audit Handler Configuration Requirements".

Buffering

buffering

Configuration for optional event buffering

enabled

enabled

Buffering: true or false

autoFlush

autoFlush

Buffering: avoids flushing after each event

Except for the common properties shown in "Common Audit Event Handler Property Configuration", the Repository and Router audit event handlers share one unique property: resourcePath:

{
    "class" : "org.forgerock.openidm.audit.impl.RouterAuditEventHandler",
    "config" : {
        "name" : "router",
        "topics" : [ "access", "activity", "recon", "sync", "authentication", "config" ],
        "resourcePath" : "system/auditdb"
    }
 },
Repository / Router Audit Event Handler Unique config Properties
UI Label / Text audit.json File Label Description

resourcePath

resourcePath

Path to the repository resource

Take care when reading JMS properties in the audit.json file. They include the standard Open Identity Platform audit event topics, along with JMS-unique topics:

JMS Audit Event Handler Unique config Properties
UI Label / Text audit.json File Label Description

Delivery Mode

deliveryMode

For messages from a JMS provider; may be PERSISTENT or NON_PERSISTENT

Session Mode

sessionMode

Acknowledgement mode, in sessions without transactions. May be AUTO, CLIENT, or DUPS_OK.

Batch Configuration Settings

batchConfiguration

Options when batch messaging is enabled

Batch Enabled

batchEnabled

Boolean for batch delivery of audit events

Capacity

capacity

Maximum event count in the batch queue; additional events are dropped

Thread Count

threadCount

Number of concurrent threads that pull events from the batch queue

Maximum Batched Events

maxBatchedEvents

Maximum number of events per batch

Insert Timeout (Seconds)

insertTimeoutSec

Waiting period (seconds) for available capacity, when a new event enters the queue

Polling Timeout (Seconds)

pollTimeoutSec

Worker thread waiting period (seconds) for the next event, before going idle

Shutdown Timeout (Seconds)

shutdownTimeoutSec

Application waiting period (seconds) for worker thread termination

JNDI Configuration

jndiConfiguration

Java Naming and Directory Interface (JNDI) Configuration Settings

JNDI Context Properties

contextProperties

Settings to populate the JNDI initial context with

JNDI Context Factory

java.naming.factory.initial

Initial JNDI context factory, such as com.tibco.tibjms.naming.TibjmsInitialContextFactory

JNDI Provider URL

java.naming.provider.url

Depends on provider; options include tcp://localhost:61616 and tibjmsnaming://192.168.1.133:7222

JNDI Topic

topic.audit

Relevant JNDI topic; default=audit

JNDI Topic Name

topicName

JNDI lookup name for the JMS topic

Connection Factory

connectionFactoryName

JNDI lookup name for the JMS connection factory

The Elasticsearch audit event handler is relatively complex, with config subcategories for connection, indexMapping, buffering, and topics.

Elasticsearch Audit Event Handler Unique config Properties
UI Label / Text audit.json File Label Description

Connection

connection

Elasticsearch audit event handler

useSSL

useSSL

Connection: Use SSL/TLS to connect to Elasticsearch

host

host

Connection: Hostname or IP address of Elasticsearch (default: localhost)

port

port

Connection: Port used by Elasticsearch (default: 9200)

username

username

Connection: Username when Basic Authentication is enabled via Elasticsearch Shield

password

password

Connection: Password when Basic Authentication is enabled via Elasticsearch Shield

Index Mapping

indexMapping

Defines how an audit event and its fields are stored and indexed

indexName

indexName

Index Mapping: Index Name (default=audit). Change if 'audit' conflicts with an existing Elasticsearch index

Buffering

buffering

Configuration for buffering events and batch writes (increases write-throughput)

enabled

enabled

Buffering: recommended

maxSize

maxSize

Buffering: Fixed maximum number of events that can be buffered (default: 10000)

Write Interval

writeInterval

Buffering: Interval (default: 1 s) at which buffered events are written to Elasticsearch (units of 'ms' or 's' are recommended)

maxBatchedEvents

maxBatchedEvents

Buffering: Maximum number of events per batch-write to Elasticsearch for each Write Interval (default: 500)