Auditing and Monitoring OpenIG

Routes have a monitor boolean attribute that you can use to have OpenIG collect statistics for the route. The statistics are then exposed as a JSON resource that you can access over HTTP. In addition, OpenIG supports the ForgeRock common audit framework. You can add an audit service to a route, and the service can then publish messages to a consumer such as a CSV file, a relational database, or the Syslog facility. In this chapter, you will learn to:

Enable monitoring for a route
Read monitoring statistics for a route as a JSON resource
Add an audit service to a route to integrate with the ForgeRock common audit event framework, sometimes referred to as Common Audit

Monitoring a Route

To monitor a route, you set "monitor": true in the top-level attributes on the route. The value of the attribute can be an expression that evaluates to a boolean, such as "monitor": "${true}", or an object that indicates the percentile thresholds. By using an appropriate boolean expression, for example, you can enable or disable monitoring with an environment variable or system property. For details, see Expressions(5) in the Configuration Reference and Route(5) in the Configuration Reference.

The following example route has monitoring enabled:

{
  "handler": {
    "type": "StaticResponseHandler",
    "config": {
      "status": 200,
      "reason": "OK",
      "entity": "Hello, world!"
    }
  },
  "monitor": "${true}",
  "condition": "${matches(request.uri.path, '^/monitor')}"
}

Before you try this route, prepare OpenIG and the minimal HTTP server as shown in Getting Started.

Add the route file to the OpenIG configuration as $HOME/.openig/config/routes/00-monitor.json (on Windows, %appdata%\OpenIG\config\routes\00-monitor.json).

With the route in place and OpenIG running, access the route a few times at http://openig.example.com:8080/monitor.

Your access causes OpenIG to collect monitoring statistics for the route. After generating statistics by accessing the route a few times, read the JSON monitoring resource for the route at http://openig.example.com:8080/openig/api/system/objects/router-handler/routes/00-monitor/monitoring. The monitoring resource provides statistics on requests and responses as in the following example:

{
    "requests": {
        "total": 100,
        "active": 0
    },
    "responses": {
        "total": 100,
        "info": 0,
        "success": 100,
        "redirect": 0,
        "clientError": 0,
        "serverError": 0,
        "other": 0,
        "errors": 0,
        "null": 0
    },
    "throughput": {
        "mean": 15.6,
        "lastMinute": 20.0,
        "last5Minutes": 20.0,
        "last15Minutes": 20.0
    },
    "responseTime": {
        "mean": 0.093,
        "median": 0.046,
        "standardDeviation": 0.371,
        "total": 9,
        "percentiles": {
            "0.999": 3.762,
            "0.9999": 3.762,
            "0.99999": 3.762
        }
    }
}

For details about the content of the monitoring resource, see "The REST API for Monitoring" in the Configuration Reference.

By default, monitoring statistics are accessible from the local host. OpenIG uses an ApiProtectionFilter that protects the reserved routes for paths under /openig. By default, the filter allows access to reserved routes only from the local host. You can override this behavior by declaring a custom ApiProtectionFilter in the top-level heap. For an example, see the CORS filter described in Setting Up OpenIG As an UMA Resource Server.

Recording Audit Event Messages

The ForgeRock common audit framework is a platform-wide infrastructure to handle audit events by using common audit event handlers. The handlers record events by logging them into files, relational databases, or syslog. OpenIG provides audit event handlers to write messages to the following formats:

CSV files, with support for retention, rotation, and tamper-evident logs.
Relational databases using JDBC.
The UNIX system log (Syslog) facility.
The Elasticsearch search and analytics engine.

For Elasticsearch downloads and installation instructions, see the Elasticsearch Getting Started document.

Each audit event is identified by a unique transaction ID that can be communicated across products and recorded for each local event. By using the transaction ID, requests can be tracked as they traverse the platform, making it easier to monitor activity and to enrich reports.

Transaction IDs from other services in the ForgeRock platform are sent as X-ForgeRock-TransactionId header values.

By default, OpenIG does not trust transaction ID headers from client applications.

If you trust transaction IDs sent by client applications, and want monitoring and reporting systems consuming the logs to allow correlation of requests as they traverse multiple servers, then set the boolean system property org.forgerock.http.TrustTransactionHeader to true in the Java command to start the container where OpenIG runs.

To enable the audit framework for a route, you specify an audit service and configure an audit event handler. The following procedures describe how to record audit events in a CSV file and to the Elasticsearch search and analytics engine. For more information about recording audit events, see Audit Framework in the Configuration Reference.

To Record Audit Events In a CSV File

Before you start, prepare OpenIG and the minimal HTTP server as shown in Getting Started.

Add the following route to the OpenIG as $HOME/.openig/config/routes/30-audit.json.

On Windows, add the route as %appdata%\OpenIG\config\routes\30-audit.json.

{
  "handler": "ForgeRockClientHandler",
  "baseURI": "http://app.example.com:8081",
  "condition": "${matches(request.uri.path, '^/audit')}",
  "auditService": {
    "type": "AuditService",
    "config": {
      "config": {},
      "event-handlers": [
        {
          "class": "org.forgerock.audit.handlers.csv.CsvAuditEventHandler",
          "config": {
            "name": "csv",
            "logDirectory": "/tmp/logs",
            "buffering": {
              "enabled": "true",
              "autoFlush": "true"
            },
            "topics": [
              "access"
            ]
          }
        }
      ]
    }
  }
}

The route calls an audit service configuration for publishing log messages to the CSV file, /tmp/logs/access.csv. When a request matches audit, audit events are logged to the CSV file.

The route uses the ForgeRockClientHandler as its handler, to send the X-ForgeRock-TransactionId header with its requests to external services.

Access the route on http://openig.example.com:8080/audit.

The home page of the minimal HTTP server should be displayed and the file /tmp/logs/access.csv should be created.

To Record Audit Events In Elasticsearch

Before you start, make sure that Elasticsearch is installed and running. For Elasticsearch downloads and installation instructions, see the Elasticsearch Getting Started document. For information about configuring the Elasticsearch event handler, see ElasticsearchAuditEventHandler(5) in the Configuration Reference.

Add the following route to the OpenIG as $HOME/.openig/config/routes/30-elasticsearch.json.

On Windows, add the route as %appdata%\OpenIG\config\routes\30-elasticsearch.json.

{
  "MyCapture": "all",
  "auditService": {
    "name": "audit-service",
    "type": "AuditService",
    "config": {
      "config": {},
      "enabled": true,
      "event-handlers": [
        {
          "class": "org.forgerock.audit.handlers.elasticsearch.ElasticsearchAuditEventHandler",
          "config": {
            "name": "elasticsearch",
            "topics": [
              "access"
            ],
            "connection": {
              "useSSL": false,
              "host": "localhost",
              "port": 9200
            },
            "indexMapping": {
              "indexName": "audit"
            },
            "buffering": {
              "enabled": true,
              "maxSize": 10000,
              "writeInterval": "250 millis",
              "maxBatchedEvents": 500
            }
          }
        }
      ]
    }
  },
  "condition": "${matches(request.uri.path, '^/elasticsearch')}",
  "handler": {
    "type": "StaticResponseHandler",
    "config": {
      "entity": "View audit events in Elasticsearch at\rhttp://localhost:9200/audit/access/_search?q='\"OPENIG-HTTP-ACCESS\"'",
      "reason": "found",
      "status": 200,
      "headers": {
        "content-type": [
          "text/plain"
        ]
      }
    }
  }
}

The route calls an audit service configuration for publishing log messages in Elasticsearch. When a request matches the /elasticsearch route, audit events are logged to the ElasticsearchAuditEventHandler.

The URL where you can view the messages logged by Elasticsearch is displayed. The URL is constructed from the host, port, index name, and topics defined in the event handler.

Access the route on http://openig.example.com:8080/elasticsearch.

The audit events are logged in Elasticsearch and the URL where you can view the messages is displayed.
Access the URL http://localhost:9200/audit/access/_search?q='"OPENIG-HTTP-ACCESS".

The audit events logged in Elasticsearch are displayed.
Repeat the previous two steps again to access the OpenIG route and then the Elasticsearch URL.

Each time you access the OpenIG route, the audit events logged in Elasticsearch should be updated.