Troubleshooting Server Problems

In order to solve your problem methodically, save time by defining the problem clearly up front. In a replicated environment with multiple directory servers and many client applications, it can be particularly important to pin down not only the problem (difference in observed behavior compared to expected behavior), but also the circumstances and steps that lead to the problem occurring. Answer the following questions:

Pinpointing the problem can sometimes indicate where you should start looking for solutions.

Installation and upgrade procedures result in a log file tracing the operation. The log location differs by operating system, but look for lines in the command output of the following form:

Prevent antivirus and intrusion detection systems from interfering with OpenDJ directory server.

Antivirus and intrusion detection systems that do a deep inspection of database files are not compatible with OpenDJ directory server. Disable antivirus and intrusion detection systems, or at least prevent them from operating on OpenDJ directory server files.

This section describes what to do if you forgot the password for Directory Manager or for the global (replication) administrator.

OpenDJ can write debug information and stack traces to the server debug log. What is logged depends both on debug targets that you create, and on the debug level that you choose.

Misconfiguration can potentially put OpenDJ in a state where you must intervene, and where you need to prevent users and applications from accessing the directory until you are done fixing the problem.

OpenDJ provides a lockdown mode that allows connections only on the loopback address, and allows only operations requested by root users, such as cn=Directory Manager. You can use lockdown mode to prevent all but administrative access to OpenDJ in order to repair the server.

To put OpenDJ into lockdown mode, the server must be running. You cause the server to enter lockdown mode by using a task. Notice that the modify operation is performed over the loopback address (accessing OpenDJ on the local host):

OpenDJ logs a notice message in logs/errors when lockdown mode takes effect:

Client applications that request operations get a message concerning lockdown mode:

You also leave lockdown mode by using a task:

OpenDJ also logs a notice message when leaving lockdown:

By default OpenDJ requires that LDIF data you import respect standards. In particular, OpenDJ is set to check that entries to import match the schema defined for the server. You can temporarily bypass this check by using the --skipSchemaValidation with the import-ldif command.

OpenDJ also ensures by default that entries have only one structural object class. You can relax this behavior by using the advanced global configuration property, single-structural-objectclass-behavior. This can be useful when importing data exported from Sun Directory Server. For example, to warn when entries have more than one structural object class instead of reject such entries being added, set single-structural-objectclass-behavior:warn as follows:

By default, OpenDJ also checks syntax for a number of attribute types. Relax this behavior by using the dsconfig set-attribute-syntax-prop command. Use the --help option for further information.

When running import-ldif, you can use the -R rejectFile option to capture entries that could not be imported, and the --countRejects option to return the number of rejected entries as the import-ldif exit code.

Once you work through the issues with your LDIF data, reinstate the default behavior to ensure automated checking.

In order to trust the server certificate, client applications usually compare the signature on certificates with those of the Certificate Authorities (CAs) whose certificates are distributed with the client software. For example, the Java environment is distributed with a keystore holding many CA certificates:

The self-signed server certificates that can be configured during OpenDJ setup are not recognized as being signed by any CAs. Your software therefore is configured not to trust the self-signed certificates by default. You must either configure the client applications to accept the self-signed certificates, or else use certificates signed by recognized CAs.

You can further debug the network traffic by collecting debug traces. To see the traffic going over TLS/SSL in debug mode, configure OpenDJ to dump debug traces from javax.net.debug into the logs/server.out file:

By default OpenDJ logs information about all LDAP client operations in logs/access, and all HTTP client operations in logs/http-access. The following lines are wrapped for readability, showing a search for the entry with uid=bjensen as traced in the LDAP access log. In the access log itself, each line starts with a time stamp:

As you see, each client connection and set of LDAP operations are traced, starting with a time stamp and information about the operation performed, then including information about the connection, the operation number for the sequence of operations performed by the client, a message identification number, and additional information about the operation.

To match HTTP client operations with related internal server operations, first prevent OpenDJ from suppressing internal operations from the LDAP access log by using the dsconfig command to set the LDAP access log publisher suppress-internal-operations advanced property to false. Then match the values of the x-connection-id field in the HTTP access log with conn=id values in the LDAP access log.

For example, consider an HTTP GET request for the _id field of the user newuser, which is handled by connection 4 as shown in logs/http-access:

With internal operations logged in logs/access, log lines for the related operations have conn=4:

To help diagnose errors due to access permissions, OpenDJ supports the get effective rights control. The control OID, 1.3.6.1.4.1.42.2.27.9.5.2, is not allowed by the default global ACIs. You must therefore add access to use the get effective rights control when not using it as Directory Manager.

Replication can generally recover from conflicts and transient issues. Replication does, however, require that update operations be copied from server to server. It is therefore possible to experience temporary delays while replicas converge, especially when the write operation load is heavy. OpenDJ’s tolerance for temporary divergence between replicas is what allows OpenDJ to remain available to serve client applications even when networks linking the replicas go down.

In other words, the fact that directory services are loosely convergent rather than transactional is a feature, not a bug.

That said, you may encounter errors. Replication uses its own error log file, logs/replication. Error messages in the log file have category=SYNC. The messages have the following form. Here the line is folded for readability:

OpenDJ maintains historical information about changes in order to bring replicas up to date, and to resolve replication conflicts. To prevent historical information from growing without limit, OpenDJ purges historical information after a configurable delay (replication-purge-delay, default: 3 days). A replica can become irrevocably out of sync if you restore it from a backup archive older than the purge delay, or if you stop it for longer than the purge delay. If this happens to you, disable the replica, and then reinitialize it from a recent backup or from a server that is up to date.

When you cannot resolve a problem yourself, and want to ask for help, clearly identify the problem and how you reproduce it, and also the version of OpenDJ you use to reproduce the problem. The version includes both a version number and also a build time stamp:

Be ready to provide the following additional information: