I’ve moved my blog

I’ve decided to move my blog to a domain that I own to allow me to have more control over the appearance and content. My new blog is now at http://www.dirmgr.com/.

I’ve moved all of my existing content there and don’t intend to post here anymore. Please update your bookmarks, RSS feeds, etc. accordingly.

LDAPCon recorded presentation posted

I have now updated the UnboundID website to include a recorded version of the presentation that I gave at LDAPCon. The total presentation is right at 45 minutes, but I’ve broken it up into a few pieces so that you don’t have to try to digest it all at once.

You can find the recorded presentation, as well as a copy of the slides, at http://www.unboundid.com/products/ldapsdk/LDAPCon-2009-presentation/index.php.

LDAPCon slides posted

The LDAP SDK presentation that I gave Monday at LDAPCon went pretty smoothly and seemed to be pretty well-received. When I got home yesterday afternoon, I recorded a version of the presentation and we’ll hopefully be making that available online in the near future. However, a version of the slides that I used for the presentation is now available online at http://www.symas.com/ldapcon2009/papers/ely1.shtml.

Slides and/or papers for some of the other presentations are also available online. You can see them at http://www.symas.com/ldapcon2009/papers.shtml.

LDAP SDK now in Maven Central Repository

Just in time for my talk at LDAPCon on Monday, it looks like the UnboundID LDAP SDK for Java is now available in the Maven Central Repository.

You can get it with the following information:

• groupId — com.unboundid
• artifactId — unboundid-ldapsdk
• version — 1.1.0

The link to the POM file for this release is at http://repo1.maven.org/maven2/com/unboundid/unboundid-ldapsdk/1.1.0/unboundid-ldapsdk-1.1.0.pom.

UnboundID LDAP SDK for Java 1.1.0

UnboundID has released version 1.1.0 of the UnboundID LDAP SDK for Java. This is a significant update over the previous 1.0.0 release, both in terms of functionality and project policies. You can see the release notes for more detailed information about the changes, but some of the most significant updates include:

• The LDAP SDK is now covered by the GNU Lesser General Public License (LGPL) version 2.1. This is an addition to the existing GPLv2 and UnboundID LDAP SDK Free Use Licenses that were already in use for the project. LGPLv2.1 is an OSI-approved open source license that is very friendly for use in both open source and proprietary applications.

• As previously mentioned, the LDAP SDK is now available on SourceForge, including a public source repository, mailing lists, forums, and download files.

• The LDAP SDK has been updated to make it easier for use with the Apache Maven build system. It should be included in the Central Repository in the near future, but in the meantime, you can use a repository URL of http://ldap-sdk.sourceforge.net/maven/ with a groupId of “com.unboundid”, an artifactId of “unboundid-ldapsdk”, and a version of “1.1.0″.

• The LDAP SDK jar file has been updated so that its manifest includes the appropriate headers to make it easier to use with the OSGi module system.

• We have added code to help make it easier to migrate applications written using the Netscape Directory SDK for Java or JNDI to the UnboundID LDAP SDK for Java. The LDAP SDK contains a number of analogs for classes in the Netscape Directory SDK for Java, and in many cases you can simply change import statements from “netscape.ldap” to “com.unboundid.ldap.sdk.migrate.ldapjdk” and recompile your code. If you’re currently using JNDI, then you can use classes in the “com.unboundid.ldap.sdk.migrate.jndi” package to aid in the conversion process.

• The LDAP SDK has been updated to make it easier to have server schema information taken into account when reading responses. A new “use schema” connection option has been defined, and when it is enabled then the LDAP SDK will use the server schema to select appropriate matching rules for attributes in search result entries. A number of other schema-related improvements have also been made.

• The searchrate, modrate, and authrate tools have been updated to provide the ability to load-balance requests across multiple servers, to include timestamps in the output, and to run for a specified number of warm-up intervals before beginning overall statistics collection.

LDAP SDK now on SourceForge

Earlier today, I created a SourceForge project for the UnboundID LDAP SDK for Java. You can find it at http://sourceforge.net/projects/ldap-sdk/. This project contains a subversion repository containing the source code for the Standard Edition, as well as a set of mailing lists and a forum and a download of the current 1.0.0 release. You can now easily check out and build the LDAP SDK for yourself.

This makes it easy to see what we’re working on without having to wait for the next release (which currently includes things like OSGi support and easier migration from other Java-based LDAP APIs), and if you do run into a problem of some kind that requires a code fix, you can get access to that fix without needing to wait for a new release and without requiring us to provide you with a private build.

UnboundID LDAP SDK for Java 1.0.0

UnboundID has released version 1.0.0 of the UnboundID LDAP SDK for Java. We have felt for a long time that it was more than stable enough for use in production applications (and in fact it is in use in production applications in several places), but had not yet declared it 1.0 just in case we received any feedback that might have caused us to consider altering some portion of the API in an incompatible way. Although we have gotten several comments and suggestions, and we have added a number of features in response to that feedback, there haven’t been any incompatible changes and we feel very confident that there won’t need to be any such changes in the future (and have an extensive test framework in place to ensure that doesn’t happen). We are therefore bumping the version number to 1.0.0 just in case the pre-1.0 version number caused anyone to shy away from using it.

A complete list of the changes since the 0.9.10 release can be found in the release notes, but some of the most significant changes include:

• We have added support for the permissive modify control, which may be used to request that the directory server not reject attempts to add an attribute value that already exists or remove an attribute value that does not yet exist.
• We have added a new LDAPThreadLocalConnectionPool class, which provides an alternate connection pool implementation that uses a separate connection per client thread that attempts to use the connection pool. This eliminates the need to explicitly configure the number of connections to maintain, and also eliminates contention between threads when trying to use the pool.
• We have added the ability to create a Schema object by reading definitions from one or more LDIF files. It was previously only possible to create a Schema object by reading it from a directory server over LDAP.
• We have updated the LDIF reader to provide the ability to reject entries that contain duplicate values for the same attribute in the same entry. We have also added the ability to use schema information when reading data from LDIF to help avoid incorrectly classifying values as duplicates (e.g., if two values differ only by capitalization, but are in an attribute type defined with a caseExactMatch equality matching rule).
• We have updated the validate-ldif tool so that it has the ability to use schema definitions read from files (allowing it to be used in a completely offline manner), and to take advantage of the ability to reject entries with duplicate attribute values (although this can be disabled if desired).

UnboundID Directory Proxy Server

Today, UnboundID is announcing the UnboundID Directory Proxy Server, which is a fast, scalable, robust, and easy-to-use LDAPv3 proxy server. It is designed to work with any LDAPv3 directory server, although some advanced functionality is intended for use with the UnboundID Directory Server.

This post describes some of the features and advantages we offer in the Directory Proxy Server product. For more information, contact sales@unboundid.com.

Advanced Health Checking

The UnboundID Directory Proxy Server offers advanced health checking capabilities that can detect and react to problems as soon as they happen, and in some cases even before they happen, so that clients are not adversely impacted by (and often won’t even notice) unexpected hardware, software, or networking failures.

Some of the things that the Directory Proxy Server can consider when monitoring the health of the backend servers include:

• The length of time required to process operations in the server.
• The existence and content of specified entries in the server.
• The replication backlog in the server (both in terms of the number of pending changes and the age of the oldest outstanding change).
• The level of busyness for that server (both for operations requested through the Directory Proxy Server, as well as through other routes).

In the event that a problem is detected with a backend server, then the Directory Proxy Server can de-prioritize that server relative to other servers in the environment, or can take it out of the mix altogether. Further, if the Directory Proxy Server is used in front of the UnboundID Directory Server, then the Directory Server instances can proactively notify the Directory Proxy Server of any problems that arise. This can help avoid problems that the Directory Proxy Server may not be able to detect on its own (e.g., running low on disk space on the Directory Server system) until they become critical problems resulting in operation failures.

Flexible Load Balancing

The UnboundID Directory Proxy Server offers a number of options for load balancing requests across multiple backend servers, which can help provide both high performance and high availability. Some of the things that it can take into account when deciding where to send a request include:

• The availability of the backend servers, so that servers which are experiencing a problem of some kind can be avoided.
• The relative healths of the backend servers, so that servers with a higher health check score can be given a proportionally larger percent of the load.
• The locations of the backend servers, so that servers in the same data center as the Directory Proxy Server will be preferred over those that are remote. If it is necessary to send requests to remote servers, then those locations can also be prioritized to prefer closer data centers over those that are farther away.
• Statically-defined weights assigned to individual servers or sets of servers.
• The type of operation being requested. If you want, you can configure different types of load balancing for each type of operation.
• Information about the way that previous requests have been handled, so that related requests are consistently routed to the same backend server.
• Information about the client issuing the request. You can define different policies for different kinds of clients, based on a wide range of criteria.

Scalability and Entry Balancing

The UnboundID Directory Proxy Server can help you improve the overall performance and scalability of your directory environment. The load balancing capabilities mentioned above allow you to spread requests across multiple servers with the same content to ensure high throughput and availability.

In order to achieve optimal performance, you need to be able to cache all of the data in memory. The very compact way in which the UnboundID Directory Server represents entry data both on disk and in memory ensures that any single instance can hold a large number of entries, but sometimes even that may not be enough and you may need to utilize more memory than is available in any single instance. For these cases, the UnboundID Directory Server offers entry balancing, which allows you to seamlessly spread data across multiple sets of servers to take advantage of the aggregate memory across all of those sets. You can do this without the need to alter the content or hierarchy of your data, and without restricting which attributes clients can use in their requests for optimal performance.

Simple Installation, Configuration, and Management

The process for installing and configuring the UnboundID Directory Proxy Server is faster easier than any other LDAP proxy server I have ever used, even for relatively complex deployments across multiple data centers or when using entry balancing. And because the Directory Proxy Server shares much of the same code base as the Directory Server, you can use the same graphical, interactive text-based, and non-interactive command-line interfaces to manage both products. If you’re already familiar with the UnboundID Directory Server, then you can be up and running with the Directory Proxy Server in no time. Even if you’re completely new to the products then you should find the administrative interfaces simple and intuitive, yet still very powerful.

Other Features

Some of the other capabilities offered by the UnboundID Directory Proxy Server include:

• Easily track requests through the directory environment. Both the Directory Server and the Directory Proxy Server access logs contain information that can easily allow you to correlate information about operations across multiple instances.
• Transform requests and responses passing through the Directory Proxy Server to match the way that clients expect to access the data, even if that isn’t how it actually appears in the backend servers.
• Define limits on what clients are allowed to do in the directory environment, including what types of operations they may request and which portions of the DIT they can access. As with load balancing, you can create multiple policies for different kinds of clients based on a very wide range of criteria.
• Allow administrators to easily access individual backend servers through the Directory Proxy Server (including access to things like the server configuration, schema, and monitor data), bypassing any load balancing and health checking processing that would otherwise be enforced for normal client requests.

SLAMD 2.0.0-20090712

A new build of SLAMD is available for download from http://www.slamd.com/. The release notes provide a full list of the changes since the previous 2.0.0-20090227 release, but some of the most significant changes are listed below.

• The process used to load classes has been updated so that if an attempt to load a class fails, it will re-try the attempt after substituting “com.sun.slamd” with just “com.slamd”. This makes it possible to view statistics collected from builds of SLAMD prior to the 2.0.0-20090227 release.

• Two new jobs have been added which make it possible to perform LDAP searches using filters read from a file and LDAP modifications with target entry DNs read from a file. These jobs can be used if the target search filters or entry DNs do not follow a pattern that can be constructed like the other jobs provide.

• A number of jobs have been updated to provide support for rate limiting. You can now specify the desired number of operations per second per client, and the clients will attempt to maintain that rate.

• A number of shell scripts used to launch tools have been updated to fix problems that prevented them from working properly in some cases.

• LDAP jobs have been updated to use the synchronous mode provided in the latest release of the LDAP SDK for Java. This allows them to be significantly more efficient and drive more load against LDAP directory servers.

UnboundID LDAP SDK for Java 0.9.10

The UnboundID LDAP SDK for Java version 0.9.10 has just been released. The release notes provide the complete set of changes, but this post will outline two of the most significant changes in this release.

Synchronous Mode

The LDAP SDK now includes an option to operate in synchronous mode. If an application using the LDAP SDK does not need to perform asynchronous operations, and if it does not attempt to process multiple operations on the same connection at the same time, then you can use the new synchronous mode to achieve a significant increase in throughput and reduction in response time. In my testing, I have generally seen a performance improvement of about 15-20% by enabling the option to use synchronous mode.

Prior to this release, the LDAP SDK only operated in an asynchronous manner, in which each connection had a dedicated reader thread that could be used to read responses from the server. Whenever a request was sent to the server, a blocking queue was used to provide communication between the reader thread and the thread that sent the request. After sending a request to the server, the thread that sent that request would block on this queue waiting for the response from the server to be provided by the reader thread. This design is necessary to be able to handle the possibility of asynchronous operations, in which multiple requests can be outstanding at the same time, but it does incur overhead as a result of thread synchronization and context switching. This asynchronous manner of communication is still the default, but if your application doesn’t need the ability to process multiple operations on the same connection at the same time, then you can choose to use the new synchronous mode for better performance.

To enable synchronous mode, all you need to do is to set the useSynchronousMode connection option on the connection before it is established. For example:

     LDAPConnectionOptions options = new LDAPConnectionOptions();
     options.setUseSynchronousMode(true);
     LDAPConnection connection =
          new LDAPConnection(options, host, port, bindDN, password);

You shouldn’t need to make any other changes to your application, since everything else should be completely transparent. If you do attempt to invoke an asynchronous operation (or attempt to perform an abandon or invoke a cancel extended request, which rely on the ability to perform asynchronous operations), then the SDK will throw an LDAP exception with a “not supported” result.

The synchronous mode is completely compatible with the use of connection pools, and in fact if you are currently using connection pools in your application then it’s very likely that your application meets the requirements for using synchronous mode, since it’s not all that easy to invoke asynchronous operations using pooled connections or to use a pooled connection concurrently across multiple threads.

There has been some other restructuring in the LDAP SDK to provide for this capability, and those underlying changes may also provide some level of benefit even for applications using the default asynchronous mode.

Spurious “Server Down” Responses

A user reported periodically getting “server down” responses, especially when reading large result sets. In the UnboundID LDAP SDK for Java, the “server down” response indicates that a previously-established connection is no longer usable, and generally indicates one of two things:

• The connection between the client and the server has been lost unexpectedly, most likely because of a network problem or because the server closed the connection.

• The client encountered an unrecoverable error while decoding a response from the server that prevents it from being able to perform any more communication over that connection.

Upon closer investigation, I did find that there was a corner case in which the LDAP SDK could experience a problem if it was in the process of decoding a response from the server in which the entire LDAP message for the response was not sent all at once but instead came in multiple pieces and there was a delay of at least 50 milliseconds (which is the default socket timeout in asynchronous mode, needed to handle a corner case for StartTLS processing) when trying to read a subsequent piece of the response message. In this case, the LDAP SDK had a problem in which it would try to start reading a completely new response rather than finishing the old response, and it would encounter an unrecoverable decoding error when trying to read in the middle of the old message as if it were the beginning of a new one.

This problem has been corrected, and the LDAP SDK will now properly handle this condition and is more tolerant of delays encountered when reading a response in multiple segments.