Archive for the ‘Technology’ Category

Unlocking the Mystery behind the OpenDJ User Database

June 8, 2012 1 comment

One question that arises time and time again pertains to the manner in which OpenDJ stores it entry data and how this differs from the Oracle Directory Server Enterprise Edition (previously known as Sun Directory Server Enterprise Edition).

The following information is from ForgeRock’s OpenDJ Administration, Maintenance and Tuning Class and has been used with the permission of ForgeRock.

OpenDJ includes the Berkeley DB Java Edition database as the backend repository for user data.  The Java version is quite different from the Berkeley C version which is used by the Sun Directory Server Enterprise Edition.

The Berkeley DB Java Edition is a Java implementation of a raw database using the B-Tree technology.  A Berkeley DB JE environment can be composed of multiple databases, each of which is stored in a single folder on the file system. Rather than having separate files for records and transaction logs, Berkeley DB JE uses a rolling log file to store everything; this includes the B-Tree structure, the user provided records and the indexes.  Write operations append entries as the last items in the log file.  When a certain size is reached (10MB by default), a new log file is created.  This results in consistent write performance regardless of the database size.

Note: Initial log files are located beneath the db/userRoot folder in the installation directory.  The initial log file is 00000000.jdb.  When that file reaches a size of 10MB, a new file is created as 00000001.jdb.

Over time records are deleted or modified in the log.  OpenDJ performs periodic cleanup of log files and rewrites them to new log files.  This task is performed without action by a system administrator and ensures consistency of the data contained in the log files.

You can see a list of all entries contained in the database with the dbtest utility.  This command returns the entry specific information that can also be used to debug the backend.

The following diagram demonstrates the periodic processing of the Berkeley DB Java Edition database over time.

Log (Entry) Processing

The log shown at the far left (row 1, column 1) contains entries after an immediate population.  You will note that it contains five data entries (Entry 1 through Entry 5) as well as the associated index entries.  To keep things simple, only the common name and object class attributes have been indexed (as shown by the cn and OC keys).

Note:  The data does not appear in this actual format.  This notation is used for demonstration purposes.

As time goes by, data in the database changes.  New entries are added, attribute values for existing entries are modified, and some entries are deleted.  This has an effect on both the data entries in the log as well as any associated index entries.

The second log (row 1, column 2) demonstrates the effect on the database after Entry 3 has been modified.  The modified data entry is written to the end of the log and the original entry is marked for deletion.  The modification was made to an attribute that that was not indexed so the log does not contain any modification to the index entries.

Changes made to data entries containing indexed attributes would not only appear at the end of the log, but the modifications to the indexes would appear there as well.  This can be seen in the third log (row 1, column 3).  Entry 2 was modified and the change involved a modification to the common name (cn) attribute.  Note how the previous index value for this entry is marked for deletion and the new index entry is written to the end of the log.

The logs found at row 1, column 4 and row 2, column 4 demonstrate how new log files are created as previous ones reach their limit.  Write operations are appended to the log file in a linear fashion until it reaches a maximum size of 10 MB at which time a new log is created.  This can be seen by the 00000001.jdb log in row 2, column 4.

Note: The maximum log size of 10 MB is defined in the ds-cfg-db-log-file-max attribute contained in the backend definition.

Database Cleanup

The Berkeley C Database simply purged data from the database.  This led to fragmentation (“holes”) in the database and required periodic cleanup by system administrators to eliminate the holes (similar to defragmenting your hard drive).  This is not the case with the Berkeley DB Java Edition.

The Berkeley DB Java Edition has a number of threads that periodically check the occupancy of each log.  If it detects that the size associated with active entries falls below a certain threshold (50% of its maximum size, or 5MB by default), it rewrites the active records to the end of the latest log file and deletes the old log altogether.  This can be seen in the log found at row 2, column 5.

Note: Maximum occupancy is defined in the ds-cfg-db-cleaner-min-utilization attribute contained in the backend definition.

The default occupancy is 50% so at its maximum size, the log will be twice as big as its sum of records. Increasing the occupancy % will reduce the log’s size, but induce more copying, thus increasing CPU utilization.

This process of always appending data to the end of the log and periodically rewriting the log as entries are obsoleted allows OpenDJ to maintain a fairly consistent size – even if entries are heavily modified.  It does, however, allow the database to shrink in size if many entries are deleted.

Check out ForgeRock’s website for more information on OpenDJ or click here if you are interested in attending one of ForgeRock’s upcoming training classes.

The Most Complete History of Directory Services You Will Ever Find

April 13, 2012 19 comments
I started working with Directory Servers back in 1997 when Netscape was but a fledging company. Over the past 20 years a lot has changed. Companies have come and gone and code has changed hands more times than I care to remember. But one thing remains the same – that little effort started by Tim Howes, Mark Smith, and Gordon Good at the University of Michigan is as important today as it was two decades ago.
I thought it might be worthwhile to take a look back at the various companies that have carried the LDAP mantle for stand-alone directory servers and see where we are today. As such, I have created a table of pertinent events (see below) as well as a graphical timeline (see graphic).
I offer you the industry’s most most complete history of directory services that you will ever find – well, at least until the next one comes along.

Directory Services Timeline

The Most Complete History of Directory Services You Will Ever Find

(Until the next one comes along)




1969First Arpanet node comes online; first RFC published.


1973Ethernet invented by Xerox PARC researchers.


1982TCP/IP replaces older Arpanet protocols on the Internet.


1982First distributed computing research paper on Grapevine published by Xerox PARC researchers.


1984Internet DNS comes online.


1986IETF formally chartered.


1989Quipu (X.500 software package) released.


1990Estimated number of Internet hosts exceeds 250,000.


1990First version of the X.500 standard published.


1991A team at CERN headed by Tim Berners-Lee releases the first World Wide Web software.


1992University of Michigan developers release the first LDAP software.


1993NDS debuts in Netware 4.0.


July 1993LDAP specification first published as RFC 1487.


December 1995First standalone LDAP server (SLAPD) ships as part of U-M LDAP 3.2 release.


April 1996Consortium of more than 40 leading software vendors endorses LDAP as the Internet directory service protocol of choice.


1996Netscape Hires Tim Howes, Mark Smith, and Gordon Good from University of Michigan.  Howes serves as a directory server architect.


September 1997Sun Microsystems releases Sun Directory Services 1.0, derived from U-M LDAP 3.2


November 1997LDAPv3 named the winner of the PC Magazine Award for Technical Excellence.


December 1997LDAPv3 approved as a proposed Internet Standard.


1998The OpenLDAP Project was started by Kurt Zeilenga.  The project started by cloning the LDAP reference source from the University Of Michigan.


January 1998Netscape ships the first commercial LDAPv3 directory server.


March 1998Innosoft acquires Mark Walh’s Critical Angle company, relesases LDAP directory server product 4.1 one month later.


July 1998Sun Microsystems ships Sun Directory Server 3.1, implementing LDAPv3 standards


July 1998Estimated number of Internet hosts exceeds 36 million.


1999AOL acquires Netscape and forms the iPlanet Alliance with Sun Microsystems.


March 1999Innosoft team, led by Mark Wahl, releases Innosoft Distributed Directory Server 5.0


March 2000Sun Microsystems acquires Innosoft, merges Innosoft directory code with iPlanet.  This forms the foundation for the iPlanet Directory Access Router.


October 2001The iPlanet Alliance ends and Sun and Netscape fork the codebase.


October 2004Apache Directory Server Top Level Project is formed after 1 year in incubation


December 2004RedHat Purchases Netscape Server products


2005Sun Microsystems initiates the OpenDS project.  An open source directory server based on the Java platform.


June 2005RedHat Releases Fedora Directory Server


October 2006Apache Directory Server 1.0 is released


2007UnboundID releases its directory server


2008AOL Stops Supporting Netscape Products


April 2009Oracle purchases Sun Microsystems


May 2009RedHat changes the Fedora Directory Server to 389 Directory Server


Feb 1, 2010ForgeRock is founded


Dec 2010ForgeRock releases OpenDJ


July 2011Oracle releases Oracle Unified Directory



(1) Understanding and Deploying LDAP Directory Services; Second Edition; Timothy A. Howes, Ph.D., Mark C. Smith, and Gordon S. Good.
(2) 389 Directory Server; History (
(3) Email exchange with Ludovic Poitou (ForgeRock).
(4) Press Release, March 16th, 1998; “Innosoft Acquires LDAP Technology Leader Critical Angle Inc. (
(5) OpenLDAP; Wikipedia (
(6) iPlanet; Wikipedia (
(7) OpenDS; Wikipedia (
(8) Netscape; Wikipedia (
(9) Press Release, April 20th, 2000; “Oracle Buys Sun” (
(10) 389 Directory Server; 389 Change FAQ (
(11) OpenDJ; Wikipedia (
(12)  Email exchange with Nick Crown (UnboundID).
(13) Press Release, July 20th, 2011; “Oracle Announces Oracle Unified Directory 11g” (

The Road Back to Eden

March 28, 2012 1 comment

We oftentimes view new technologies as providing us with the path back to Eden – that which returns us to leisure and care free living.  Yet with each new technological improvement, our lives do not become easier, they are simply changed.

The widespread availability of electricity and electric appliances in the early 1900s, promised to transform the home from places of labor (cleaning, cooking, etc.) to places of ease.  But ironically,
the number of hours spent working in the home remained relatively unchanged; the only thing that did change was the nature of the work.  Instead of beating the rugs twice a year or ironing clothes for special occasions (tasks that used to take all day to complete), it was expected that our rugs would always be clean and our shirts would always be ironed.  Hence the amount of time spent on these activities was not really reduced, it was simply spread out over different times and expected to be completed more often.

Pagers were introduced in the 1970s and cell phones started becoming more common in the early 1980s.  Both of these technological breakthroughs promised to free us from the office and allow us to spend more time with our families and friends.  The reality, however, was much different from the from the promise.  Instead of spending time with our loved ones, these electronic leashes simply changed where and how our work was to be performed.  We were now expected to be available at anytime and oftentimes at a moments notice.  With the advent of remote connectivity and mobile computing devices, we are expected to perform our work from anywhere – just as if we were in the office.  Once again the nature of our work has changed as our homes (or even the road) have become our second offices and our availability is not our own.

In the late 1960s the Internet was envisioned as a “cybernetic meadow” where humans and machines could live together in harmony.  A poem by Richard Brautigan described a future where

we are free of our labors
and joined back to nature…

where deer stroll peacefully
past computers
as if they were flowers
with spinning blossoms.

…and all watched over
by machines of loving grace.

(“All Watched Over By Machines Of Loving Grace“, Richard Brautigan)

The reality of what the Internet has become is very different from the original idealistic vision but as with the introduction of other technologies, the Internet is wondrous indeed.  The Internet has become a vast “computer in the cloud” as described by Google’s former CEO, Eric Schmidt, and with that computer you are able to access almost any type of application and/or information at the touch of a button.  This can be demonstrated in a commercial for telecommunications carrier, Qwest Communications.  A weary traveler walks into a cheap hotel in the middle of nowhere.  As expected, the amenities are sparse, but when he asks about in-room entertainment, the clerk responds, “all rooms have every movie ever made in any language anytime, day or night.”

Is this amazing?  Yes.

Does this allow us to do things previously unimagined?  Yes.

Does this move us closer to leisure and Eden, itself?  No.

In fact, we are becoming deluged with data as not only do we have access to almost any type of information, but we do so at any time of day and from almost any device.  Facebook status, Twitter feeds, unending SMS messages are filling our lives with information; like Pavlov’s dog, we are forced to respond every time we hear the bell ring.  As humans we have a natural tendency to explore, but it is not possible to explore everything on the Internet.  As humans we have a natural tendency to socialize, but its not possible to keep up to date with all our friends who we are now connected with 24 hours a day.  But that does not stop us from trying.

It is a natural response to view new technologies in an optimistic way, but we must be careful not to look to technology as the ultimate savior or it could end up hastening our ultimate demise.  Norbert Wiener, the Father of Cybernetics explored the relationship between technology and humans in his ground-breaking book, Cybernetics: Or, Control and Communication in the Animal and the Machine. While he hoped that machines would augment people’s lives and free them towards more creative pursuits, he warned that humans would look to technology as a panacea and ultimately lead to our downfall.

“The simple faith in progress is not a conviction belonging to strength, but one belong to acquiescence and hence to weakness.”

While technology can be amazing and allow us to do wondrous things, technology in and of itself is not the answer.  It is simply a means to an end.  As with any other tool, you need to manage the tool and not allow it to manage you.  You need to periodically set technology aside and focus on those things that truly matter – your God, your family, your friends, and yourself.

That is the only way you will truly get back to Eden.