Home > Directory Server, OpenDJ > The Case of the Mysteriously Creeping Database

The Case of the Mysteriously Creeping Database

smiley_confused

While teaching a recent ForgeRock OpenDJ class, a student of mine observed an interesting behavior that at first seemed quite odd.  While rebuilding his attribute indexes, the student found that the overall database size seemed to grow each time he performed a reindex operation.  What seems obvious to me now sure made me scratch my head as I scrambled for an answer.  I am sharing my findings here in the hopes that others will either a) find this information useful or b) find comic relief as to my misfortune.

Note:  If you are unclear about the information contained in OpenDJ’s database files, then I highly recommend that you read my posting entitled, Unlocking the Mystery behind the OpenDJ User Database.  In that article I describe the overall structure of the Berkeley DB Java Edition database used by OpenDJ and how both entries (and indexes) are maintained in the same database.

In OpenDJ, the rebuild-index command is used to update any attribute indexes contained in the OpenDJ database. This is necessary after you make a configuration change that affects indexes (such as modifying the index entry limit). Indexes are database specific and you can elect to rebuild a single attribute index or rebuild all attribute indexes for a particular database.

The following syntax is used to rebuild ALL indexes associated with the dc=example,dc=com suffix and its use is what caused the frantic head-scratching to occur:

$ rebuild-index -h ldap.example.com -p 4444 -D "cn=Directory Manager" -w password -b dc=example,dc=com --rebuildAll

Mystery

The student observed (and questioned) that every time he rebuilt the indexes, the aggregated size of the *.jdb files actually increased by some factor. In the case of a rebuild-all, it was about 18 MB each time he ran the command; in the case of rebuilding a single index, it was only about 3 MB each time.  But the increase was consistent each time he rebuilt the index(es). This continued to occur until it reached a certain size at which time the consumption fell back to its original size (in our observations this occurred at roughly 200 MB when using the rebuild-all option).

The following details the output of the du -sh command on the userRoot database each time the rebuild-index command was run:

  • 124 MB
  • 142 MB (+ 18MB)
  • 160 MB (+ 18MB)
  • 178 MB (+ 18MB)
  • 200 MB (+ 22MB)
  • 123 MB (- 77MB)

This trend was consistent over several iterations.

We continued testing and observed that in addition to the increasing size, the database files on the file system (*.jdb) were changing as well. What was once 000000001.jdb and 000000002.jdb now became 000000002.jdb file and 000000003.jdb and later became 000000003.jdb file and 000000004.jdb. This occurred at the same time that we dropped back down to the 123 MB size and was the clue that unlocked the mystery.

Solution

Unlike the Berkeley Sleepycat database used in OpenDJ’s forefathers, when data is modified in the OpenDJ database, it is not immediately removed from the database. Instead it is marked for removal and the record essentially becomes inactive. Updated records are then appended to the end of the database in a log file fashion.

thumbs_up_smiley

This process continues until OpenDJ cleaner threads detect that a database file contains less than 50% active records. Once that occurs, the cleaner threads migrate all active records from the file and append them to the end of the last file in the OpenDJ database (a new file is created if necessary). Once migrated, the cleaner threads delete the database file containing the stale entries.

During the rebuild process, old index values in each of the *.jdb files are marked as inactive and new indexes are added to the database. Simply marking these indexes as inactive does not eliminate their existence in the database and they continue to consume disk space. This process continues until the point where the cleaner threads detect that old indexes account for > 50% of the database entries.  At this point, the migration process occurs, new *.jdb files are created to store the new indexes, old stale *.jdb files are deleted (hence the *.jdb file name changes), and the disk space is returned.

Word from ForgeRock’s OpenDJ Product Manager, Ludo Poitou confirmed my suspicions in the following response:

When an index is rebuilt, the whole btree is marked as deleted. But since it actually represents specific records of the database files, they will only be collected when the file itself reaches the threshold that triggers recollection.

With small databases, you will see the behavior you’re describing. With larger databases, this will be less noticeable as the amount of index records will be larger and cleanup point may be reached faster.

So there you go, mystery solved!

  1. Mayuri Mehrotra
    March 27, 2016 at 2:30 pm

    Hi, Would you know how data is stored on the disk in the case of kvstore for oracle nosql database. It has got something to do with jdb files but I am unable to figure out what exactly.
    Thanks!

    • idmdude
      March 28, 2016 at 7:17 am

      Hello Mayuri,

      I am not familiar with the kvstore for Oracle nosql. I did, however, take a quick look at a couple of things and found a nice introduction at https://docs.oracle.com/cd/E26161_02/html/AdminGuide/introduction.html and some administration notes at https://docs.oracle.com/cd/E26161_02/html/AdminGuide/procedures.html.

      From what I can tell, the kvstore and OpenDJ both use the Oracle Berkeley DB Java Edition found here http://www.oracle.com/technetwork/database/berkeleydb/overview/index-093405.html. As such, they both use .jdb database files and that is probably as close of a comparison that you can make. It is the software that sits on top of the database that defines what is stored in the database and how it might appear.

      If you are on a Linux system, one thing that you can try is using the “strings” command to see what you can see in the .jdb files. It won’t translate everything 100%, but it should give you a little more insight into the contents of a .jdb file. Try “strings 00000000.jdb | more” and you should see an ASCII representation of the data stored in those files.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: