Archive
The Real Reason Oracle Dropped Sun Identity Manager
I always appreciate it when someone attempts to educate others about identity management and related technologies. So when I saw the the following presentation, it quickly caught my attention as I was working with both products when the Oracle deal to purchase Sun went down.
Why Oracle Dropped Waveset Lighthouse and Went to Oracle Identity Manager (OIM)
Not to be too nit picky, but there are quite a few errors in this presentation that I simply couldn’t ignore.
- OID is not the acronym for “Oracle Identity Management”, it is an acronym for “Oracle Internet Directory” – Oracle’s LDAPv3 directory server. OIM is the acronym for “Oracle Identity Manager”.
- SIM (“Sun Identity Manager”) was not a “suite of identity manager products” as you state. SIM was a data synchronization and provisioning product. SIM was part of the suite of Sun Identity Management products that also included Sun Directory Server Enterprise Edition (SDSEE), Sun Access Manager/OpenSSO, and Sun Role Manager (SRM).
- It is stated that one reason that Oracle did not elect to continue with SIM was because it did not have a Web 2.0 UI. SIM version 9.0 (the version being developed when Oracle purchased Sun) did have a Web 2.0 UI. So this is not quite an accurate representation.
- “Oracle IDM” is Oracle’s suite of identity management products which includes Oracle Virtual Directory (OVD), Oracle Identity Directory (OID), Oracle Access Manager (OAM), and Oracle Identity Manager (OIM). The presentation uses “Oracle IDM” (and later, simply “IDM”) to refer specifically to Oracle Identity Manager, however. This is both confusing and misleading.
- It is stated that “IDM allowed faster application on-boarding.” As an integrator of both OIM and SIM, I can honestly say that this is not a true statement. We could have simple SIM deployments up and running in the order of days/week and a production deployment in a month or two. OIM, consistently took several months to deploy – which was great for a billable professional services firm, but not so great for the customer (who had to pay for those services).
- It is inferred that OIM is able to provision to cloud and SIM was not and that was a reason why Oracle chose to go with OIM. That is a misleading statement as SIM was able to provision to cloud applications as well. SIM also supported SPML (not a big fan, myself) and SCIM for provisioning to other identity platforms and cloud based applications.
The main reasons that Oracle chose to go with OIM versus SIM was simply the deeper integration with Oracle products and their not wanting to alter the Oracle IDM roadmap. I was on the early calls with Oracle when they announced which products they would keep and which products they were getting rid of. During those calls, they had their “politically correct” reasons as well as the “real” reasons and it always came back to these two.
There was only one place where I saw Oracle forced into altering their position and they had to update their roadmap; this was with the SDSEE product. Oracle made it very clear that the only product they wanted in Sun’s identity product line was Sun Role Manager (which later became Oracle Identity Analytics). In fact, only a couple weeks after the purchase was made, Oracle had already set an end of life date for all identity products including SDSEE. What Oracle hadn’t counted on was how well entrenched that product was across Sun’s major customers (including the US Government and major Telcos). It wasn’t until the outcry from their customers was raised that Oracle “decided” to continue product development.
Purely from a technology perspective, if you are a company that has deployed a wide array of Oracle products, then it made sense to go with OIM due to the deeper integration with Oracle products, but not so much if you are a heterogenous company. In such cases, I have found other products to be more flexible than OIM and provide a much quicker deployment times at much lower costs.
OpenDJ and the Fine Art of Impersonation
Directory servers are often used in multi-tier applications to store user profiles, preferences, or other information useful to the application. Oftentimes the web application includes an administrative console to assist in the management of that data; allowing operations such as user creation or password reset. Multi-tier environments pose a challenge, however, as it is difficult to determine the identity of the user that actually performed the operation as opposed to the user that simply showed up in the log file(s).
Consider the relationship between the user logging in to the web application and the interaction between the web application and a directory server such as OpenDJ.
There are two general approaches that many web applications follow when performing actions against the directory server; I will refer to these as Application Access and User Access. In both scenarios, the user must first log in to the web application. Their credentials may be validated directly against the directory server (using local authentication) or they may be accessing the web application using single sign-on. In either pattern, the user must first prove his identity to the web application before they are allowed to perform administrative tasks. The differences become apparent post authentication and can be found in the manner in which the web application integrates with the directory server to perform subsequent administrative tasks.
Note: The following assumes that you are already familiar with OpenDJ access control. If this is not the case, then it is highly advisable that you review the following: OpenDJ Access Control Explained.
Approach 1: Application Access
In the case of the Application Access approach all operations against the directory server are performed as an application owner account configured in the directory server. This account typically has a superset of privileges required by all Web Application administrators in order to perform the tasks required of those users. In this scenario, the Web Application binds to the directory server using its Web Application service account and performs the operation. A quick look in the directory server log files demonstrates that all operations coming from the Web Application are performed by the service account and not the user who logged in to the Web Application.
[27/Mar/2015:16:37:40 +0000] BIND REQ conn=2053 op=0 msgID=1 version=3 type=SIMPLE dn=”uid=WebApp1,ou=AppAccounts,dc=example,dc=com”
[27/Mar/2015:16:37:40 +0000] BIND RES conn=2053 op=0 msgID=1 result=0 authDN=”uid=WebApp1,ou=AppAccounts,dc=example,dc=com” etime=1
[27/Mar/2015:16:37:40 +0000] SEARCH REQ conn=2053 op=1 msgID=2 base=”ou=People,dc=example,dc=com” scope=wholeSubtree filter=”(l=Tampa)” attrs=”ALL”
[27/Mar/2015:16:37:40 +0000] SEARCH RES conn=2053 op=1 msgID=2 result=0 nentries=69 etime=2
While easiest to configure, one drawback to this approach is that you need to reconcile the directory server log files with the Web Application log files in order to determine the identity of the user performing the action. This makes debugging more difficult. Not all administrators have the same access rights; so another problem with this approach is that entitlements must be maintained and/or recognized in the Web Application and associated with Web Application users. This increases complexity in the Web Application as those relationships must be maintained in yet another database. Finally, some security officers may find this approach to be insecure as the entry appearing in the log files is not indicative of the user performing the actual operation.
Approach 2: User Access
The User Access approach is an alternative where the Web Application impersonates the user when performing operations. Instead of the Web Application binding with a general service account, it takes the credentials provided by the user, crafts a user-specific distinguished name, and then binds to the directory server with those credentials. This approach allows you to manage access control in the directory server and the logs reflect the identity of the user that performed the operation.
[27/Mar/2015:17:01:01 +0000] BIND REQ conn=2059 op=0 msgID=1 version=3 type=SIMPLE dn=”uid=bnelson,ou=Administators,dc=example,dc=com”
[27/Mar/2015:17:01:01 +0000] BIND RES conn=2059 op=0 msgID=1 result=0 authDN=” uid=bnelson,ou=Administators,dc=example,dc=com ” etime=1
[27/Mar/2015:17:40:40 +0000] SEARCH REQ conn=2059 op=1 msgID=2 base=”ou=People,dc=example,dc=com” scope=wholeSubtree filter=”(l=Tampa)” attrs=”ALL”
[27/Mar/2015:17:40:40 +0000] SEARCH RES conn=2059 op=1 msgID=2 result=0 nentries=69 etime=2
A benefit to this approach is that entitlements can be maintained in the directory server, itself. This reduces the complexity of the application, but requires that you configure appropriate access controls for each user. This can easily be performed at the group level, however, and even dynamically configured based on user attributes. A drawback to this approach is that the Web Application is acting as if they are the user – which they are not. The Browser is essentially the user and the Browser is not connecting directly to the directory server. So while the log files may reflect the user, they are somewhat misleading as the connection will always be from the Web Application. The other problem with this approach is the user’s credentials must be cached within the Web Application in order to perform subsequent operations against the directory server. One could argue that you could simply keep the connection between the Web Application and the directory server open, and that is certainly an option, but you would need to keep it open for the user’s entire session to prevent them from having to re-authenticate. This could lead to performance problems if you have extended session durations, a large number of administrative users, or a number of concurrent sessions by each administrative user.
Proxy Control – The Hybrid Approach
There are both benefits and drawbacks to each of the previously mentioned approaches, but I would like to offer up an alternative proxy-based approach that is essentially a hybrid between the two. RFC 4370 defines a proxied authorization control (2.16.840.1.113730.3.4.18) that allows a client (i.e. the Web Application) to request the directory server (i.e. OpenDJ) to perform an operation not based on the access control granted to the client, but based on another identity (i.e. the person logging in to the Web Application).
The proxied authorization control requires a client to bind to the directory server as themselves, but it allows them to impersonate another entry for a specific operation. This control can be used in situations where the application is trusted, but they need to perform operations on behalf of different users. The fact that the client is binding to the directory server eliminates the need to cache the user’s credentials (or re-authenticate for each operation). The fact that access is being determined based on that of the impersonated user means that you can centralize entitlements in the directory server and grant access based on security groups. This is essentially the best of both worlds and keeps a smile on the face of your security officer (as if that were possible).
So how do you configure proxy authorization? I am glad you asked.
Configuring Proxied Access
Before configuring proxied access, let’s return to the example of performing a search based on Application Access. The following is an example of a command line search that can be used to retrieve information from an OpenDJ server. The search operation uses the bindDN and password of the WebApp1 service account.
./ldapsearch -h localhost -D “uid=WebApp1,ou=AppAccounts,dc=example,dc=com ” -w password -b “ou=People,dc=example,dc=com” “l=Tampa”
The response to this search would include all entries that matched the filter (l=Tampa) beneath the container (ou=People). My directory server has been configured with 69 entries that match this search and as such, the OpenDJ access log would contain the following entries:
[27/Mar/2015:16:37:40 +0000] SEARCH REQ conn=2053 op=1 msgID=2 base=”ou=People,dc=example,dc=com” scope=wholeSubtree filter=”(l=Tampa)” attrs=”ALL”
[27/Mar/2015:16:37:40 +0000] SEARCH RES conn=2053 op=1 msgID=2 result=0 nentries=69 etime=2
As previously mentioned, these are the results you would expect to see if the search was performed as the WebApp1 user. So how can you perform a search impersonating another user? The answer lies in the parameters used in the search operation. The LDAP API supports a proxied search, you just need to determine how to access this functionality in your own LDAP client.
Note: I am using ldapsearch as the LDAP client for demonstration purposes. This is a command line tool that is included with the OpenDJ distribution. If you are developing a web application to act as the LDAP client, then you would need to determine how to access this functionality within your own development framework.
The OpenDJ search command includes a parameter that allows you to use the proxy authorization control. Type ./ldapsearch –help to see the options for the ldapsearch command and look for the -Y or –proxyAs parameter as follows.
Now perform the search again, but this time include the proxy control (without making any changes to the OpenDJ server). You will be binding as the WebApp1 account, but using the -Y option to instruct OpenDJ to evaluate ACIs based on the following user: uid=bnelson,ou=People,dc=example,dc=com.
./ldapsearch -h localhost -D “uid=WebApp1,ou=AppAccounts,dc=example,dc=com” -w password –Y “uid=bnelson,ou=People,dc=example,dc=com” -b “ou=People,dc=example,dc=com” “l=Tampa”
You should see the following response:
SEARCH operation failed
Result Code: 123 (Authorization Denied)
Additional Information: You do not have sufficient privileges to use the proxied authorization control The request control with Object Identifier (OID) “2.16.840.1.113730.3.4.18” cannot be used due to insufficient access rights
The corresponding entries in OpenDJ’s access log would be as follows:
[27/Mar/2015:10:47:18 +0000] SEARCH REQ conn=787094 op=1 msgID=2 base=”ou=People,dc=example,dc=com” scope=wholeSubtree filter=”(l=Tampa)” attrs=”ALL”
[27/Mar/2015:10:47:18 +0000] SEARCH RES conn=787094 op=1 msgID=2 result=123 message=”You do not have sufficient privileges to use the proxied authorization control You do not have sufficient privileges to use the proxied authorization control” nentries=0 etime=1
The key phrase in these messages is the following:
You do not have sufficient privileges to use the proxied authorization control
The key word in that phrase is “privileges” as highlighted above; the WebApp1 service account does not have the appropriate privileges to perform a proxied search and as such, the search operation is rejected. The first step in configuring proxied access control is to grant proxy privileges to the Application Account.
Step 1: Grant Proxy Privileges to the Application Account
The first step in allowing the WebApp1 service account to perform a proxied search is to give that account the proxied-auth privilege. You can use the ldapmodify utility to perform this action as follows:
./ldapmodify -D “cn=Directory Manager” -w password
dn: uid=WebApp1,ou=AppAccounts,dc=example,dc=com
changetype: modify
add: ds-privilege-name
ds-privilege-name: proxied-auth
Processing MODIFY request for uid=WebApp1,ou=AppAccounts,dc=example,dc=com
MODIFY operation successful for DN uid=WebApp1,ou=AppAccounts,dc=example,dc=com
Now repeat the proxied search operation.
./ldapsearch -h localhost -D “uid=WebApp1,ou=AppAccounts,dc=example,dc=com” -w password –Y “uid=bnelson,ou=People,dc=example,dc=com” -b “ou=People,dc=example,dc=com” “l=Tampa”
Once again your search will fail, but this time it is for a different reason.
SEARCH operation failed
Result Code: 12 (Unavailable Critical Extension)
Additional Information: The request control with Object Identifier (OID) “2.16.840.1.113730.3.4.18” cannot be used due to insufficient access rights
The corresponding entries in OpenDJ’s access log would be as follows:
[27/Mar/2015:11:39:17 +0000] SEARCH REQ conn=770 op=1 msgID=2 base=” ou=People,dc=example,dc=com ” scope=wholeSubtree filter=”(l=Tampa)” attrs=”ALL”
[27/Mar/2015:11:39:17 +0000] SEARCH RES conn=770 op=1 msgID=2 result=12 message=”” nentries=0 authzDN=”uid=bnelson,ou=People,dc=example,dc=com” etime=3
As discussed in OpenDJ Access Control Explained, authorization to perform certain actions may consist of a combination of privileges and ACIs. You have granted the proxied-auth privilege to the WebApp1 service account, but it still needs an ACI to allow it to perform proxy-based operations. For the purposes of this demonstration, we will use the following ACI to grant this permission.
(targetattr=”*”) (version 3.0; acl “Allow Proxy Authorization to Web App 1 Service Account”; allow (proxy) userdn=”ldap:///uid=WebApp1,ou=AppAccounts,dc=example,dc=com”;)
This ACI will be placed at the root suffix for ease of use, but you should consider limiting the scope of the ACI by placing it at the appropriate branch in your directory tree (and limiting the targetattr values).
Step 2: Create a (Proxy) ACI for the Application Account
Once again, you can use the ldapmodify utility to update OpenDJ with this new ACI.
./ldapmodify -D “cn=Directory Manager” -w password
dn: dc=example,dc=com
changetype: modify
add: aci
aci: (targetattr=”*”) (version 3.0; acl “Allow Proxy Authorization to Web App 1 Service Account”; allow (proxy) userdn=”ldap:///uid=WebApp1,ou=AppAccounts,dc=example,dc=com”;)
Processing MODIFY request for dc=example,dc=com
MODIFY operation successful for DN dc=example,dc=com
Now repeat the proxied search a final time.
./ldapsearch -h localhost -D “uid=WebApp1,ou=AppAccounts,dc=example,dc=com” -w password –Y “uid=bnelson,ou=People,dc=example,dc=com” -b “ou=People,dc=example,dc=com” “l=Tampa”
This time you should see the results of the search performed correctly. But how do you know that this was a proxied search and not simply one performed by the WebApp1 as before? The clue is once again in the OpenDJ access log file. Looking in this file, you will see the following entries:
[27/Mar/2015:11:40:23 +0000] SEARCH REQ conn=797 op=1 msgID=2 base=”ou=People,dc=example,dc=com” scope=wholeSubtree filter=”(l=Tampa)” attrs=”ALL”
[27/Mar/2015:11:40:23 +0000] SEARCH RES conn=797 op=1 msgID=2 result=12 message=”” nentries=69 authzDN=”uid=bnelson,ou=people,dc=example,dc=com” etime=1
The authzDN value contains the DN of the entry used for authorization purposes. This is a clear indicator that access control was based on the uid=bnelson entry and not uid=WebApp1.
Still not convinced? You can verify this by removing the rights for the uid=bnelson entry and running your search again. Add the following ACI to the top of your tree.
(targetattr=”*”)(version 3.0;acl ” Deny Access to BNELSON”; deny (all)(userdn = “ldap:///uid=bnelson,out=people,dc=example,dc=com”);)
Now run the search again. This time, you will not see any errors, but you will also not see any entries returned. While you are binding as the WebApp1 service account, for all intents and purposes, you are impersonating the uid=bnelson user when determining access rights.
Summary of Steps
The following steps should be performed when configuring OpenDJ for proxied access control.
Create the Application Account in OpenDJ (i.e. WebApp1)
- Create the Application Account in OpenDJ (i.e. WebApp1)
- Add the proxy-auth privilege to the Application Account
- Create an ACI allowing the Application Account to perform proxy operations
- Create a User Account in OpenDJ (i.e. bnelson)
- Configure ACIs for User Account as appropriate
- Test the configuration by performing a command line search using the proxied access control parameter.
OpenDJ Access Control Explained
An OpenDJ implementation will contain certain data that you would like to explicitly grant or deny access to. Personally identifiable information (PII) such as a user’s home telephone number, their address, birth date, or simply their email address might be required by certain team members or applications, but it might be a good idea to keep this type of information private from others. On the other hand, you may want their office phone number published for everyone within the company to see but limit access to this data outside of the company.
Controlling users’ access to different types of information forms the basis of access control in OpenDJ and consists of the following two stages:
- Authentication (AuthN) – the process of positively identifying a subject
- Authorization (AuthZ) – the process of determining the rights that a subject has on a particular resource
Before you are allowed to perform any action within OpenDJ, it must first know who you are. Once your identity has been established, OpenDJ can then ascertain the rights you have to perform actions either on the data contained in its database(s) or within the OpenDJ process, itself.
Access Control = Authentication + Authorization
Note: Access control is not defined in any of the LDAP RFCs so the manner in which directory servers implement access control varies from vendor to vendor. Many directory services (including OpenDJ) follow the LDAP v3 syntax introduced by Netscape.
Access control is implemented with an operational attribute called aci (which stands for access control instruction). Access control instructions can be configured globally (the entire OpenDJ instance) or added to specific directory entries.
1. Global ACIs:
Global ACIs are not associated with directory entries and therefore are not available when searching against a typical OpenDJ suffix (such as dc=example,dc=com). Instead, Global ACIs are considered configuration objects and may be found in the configuration suffix (cn=config). You can find the currently configured Global ACIs by opening the config.ldif file and locating the entry for the “Access Control Handler”. Or, you can search for “cn=Access Control Handler” in the configuration suffix (cn=config) as follows:
./ldapsearch –h hostname –p portnumber –D “cn=directory manager” –w “password” -b “cn=config” -s sub “cn=Access Control Handler” ds-cfg-global-aci
This returns the following results on a freshly installed (unchanged) OpenDJ server.
dn: cn=Access Control Handler,cn=config
ds-cfg-global-aci: (extop=”1.3.6.1.4.1.26027.1.6.1 || 1.3.6.1.4.1.26027.1.6.3 || 1.3.6.1.4.1.4203.1.11.1 || 1.3.6.1.4.1.1466.20037 || 1.3.6.1.4.1.4203.1.11.3″) (version 3.0; acl “Anonymous extended operation access”; allow(read) userdn=”ldap:///anyone”;)
ds-cfg-global-aci: (target=”ldap:///”)(targetscope=”base”)(targetattr=”objectClass||namingContexts||supportedAuthPasswordSchemes||supportedControl||supportedExtension||supportedFeatures||supportedLDAPVersion||supportedSASLMechanisms||supportedTLSCiphers||supportedTLSProtocols||vendorName||vendorVersion”)(version 3.0; acl “User-Visible Root DSE Operational Attributes”; allow (read,search,compare) userdn=”ldap:///anyone”;)
ds-cfg-global-aci: target=”ldap:///cn=schema”)(targetattr=”attributeTypes||objectClasses”)(version 3.0;acl “Modify schema”; allow (write)(userdn = “ldap:///uid=openam,ou=Service Accounts,dc=example,dc=com”);)
ds-cfg-global-aci: target=”ldap:///cn=schema”)(targetscope=”base”)(targetattr=” objectClass||attributeTypes||dITContentRules||dITStructureRules||ldapSyntaxes||matchingRules||matchingRuleUse||nameForms||objectClasses”)(version 3.0; acl “User-Visible Schema Operational Attributes”; allow (read,search,compare) userdn=”ldap:///anyone”;)
ds-cfg-global-aci: (target=”ldap:///dc=replicationchanges”)(targetattr=”*”)(version 3.0; acl “Replication backend access”; deny (all) userdn=”ldap:///anyone”;)
ds-cfg-global-aci: (targetattr!=”userPassword||authPassword||changes||changeNumber||changeType||changeTime||targetDN||newRDN||newSuperior||deleteOldRDN”)(version 3.0; acl “Anonymous read access”; allow (read,search,compare) userdn=”ldap:///anyone”;)
ds-cfg-global-aci: (targetattr=”audio||authPassword||description||displayName||givenName||homePhone||homePostalAddress||initials||jpegPhoto||labeledURI||mobile||pager||postalAddress||postalCode||preferredLanguage||telephoneNumber||userPassword”)(version 3.0; acl “Self entry modification”; allow (write) userdn=”ldap:///self”;)
ds-cfg-global-aci: (targetattr=”createTimestamp||creatorsName||modifiersName||modifyTimestamp||entryDN||entryUUID||subschemaSubentry||etag||governingStructureRule||structuralObjectClass||hasSubordinates||numSubordinates”)(version 3.0; acl “User-Visible Operational Attributes”; allow (read,search,compare) userdn=”ldap:///anyone”;)
ds-cfg-global-aci: (targetattr=”userPassword||authPassword”)(version 3.0; acl “Self entry read”; allow (read,search,compare) userdn=”ldap:///self”;)
ds-cfg-global-aci: (targetcontrol=”1.3.6.1.1.12 || 1.3.6.1.1.13.1 || 1.3.6.1.1.13.2 || 1.2.840.113556.1.4.319 || 1.2.826.0.1.3344810.2.3 || 2.16.840.1.113730.3.4.18 || 2.16.840.1.113730.3.4.9 || 1.2.840.113556.1.4.473 || 1.3.6.1.4.1.42.2.27.9.5.9″) (version 3.0; acl “Authenticated users control access”; allow(read) userdn=”ldap:///all”;)
ds-cfg-global-aci: (targetcontrol=”2.16.840.1.113730.3.4.2 || 2.16.840.1.113730.3.4.17 || 2.16.840.1.113730.3.4.19 || 1.3.6.1.4.1.4203.1.10.2 || 1.3.6.1.4.1.42.2.27.8.5.1 || 2.16.840.1.113730.3.4.16 || 1.2.840.113556.1.4.1413″) (version 3.0; acl “Anonymous control access”; allow(read) userdn=”ldap:///anyone”;)
2. Entry-Based ACIs:
Access control instructions may also be applied to any entry in the directory server. This allows fine grained access control to be applied anywhere in the directory information tree and therefore affects the scope of the ACI.
Note: Placement has a direct effect on the entry where the ACI is applied as well as any children of that entry.
You can obtain a list of all ACIs configured in your server (sans the Global ACIs) by performing the following search:
./ldapsearch –h hostname –p portnumber –D “cn=directory manager” –w “password” –b “dc=example,dc=com” –s sub aci=* aci
By default, there are no ACIs configured at the entry level. The following is an example of ACIs that might be returned if you did have ACIs configured, however.
dn: dc=example,dc=com
aci: (targetattr=”*”)(version 3.0;acl “Allow entry search”; allow (search,read)(userdn = “ldap:///uid=openam,ou=Service Accounts,dc=example,dc=com”);)
aci: (targetattr=”*”)(version 3.0;acl “Modify config entry”; allow (write)(userdn = “ldap:///uid=openam,ou=Service Accounts,dc=example,dc=com”);)
aci: (targetcontrol=”2.16.840.1.113730.3.4.3″)(version 3.0;acl “Allow persistent search”; allow (search, read)(userdn = “ldap:///uid=openam,ou=Service Accounts,dc=example,dc=com”);)
aci: (version 3.0;acl “Add config entry”; allow (add)(userdn = “ldap:///uid=openam,ou=Service Accounts,dc=example,dc=com”);)
aci: (version 3.0;acl “Delete config entry”; allow (delete)(userdn = “ldap:///uid=openam,ou=Service Accounts,dc=example,dc=com”); )
dn: ou=Applications,dc=example,dc=com
aci: (target =”ldap:///ou=Applications,dc=example,dc=com”)(targetattr=”*”)(version 3.0;acl “Allow Application Config Access to Web UI Admin”; allow (all)(userdn = “ldap:///uid=webui,ou=Applications,dc=example,dc=com”); )
ACI Syntax:
The syntax for access control instructions is not specific to OpenDJ, in fact, for the most part, it shares the same syntax with the Oracle Directory Server Enterprise Edition (“ODSEE”). This is mainly due the common lineage with Sun Microsystems, but other directory servers do not use the same syntax and this makes migration more difficult (even the schema in both servers contains an attribute called aci). If you export OpenDJ directory entries to LDIF and attempt to import them into another vendor’s server, the aci statements would either be ignored, or worse, might have unpredictable results, altogether.
The following syntax is used by the OpenDJ server.
Access control instructions require three inputs: target, permission, and subject. The target specifies the entries to which the aci applies. The subject applies to the client that is performing the operation and the permissions specify what the subject is allowed to do. You can create some very powerful access control based on these three inputs.
The syntax also includes the version of the aci syntax, version 3.0. This is the aci syntax version, not the LDAP version. Finally, the syntax allows you to enter a human readable name. This allows you to easily search for and identify access control statements in the directory server.
Note: Refer to the OpenDJ Administration Guide for a more detailed description of the aci A components.
The following is an example of an ACI that permits a user to write to their own password and mobile phone attributes.
You cannot read the ACI from left to right, or even right to left, you simply have to dive right in and look for the information required to understand the intent of the ACI. If you have been working with ACIs for some time, you probably already have your own process, but I read/interpret the preceding ACI as follows:
This ACI “allows” a user to “write” to their own (“ldap:///self”) userPassword and mobile attributes “(targetattr=”userPassword||mobile”)”
If you place this ACI on a particular user’s object (i.e. uid=bnelson, ou=people,dc=example,dc=com), then this ACI would only apply to this object. If you place this ACI on a container of multiple user objects (i.e. ou=people,dc=example,dc=com), then this ACI would apply to all user objects included in this container.
Access Control Processing:
Access control instructions provide fine-grained control over what a given user or group member is authorized to do within the directory server.
When a directory-enabled client tries to perform an operation on any entry in the server, an access control list (ACL) is created for that particular entry. The ACL for any given entry consists of the entry being accessed as well as any parent entries all the way up to the root entry.
The ACL is essentially the summation of all acis defined for the target(s) being accessed plus the acis for all parent entries all the way to the top of the tree. Included in this list are any Global ACIs that may have been configured in the cn=config as well. While not entirely mathematically accurate, the following formula provides an insight into how the ACL is generated.
Using the previous formula, the access control lists for each entry in the directory information tree would be as follows:
Once the ACL is created, the list is then processed to determine if the client is allowed to perform the operation or not. ACLs are processed as follows:
- If there exists at least one explicit DENY rule that prevents a user from performing the requested action (i.e. deny(write)), then the user is denied.
- If there exists at least one explicit ALLOW rule that allows a user to perform the requested action (i.e. allow(write)), then the user is allowed (as long as there are no other DENY rules preventing this).
- If there are neither DENY nor ALLOW rules defined for the requested action, then the user is denied. This is referred to as the implicit deny.
Something to Think About…
 |
Thought 1: If in the absence of any access control instructions, the default is to deny access, then what is the purpose of access control instructions you might ask? ACIs with ALLOW rules are used to grant a user permission to perform some action. Without ALLOW ACIs, all actions are denied (due to the implicit deny rule).
Thought 2: If the default is to implicitly deny a user, then what is the purpose of DENY rules? DENY rules are used to revoke a previously granted permission. For instance, suppose that you create an ALLOW rule for the Help Desk Admin group to access a user’s PII data in order to help determine the user’s identity for a password reset. But you have a recently hired Help Desk Admin that has not completed the required sensitivity training. You may elect to keep him in the Help Desk Admin group for other reasons, but revoke his ability to read users’ PII data until his training has been completed.Note: You should use DENY rules sparingly. If you are creating too many DENY rules you should question how you have created your ALLOW rules. Thought 3: If the absence of access control instructions means that everyone is denied, then how can we manage OpenDJ in the event that conflicting ACIs are introduced? Or worse, ACIs are dropped altogether? That is where the OpenDJ Super User and OpenDJ privileges come in. |
OpenDJ’s Super User:
The RootDN user (“cn=Directory Manager” by default) is a special administrative user that can pretty much perform any action in OpenDJ. This user account is permitted full access to directory server data and can perform almost any action in the directory service, itself. Essentially, this account is similar to the root or Administrator accounts on UNIX and Windows systems, respectively.
If you look in the directory server you will find that there are no access control instruction granting the RootDN this unrestricted access; but there are however privileges that do so.
Privileges:
While access control instructions restrict access to directory data through LDAP operations, privileges define administrative tasks that may be performed by users within OpenDJ. Assignment of privileges to users (either directly or through groups) effectively allows those users the ability to perform the administrative tasks defined by those privileges.
The following table provides a list of common privileges and their relationship to the RootDN user.
The RootDN user is assigned these privileges by default and similar to Global ACIs, these privileges are defined and maintained in the OpenDJ configuration object. The following is the default list of privileges associated with Root DN users (of which the Directory Manager account is a member).
dn: cn=Root DNs,cn=config
objectClass: ds-cfg-root-dn
objectClass: top
ds-cfg-default-root-privilege-name: bypass-lockdown
ds-cfg-default-root-privilege-name: bypass-acl
ds-cfg-default-root-privilege-name: modify-acl
ds-cfg-default-root-privilege-name: config-read
ds-cfg-default-root-privilege-name: config-write
ds-cfg-default-root-privilege-name: ldif-import
ds-cfg-default-root-privilege-name: ldif-export
ds-cfg-default-root-privilege-name: backend-backup
ds-cfg-default-root-privilege-name: backend-restore
ds-cfg-default-root-privilege-name: server-lockdown
ds-cfg-default-root-privilege-name: server-shutdown
ds-cfg-default-root-privilege-name: server-restart
ds-cfg-default-root-privilege-name: disconnect-client
ds-cfg-default-root-privilege-name: cancel-request
ds-cfg-default-root-privilege-name: password-reset
ds-cfg-default-root-privilege-name: update-schema
ds-cfg-default-root-privilege-name: privilege-change
ds-cfg-default-root-privilege-name: unindexed-search
ds-cfg-default-root-privilege-name: subentry-write
cn: Root DNs
This list can retrieved using the OpenDJ dsconfig command:
./dsconfig –h localhost –p 4444 –D “cn=directory manager” –w password get-root-dn-prop
with the ldapsearch command:
./ldapsearch –h hostname –p portnumber –D “cn=directory manager” –w “password” -b “cn=config” -s sub “cn=Root DNs” ds-cfg-default-root-privilege-name
or simply by opening the config.ldif file and locating the entry for the “cn=Root DNs” entry.
Most operations involving sensitive or administrative data require that a user has both the appropriate privilege(s) as well as certain access control instructions. This allows you to configure authorization at a fine grained level – such as managing access control or resetting passwords.
Privileges are assigned to users and apply globally to the directory service. Any user can be granted or denied any privilege and by default only the RootDN users are assigned a default set of privileges.
Note: Consider creating different types of administrative groups in OpenDJ and assign the privileges and ACIs to those groups to define what a group member is allowed to do. Adding users to that group then automatically grants those users the rights defined in the group and conversely, removing them from the group drops those privileges (unless they are granted through another group).
Effective Rights:
Once you set up a number of ACIs, you may find it difficult to understand how the resulting access control list is processed and ultimately the rights that a particular user may have. Fortunately OpenDJ provides a method of evaluating the effective rights that a subject has on a given target.
You can use the ldapsearch command to determine the effective rights that a user has on one or more attributes on one or more entries.
$ ldapsearch –h localhost –p 1389 -D “cn=Directory Manager” -w password
-g “dn:uid=helpdeskadmin,ou=administrators, dc=example,dc=com” -b “uid=scarter,ou=people, dc=example,dc=com” -s base ‘(objectclass=*)’ ‘*’ aclrights
The preceding search is being performed by the Root DN user (“cn=Directory Manager”). It is passing the –g option requesting the get effective rights control (to which the Directory Manager has the appropriate access configured). The command wants to determine what rights the Help Desk Administrator (uid=helpdeskadmin,…) has on Sam Carter’s entry (uid=scarter,…). The scope of the search has been limited only to Sam Carter’s entry using the base parameter. Finally, the search operation is returning not only the attributes, but the effective rights (aclrights) as well.
Possible results from a search operation such as this are as follows:
dn: uid=scarter,ou=People,dc=example,dc=com
objectClass: person
objectClass: top
uid: scarter
userPassword: {SSHA}iMgzz9mFA6qYtkhS0Z7bhQRnv2Ic8efqpctKDQ==
givenName: Sam
cn: Sam Carter
sn: Carter
mail: sam.carter@example.com
aclRights;attributeLevel;objectclass: search:1,read:1,compare:1,write:0,selfwrit
e_add:0,selfwrite_delete:0,proxy:0
aclRights;attributeLevel;uid: search:1,read:1,compare:1,write:0,selfwrite_add:0,
selfwrite_delete:0,proxy:0
aclRights;attributeLevel;userpassword: search:0,read:0,compare:0,write:1,selfwri
te_add:0,selfwrite_delete:0,proxy:0
aclRights;attributeLevel;givenname: search:1,read:1,compare:1,write:0,selfwrite_
add:0,selfwrite_delete:0,proxy:0
aclRights;attributeLevel;cn: search:1,read:1,compare:1,write:0,selfwrite_add:0,s
elfwrite_delete:0,proxy:0
aclRights;attributeLevel;sn: search:1,read:1,compare:1,write:0,selfwrite_add:0,s
elfwrite_delete:0,proxy:0
aclRights;attributeLevel;mail: search:1,read:1,compare:1,write:0,selfwrite_add:0
,selfwrite_delete:0,proxy:0
aclRights;entryLevel: add:0,delete:0,read:1,write:0,proxy:0
The search results contain not only the attributes/attribute values associated with Sam Carter’s object, but the effective rights that the Help Desk Admins have on those attributes. For instance,
aclRights;attributeLevel;givenname: search:1,read:1,compare:1,write:0,selfwrite_
add:0,selfwrite_delete:0,proxy:0
The aclRights;attributeLevel;givenname notation indicate that this line includes the effective rights for the givenname attribute. Individual permissions are listed that demonstrate the rights that the Help Desk Administrator has on this attribute for Sam Carter’s entry (1 = allowed and 0 = denied).
Recommendations:
An OpenDJ installation includes a set of default (Global) access control instructions which by some standards may be considered insecure. For instance, there are five ACIs that allow an anonymous user the ability to read certain controls, extended operations, operational attributes, schema attributes, and user attributes. The basic premise behind this is that ForgeRock wanted to provide an easy out-of-the-box evaluation of the product while at the same time providing a path forward for securing the product. It is intended that OpenDJ should be hardened in order to meet a company’s security policies and in fact, one task that is typically performed before placing OpenDJ in production is to limit anonymous access. There are two ways you can perform this:
- Enable the reject-unauthenticated-request property using the dsconfig command.
- Update the Global ACIs
Mark Craig provides a nice blog posting on how to turn off anonymous access using the dsconfig command. You can find that blog here. The other option is to simply change the reference in the Global ACIs from ldap:///anyone to ldap:///all. This prevents anonymous users from gaining access to this information.
Note: Use of ldap:///anyone in an ACI includes both authenticated and anonymous users – essentially, anyone. Changing this to ldap:///all restricts the subject to all authenticated users.
The following comments from Ludo Poitou (ForgeRock’s OpenDJ Product Manager) should be considered before simply removing anonymous access.
You don’t want to remove the ACI rules for Anonymous access, you want to change it from granting access to anyone (ldap:///anyone) to granting access to all authenticated users (ldap:///all).
This said, there are some differences between fully rejecting unauthenticated requests and using ACI to control access. The former will block all access including the attempts to discover the server’s capabilities by reading the RootDSE. The later allows you to control which parts can be accessed anonymously, and which shouldn’t.
There’s been a lot of fuss around allowing anonymous access to a directory service. Some people are saying that features and naming context discovery is a threat to security, allowing malicious users to understand what the server contains and what security mechanisms are available and therefore not available. At the same time, it is important for generic purpose applications to understand how they can or must use the directory service before they actually authenticate to it.
Fortunately, OpenDJ has mechanisms that allow administrators to configure the directory services according to their security constraints, using either a simple flag to reject all unauthenticated requests, or by using ACIs.
A few other things to consider when configuring access control in OpenDJ include the following:
- Limit the number of Root DN user accounts
You should have one Root DN account and it should not be shared with multiple administrators. Doing so makes it nearly impossible to determine the identity of the person who performed a configuration change or operation in OpenDJ. Instead, make the password complex and store it in a password vault.
- Create a delegated administration environment
Now that you have limited the number of Root DN accounts, you need to create groups to allow users administrative rights in OpenDJ. Users would then log in as themselves and perform operations against the directory server using their own account. The tasks associated with this are as follows:
- Create administrative groups
- Use the ds-privilege-name attribute to assign privilege(s) to the group entry
- Create ACIs based on the group name, not an individual user. For instance, “groupdn:///cn=Help Desk Admin, ou=Administrators,dc=example,dc=com”
- Add users to the groups
- Associate privileges and ACIs to users for fine grained access control
Now that you have create administrative groups, you are ultimately going to need to provide certain users with more rights than others. You can create additional administrative groups, but what if you only need one user to have these rights. Creating a group of one may or may not be advisable and may actually lead to group explosion (where you end up with more groups than you actually have users). Instead, consider associating privileges to a particular user and then create ACIs based on that user.
OpenDJ Attribute Uniqueness (and the Effects on OpenAM)
In real life we tend to value those traits that make us unique from others; but in an identity management deployment uniqueness is essential to the authentication process and should not be taken for granted.
Case in point, attributes in OpenDJ may share values that you may or may not want (or need) to be unique. For instance the following two (different) entries are both configured with the same value for the email address:
dn: uid=bnelson,ou=people,dc=example,dc=com uid: bnelson mail: bill.nelson@identityfusion.com [LDIF Stuff Snipped]
dn: uid=scarter,ou=people,dc=example,dc=com uid: scarter mail: bill.nelson@identityfusion.com [LDIF Stuff Snipped]
In some cases this may be fine, but in others this may not be the desired effect as you may need to enforce uniqueness for attributes such as uid, guid, email address, or simply credit cards. To ensure that attribute values are unique across directory server entries you need to configure attribute uniqueness.
UID Uniqueness Plug-In
OpenDJ has an existing plug-in that can be used to configure unique values for the uid attribute, but this plug-in is disabled by default. You can find this entry in OpenDJ’s main configuration file (config.ldif) or by searching the cn=config tree in OpenDJ (assuming you have the correct permissions to do so).
dn: cn=UID Unique Attribute,cn=Plugins,cn=config objectClass: ds-cfg-unique-attribute-plugin objectClass: ds-cfg-plugin objectClass: top ds-cfg-enabled: false ds-cfg-java-class: org.opends.server.plugins.UniqueAttributePlugin ds-cfg-plugin-type: preOperationAdd ds-cfg-plugin-type: preOperationModify ds-cfg-plugin-type: preOperationModifyDN ds-cfg-plugin-type: postOperationAdd ds-cfg-plugin-type: postOperationModify ds-cfg-plugin-type: postOperationModifyDN ds-cfg-plugin-type: postSynchronizationAdd ds-cfg-plugin-type: postSynchronizationModify ds-cfg-plugin-type: postSynchronizationModifyDN ds-cfg-invoke-for-internal-operations: true ds-cfg-type: uid cn: UID Unique Attribute
Leaving this plug-in disabled can cause problems with OpenAM, however, if OpenAM has been configured to authenticate using the uid attribute (and you ‘accidentally’ create entries with the same uid value). In such cases you will see an authentication error during the login process as OpenAM cannot determine which account you are trying to use for authentication.
Configuring Uniqueness
To fix this problem in OpenAM, you can use the OpenDJ dsconfig command to enable the UID Unique Attribute plug-in as follows:
./dsconfig set-plugin-prop --hostname localhost --port 4444 \ --bindDN "cn=Directory Manager" --bindPassword password \ --plugin-name "UID Unique Attribute" \ --set base-dn:ou=people,dc=example,dc=com --set enabled:true \ --trustAll --no-prompt
This will prevent entries from being added to OpenDJ where the value of any existing uids conflicts with the incoming entry’s uid. This will address the situation where you are using the uid attribute for authentication in OpenAM, but what if you want to use a different attribute (such as mail) to authenticate? In such cases, you need to create your own uniqueness plug-in as follows:
./dsconfig create-plugin --hostname localhost --port 4444 \ --bindDN "cn=Directory Manager" --bindPassword password \ --plugin-name "Unique Email Address Plugin" \ --type unique-attribute --set type:mail --set enabled:true \ --set base-dn:ou=people,dc=example,dc=com --trustAll \ --no-prompt
In both cases the base-dn parameter defines the scope where the the uniqueness applies. This is useful in multitenant environments where you may want to define uniqueness within a particular subtree but not necessarily across the entire server.
Prerequisites
The uniqueness plug-in requires that you have an existing equality index configured for the attribute where you would like to enforce uniqueness. The index is necessary so that OpenDJ can search for other entries (within the scope of the base-dn) where the attribute may already have a particular value set.
The following dscconfig command can be used to create an equality index for the mail attribute:
./dsconfig create-local-db-index --hostname localhost --port 4444 \ --bindDN "cn=Directory Manager" --bindPassword password --backend-name userRoot \ --index-name mail --set index-type:equality --trustAll --no-prompt
Summary
OpenAM’s default settings (Data Store, LDAP authentication module, etc) uses the uid attribute to authenticate and uniquely identify a user. OpenDJ typically uses uid as the unique naming attribute in a user’s distinguished name. When combined together, it is almost assumed that you will be using the uid attribute in this manner, but that is not always the case. You can easily run into issues when you start coloring outside of the lines and begin using other attributes (i.e. mail) for this purpose. Armed with the information contained in this post, however, you should easily be able to configure OpenDJ to enforce uniqueness for any attribute.
Understanding OpenAM and OpenDJ Account Lockout Behaviors
The OpenAM Authentication Service can be configured to lock a user’s account after a defined number of log in attempts has failed. Account Lockout is disabled by default, but when configured properly, this feature can be useful in fending off brute force attacks against OpenAM login screens.
If your OpenAM environment includes an LDAP server (such as OpenDJ) as an authentication database, then you have options on how (and where) you can configure Account Lockout settings. This can be performed in either OpenAM (as mentioned above) or in the LDAP server, itself. But the behavior is different based on where this is configured. There are benefits and drawbacks towards configuring Account Lockout in either product and knowing the difference is essential.
Note: Configuring Account Lockout simultaneously in both products can lead to confusing results and should be avoided unless you have a firm understanding of how each product works. See the scenario at the end of this article for a deeper dive on Account Lockout from an attribute perspective.
The OpenAM Approach
You can configure Account Lockout in OpenAM either globally or for a particular realm. To access the Account Lockout settings for the global configuration,
- Log in to OpenAM Console
- Navigate to: Configuration > Authentication > Core
- Scroll down to Account Lockout section
To access Account Lockout settings for a particular realm,
- Log in to OpenAM Console
- Navigate to: Access Control > realm > Authentication > All Core Settings
- Scroll down to Account Lockout section
In either location you will see various parameters for controlling Account Lockout as follows:
Configuring Account Lockout in OpenAM
Account Lockout is disabled by default; you need to select the “Login Failure Lockout Mode” checkbox to enable this feature. Once it is enabled, you configure the number of attempts before an account is locked and even if a warning message is displayed to the user before their account is locked. You can configure how long the account is locked and even the duration between successive lockouts (which can increase if you set the duration multiplier). You can configure the attributes to use to store the account lockout information in addition to the default attributes configured in the Data Store.
Enabling Account Lockout affects the following Data Store attributes: inetUserStatus and sunAMAuthInvalidAttemptsData. By default, the value of the inetUserStatus attribute is either Active or Inactive, but this can be configured to use another attribute and another attribute value. This can be configured in the User Configuration section of the Data Store configuration as follows:
Data Store Account Lockout Attributes
These attributes are updated in the Data Store configuration for the realm. A benefit of implementing Account Lockout in OpenAM is that you can use any LDAPv3 directory, Active Directory, or even a relational database – but you do need to have a Data Store configured to provide OpenAM with somewhere to write these values. An additional benefit is that OpenAM is already configured with error messages that can be easily displayed when a user’s account is about to be locked or has become locked. Configuring Account Lockout within OpenAM, however, may not provide the level of granularity that you might need and as such, you may need to configure it in the authentication database (such as OpenDJ).
The OpenDJ Approach
OpenDJ can be configured to lock accounts as well. This is defined in a password policy and can be configured globally (the entire OpenDJ instance) or it may be applied to a subentry (a group of users or a specific user). Similar to OpenAM, a user’s account can be locked after a number of invalid authentication attempts have been made. And similar to OpenAM, you have several additional settings that can be configured to control the lockout period, whether warnings should be sent, and even who to notify when the account has been locked.
But while configuring Account Lockout in OpenAM may recognize invalid password attempts in your SSO environment, configuring it in OpenDJ will recognize invalid attempts for any application that is using OpenDJ as an authentication database. This is more of a centralized approach and can recognize attacks from several vectors.
Configuring Account Lockout in OpenDJ affects the following OpenDJ attributes: pwdFailureTime (a multivalued attribute consisting of the timestamp of each invalid password attempt) and pwdAccountLockedTime (a timestamp indicating when the account was locked).
Another benefit of implementing Account Lockout in OpenDJ is the ability to configure Account Lockout for different types of users. This is helpful when you want to have different password policies for users, administrators, or even service accounts. This is accomplished by assigning different password polices directly to those users or indirectly through groups or virtual attributes. A drawback to this approach, however, is that OpenAM doesn’t necessarily recognize the circumstances behind error messages returned from OpenDJ when a user is unable to log in. A scrambled password in OpenDJ, for instance, simply displays as an Authentication failed error message in the OpenAM login screen.
By default, all users in OpenDJ are automatically assigned a generic (rather lenient) password policy that is aptly named: Default Password Policy. The definition of this policy can be seen as follows:
dn: cn=Default Password Policy,cn=Password Policies,cn=config objectClass: ds-cfg-password-policy objectClass: top objectClass: ds-cfg-authentication-policy ds-cfg-skip-validation-for-administrators: false ds-cfg-force-change-on-add: false ds-cfg-state-update-failure-policy: reactive ds-cfg-password-history-count: 0 ds-cfg-password-history-duration: 0 seconds ds-cfg-allow-multiple-password-values: false ds-cfg-lockout-failure-expiration-interval: 0 seconds ds-cfg-lockout-failure-count: 0 ds-cfg-max-password-reset-age: 0 seconds ds-cfg-max-password-age: 0 seconds ds-cfg-idle-lockout-interval: 0 seconds ds-cfg-java-class: org.opends.server.core.PasswordPolicyFactory ds-cfg-lockout-duration: 0 seconds ds-cfg-grace-login-count: 0 ds-cfg-force-change-on-reset: false ds-cfg-default-password-storage-scheme: cn=Salted SHA-1,cn=Password Storage Schemes,cn=config ds-cfg-allow-user-password-changes: true ds-cfg-allow-pre-encoded-passwords: false ds-cfg-require-secure-password-changes: false cn: Default Password Policy ds-cfg-require-secure-authentication: false ds-cfg-expire-passwords-without-warning: false ds-cfg-password-change-requires-current-password: false ds-cfg-password-generator: cn=Random Password Generator,cn=Password Generators, cn=config ds-cfg-password-expiration-warning-interval: 5 days ds-cfg-allow-expired-password-changes: false ds-cfg-password-attribute: userPassword ds-cfg-min-password-age: 0 seconds
The value of the ds-cfg-lockout-failure-count attribute is 0; which means that user accounts are not locked by default – no matter how many incorrect attempts are made. This is one of the many security settings that you can configure in a password policy and while many of these mimic what is available in OpenAM, others go quite deeper.
You can use the OpenDJ dsconfig command to change the Default Password Policy as follows:
dsconfig set-password-policy-prop --policy-name "Default Password Policy" --set lockout-failure-count:3 --hostname localhost --port 4444 --trustAll --bindDN "cn=Directory Manager" --bindPassword ****** --no-prompt
Rather than modifying the Default Password Policy, a preferred method is to create a new password policy and apply your own specific settings to the new policy. This policy can then be applied to a specific set of users.
The syntax for using the OpenDJ dsconfig command to create a new password policy can be seen below.
dsconfig create-password-policy --set default-password-storage-scheme:"Salted SHA-1" --set password-attribute:userpassword --set lockout-failure-count:3 --type password-policy --policy-name "Example Corp User Password Policy" --hostname localhost --port 4444 --trustAll --bindDN cn="Directory Manager" --bindPassword ****** --no-prompt
Note: This example contains a minimum number of settings (default-password-storage-scheme, password-attribute, and lockout-failure-count). Consider adding additional settings to customize your password policy as desired.
You can now assign the password policy to an individual user by adding the following attribute as a subentry to the user’s object:
ds-pwp-password-policy-dn: cn=Example Corp User Password Policy,cn=Password Policies, cn=config
This can be performed using any LDAP client where you have write permissions to a user’s entry. The following example uses the ldapmodify command in an interactive mode to perform this operation:
$ ldapmodify -D "cn=Directory Manager" -w ****** <ENTER> dn: uid=bnelson,ou=People,dc=example,dc=com <ENTER> changetype: modify <ENTER> replace: ds-pwp-password-policy-dn <ENTER> ds-pwp-password-policy-dn: cn=Example Corp User Password Policy, cn=Password Policies,cn=config <ENTER> <ENTER>
Another method of setting this password policy is through the use of a dynamically created virtual attribute (i.e. one that is not persisted in the OpenDJ database backend). The following definition automatically assigns this new password policy to all users that exist beneath the ou=people container (the scope of the virtual attribute).
dn: cn=Example Corp User Password Policy Assignment,cn=Virtual Attributes,cn=config objectClass: ds-cfg-virtual-attribute objectClass: ds-cfg-user-defined-virtual-attribute objectClass: top ds-cfg-base-dn: ou=people,dc=example,dc=com cn: Example Corp User Password Policy Assignment ds-cfg-attribute-type: ds-pwp-password-policy-dn ds-cfg-enabled: true ds-cfg-java-class: org.opends.server.extensions.UserDefinedVirtualAttributeProvider ds-cfg-filter: (objectclass=sbacperson) ds-cfg-value: cn=Example Corp User Password Policy,cn=Password Policies,cn=config
Note: You can also use filters to create very granular results on how password polices are applied.
Configuring Account Lockout in OpenDJ has more flexibility and as such may be considered to be more powerful than OpenAM in this area. The potential confusion, however, comes when attempting to unlock a user’s account when they have been locked out of both OpenAM and OpenDJ. This is described in the following example.
A Deeper Dive into Account Lockout
Consider an environment where OpenAM is configured with the LDAP authentication module and that module has been configured to use an OpenDJ instance as the authentication database.
OpenDJ Configured as AuthN Database
OpenAM and OpenDJ have both been configured to lock a user’s account after 3 invalid password attempts. What kind of behavior can you expect? Let’s walk through each step of an Account Lockout process and observe the behavior on Account Lockout specific attributes.
Step 1: Query Account Lockout Specific Attributes for the Test User
$ ldapsearch -D "cn=Directory Manager" -w ****** uid=testuser1 inetuserstatus \
sunAMAuthInvalidAttemptsData pwdFailureTime pwdAccountLockedTime
dn: uid=testuser1,ou=test,dc=example,dc=com
inetuserstatus: Active
The user is currently active and Account Lockout specific attributes are empty.
Step 2: Open the OpenAM Console and access the login screen for the realm where Account Lockout has been configured.
OpenAM Login Page
Step 3: Enter an invalid password for this user
OpenAM Authentication Failure Message
Step 4: Query Account Lockout Specific Attributes for the Test User
$ ldapsearch -D "cn=Directory Manager" -w ****** uid=testuser1 inetuserstatus \ sunAMAuthInvalidAttemptsData pwdFailureTime pwdAccountLockedTime dn: uid=testuser1,ou=test,dc=example,dc=com sunAMAuthInvalidAttemptsData:: PEludmFsaWRQYXNzd29yZD48SW52YWxpZENvdW50PjE8L0 ludmFsaWRDb3VudD48TGFzdEludmFsaWRBdD4xMzk4MTcxNTAwMDE4PC9MYXN0SW52YWxpZEF0P jxMb2NrZWRvdXRBdD4wPC9Mb2NrZWRvdXRBdD48QWN0dWFsTG9ja291dER1cmF0aW9uPjA8L0Fj dHVhbExvY2tvdXREdXJhdGlvbj48L0ludmFsaWRQYXNzd29yZD4= inetuserstatus: Active pwdFailureTime: 20140422125819.918Z
You now see that there is a value for the pwdFailureTime. This is the timestamp of when the first password failure occurred. This attribute was populated by OpenDJ.
The sunAMAuthInvalidAttemptsData attribute is populated by OpenAM. This is a base64 encoded value that contains valuable information regarding the invalid password attempt. Run this through a base64 decoder and you will see that this attribute contains the following information:
<InvalidPassword><InvalidCount>1</InvalidCount><LastInvalidAt>1398171500018 </LastInvalidAt><LockedoutAt>0</LockedoutAt><ActualLockoutDuration>0 </ActualLockoutDuration></InvalidPassword>
Step 5: Repeat Steps 2 and 3. (This is the second password failure.)
Step 6: Query Account Lockout Specific Attributes for the Test User
$ ldapsearch -D "cn=Directory Manager" -w ****** uid=testuser1 inetuserstatus \ sunAMAuthInvalidAttemptsData pwdFailureTime pwdAccountLockedTime dn: uid=testuser1,ou=test,dc=example,dc=com sunAMAuthInvalidAttemptsData:: PEludmFsaWRQYXNzd29yZD48SW52YWxpZENvdW50PjI8L0 ludmFsaWRDb3VudD48TGFzdEludmFsaWRBdD4xMzk4MTcxNTUzMzUwPC9MYXN0SW52YWxpZEF0P jxMb2NrZWRvdXRBdD4wPC9Mb2NrZWRvdXRBdD48QWN0dWFsTG9ja291dER1cmF0aW9uPjA8L0Fj dHVhbExvY2tvdXREdXJhdGlvbj48L0ludmFsaWRQYXNzd29yZD4= inetuserstatus: Active pwdFailureTime: 20140422125819.918Z pwdFailureTime: 20140422125913.151Z
There are now two values for the pwdFailureTime attribute – one for each password failure. The sunAMAuthInvalidAttemptsData attribute has been updated as follows:
<InvalidPassword><InvalidCount>2</InvalidCount><LastInvalidAt>1398171553350 </LastInvalidAt><LockedoutAt>0</LockedoutAt><ActualLockoutDuration>0 </ActualLockoutDuration></InvalidPassword>
Step 7: Repeat Steps 2 and 3. (This is the third and final password failure.)
OpenAM Inactive User Page
OpenAM displays an error message indicating that the user’s account is not active. This is OpenAM’s way of acknowledging that the user’s account has been locked.
Step 8: Query Account Lockout Specific Attributes for the Test User
$ ldapsearch -D "cn=Directory Manager" -w ****** uid=testuser1 inetuserstatus \ sunAMAuthInvalidAttemptsData pwdFailureTime pwdAccountLockedTime dn: uid=testuser1,ou=test,dc=example,dc=com sunAMAuthInvalidAttemptsData:: PEludmFsaWRQYXNzd29yZD48SW52YWxpZENvdW50PjA8L0 ludmFsaWRDb3VudD48TGFzdEludmFsaWRBdD4wPC9MYXN0SW52YWxpZEF0PjxMb2NrZWRvdXRBd D4wPC9Mb2NrZWRvdXRBdD48QWN0dWFsTG9ja291dER1cmF0aW9uPjA8L0FjdHVhbExvY2tvdXRE dXJhdGlvbj48L0ludmFsaWRQYXNzd29yZD4= inetuserstatus: Inactive pwdFailureTime: 20140422125819.918Z pwdFailureTime: 20140422125913.151Z pwdFailureTime: 20140422125944.771Z pwdAccountLockedTime: 20140422125944.771Z
There are now three values for the pwdFailureTime attribute – one for each password failure. The sunAMAuthInvalidAttemptsData attribute has been updated as follows:
<InvalidPassword><InvalidCount>0</InvalidCount><LastInvalidAt>0</LastInvalidAt> <LockedoutAt>0</LockedoutAt><ActualLockoutDuration>0</ActualLockoutDuration> </InvalidPassword>
You will note that the counters have all been reset to zero. That is because the user’s account has been inactivated by OpenAM by setting the value of the inetuserstatus attribute to Inactive. Additionally, the third invalid password caused OpenDJ to lock the account by setting the value of the pwdAccountLockedTime attribute to the value of the last password failure.
Now that the account is locked out, how do you unlock it? The natural thing for an OpenAM administrator to do is to reset the value of the inetuserstatus attribute and they would most likely use the OpenAM Console to do this as follows:
OpenAM Edit User Page (Change User Status)
The problem with this approach is that while the user’s status in OpenAM is now made active, the status in OpenDJ remains locked.
$ ldapsearch -D "cn=Directory Manager" -w ****** uid=testuser1 inetuserstatus \ sunAMAuthInvalidAttemptsData pwdFailureTime pwdAccountLockedTime dn: uid=testuser1,ou=test,dc=example,dc=com sunAMAuthInvalidAttemptsData:: PEludmFsaWRQYXNzd29yZD48SW52YWxpZENvdW50PjA8L0 ludmFsaWRDb3VudD48TGFzdEludmFsaWRBdD4wPC9MYXN0SW52YWxpZEF0PjxMb2NrZWRvdXRBd D4wPC9Mb2NrZWRvdXRBdD48QWN0dWFsTG9ja291dER1cmF0aW9uPjA8L0FjdHVhbExvY2tvdXRE dXJhdGlvbj48L0ludmFsaWRQYXNzd29yZD4= inetuserstatus: Active pwdFailureTime: 20140422125819.918Z pwdFailureTime: 20140422125913.151Z pwdFailureTime: 20140422125944.771Z pwdAccountLockedTime: 20140422125944.771Z
Attempting to log in to OpenAM with this user’s account yields an authentication error that would make most OpenAM administrators scratch their head; especially after just resetting the user’s status.
OpenAM Authentication Failure Message
The trick to fixing this is to clear the pwdAccountLockedTime and pwdFailureTime attributes and the way to do this is by modifying the user’s password. Once again, the ldapmodify command can be used as follows:
$ ldapmodify -D "cn=Directory Manager" -w ****** <ENTER> dn: uid=testuser1,ou=test,dc=example,dc=com <ENTER> changetype: modify <ENTER> replace: userPassword <ENTER> userPassword: newpassword <ENTER> <ENTER>
$ ldapsearch -D "cn=Directory Manager" -w ****** uid=testuser1 inetuserstatus \ sunAMAuthInvalidAttemptsData pwdFailureTime pwdAccountLockedTime dn: uid=testuser1,ou=test,dc=example,dc=com sunAMAuthInvalidAttemptsData:: PEludmFsaWRQYXNzd29yZD48SW52YWxpZENvdW50PjA8L0 ludmFsaWRDb3VudD48TGFzdEludmFsaWRBdD4wPC9MYXN0SW52YWxpZEF0PjxMb2NrZWRvdXRBd D4wPC9Mb2NrZWRvdXRBdD48QWN0dWFsTG9ja291dER1cmF0aW9uPjA8L0FjdHVhbExvY2tvdXRE dXJhdGlvbj48L0ludmFsaWRQYXNzd29yZD4= inetuserstatus: Active pwdChangedTime: 20140422172242.676Z
This, however, requires two different interfaces for managing the user’s account. An easier method is to combine the changes into one interface. You can modify the inetuserstatus attribute using ldapmodify or if you are using the OpenAM Console, simply change the password while you are updating the user’s status.
OpenAM Edit User Page (Change Password)
There are other ways to update one attribute by simply modifying the other. This can range in complexity from a simple virtual attribute to a more complex yet powerful custom OpenDJ plugin. But in the words of Voltaire, “With great power comes great responsibility.”
So go forth and wield your new found power; but do it in a responsible manner.
It’s OK to Get Stressed Out with OpenAM
In fact, it’s HIGHLY recommended….
Performance testing and stress testing are closely related and are essential tasks in any OpenAM deployment.
When conducting performance testing, you are trying to determine how well your system performs when subjected to a particular load. A primary goal of performance testing is to determine whether the system that you just built can support your client base (as defined by your performance requirements). Oftentimes you must tweak things (memory, configuration settings, hardware) in order to meet your performance requirements, but without executing performance tests, you will never know if you can support your clients until you are actually under fire (and by then, it may be too late).
Performance testing is an iterative process as shown in the following diagram:
Each of the states may be described as follows:
- Test – throw a load at your server
- Measure – take note of the results
- Compare – compare your results to those desired
- Tweak – modify the system to help achieve your performance results
During performance testing you may continue in this loop until such time that you meet your performance requirements – or until you find that your requirements were unrealistic in the first place.
Stress testing (aka “torture testing”) goes beyond normal performance testing in that the load you place on the system intentionally exceeds the anticipated capacity. The goal of stress testing is to determine the breaking point of the system and observe the behavior when the system fails.
Stress testing allows you to create contingency plans for those ‘worse case scenarios’ that will eventually occur (thanks to Mr. Murphy).
Before placing OpenAM into production you should test to see if your implementation meets your current performance requirements (concurrent sessions, authentications per second, etc.) and have a pretty good idea of where your limitations are. The problem is that an OpenAM deployment is comprised of multiple servers – each that may need to be tested (and tuned) separately. So how do you know where to start?
When executing performance and stress tests in OpenAM, there are three areas where I like to place my focus: 1) the protected application, 2) the OpenAM server, and 3) the data store(s). Testing the system as a whole may not provide enough information to determine where problems may lie and so I prefer to take an incremental approach that tests each component in sequence. I start with the data stores (authentication and user profile databases) and work my way back towards the protected application – with each iteration adding a new component.
Note: It should go without saying that the testing environment should mimic your production environment as closely as possible. Any deviation may cause your test results to be skewed and provide inaccurate data.
Data Store(s)
An OpenAM deployment may consist of multiple data stores – those that are used for authentication (Active Directory, OpenDJ, Radius Server, etc.) and those that are used to build a user’s profile (LDAP and RDBMS). Both of these are core to an OpenAM deployment and while they are typically the easiest to test, a misconfiguration here may have a pretty big impact on overall performance. As such, I start my testing at the database layer and focus only on that component.
Performance of an authentication database can be measured by the average number of authentications that occur over a particular period of time (seconds, minutes, hours) and the easiest way to test these types of databases is to simply perform authentication operations against them.
You can write your own scripts to accomplish this, but there are many freely available tools that can be used as well. One tool that I have used in the past is the SLAMD Distributed Load Generation Engine. SLAMD was designed to test directory server performance, but it can be used to test web applications as well. Unfortunately, SLAMD is no longer being actively developed, but you can still download a copy from http://dl.thezonemanager.com/slamd/.
A tool that I have started using to test authentications against an LDAP server is authrate, which is included in ForgeRock’s OpenDJ LDAP Toolkit. Authrate allows you to stress the server and display some really nice statistics while doing so. The authrate command line tool measures bind throughput and response times and is perfect for testing all sorts of LDAP authentication databases.
Performance of a user profile database is typically measured in search performance against that database. If your user profile database can be searched using LDAP (i.e. Active Directory or any LDAPv3 server), then you can use searchrate – also included in the OpenDJ LDAP Toolkit. searchrate is a command line tool that measures search throughput and response time.
The following is sample output from the searchrate command:
------------------------------------------------------------------------------- Throughput Response Time (ops/second) (milliseconds) recent average recent average 99.9% 99.99% 99.999% err/sec Entries/Srch ------------------------------------------------------------------------------- 188.7 188.7 3.214 3.214 306.364 306.364 306.364 0.0 0.0 223.1 205.9 2.508 2.831 27.805 306.364 306.364 0.0 0.0 245.7 219.2 2.273 2.622 20.374 306.364 306.364 0.0 0.0 238.7 224.1 2.144 2.495 27.805 306.364 306.364 0.0 0.0 287.9 236.8 1.972 2.368 32.656 306.364 306.364 0.0 0.0 335.0 253.4 1.657 2.208 32.656 306.364 306.364 0.0 0.0 358.7 268.4 1.532 2.080 30.827 306.364 306.364 0.0 0.0
The first two columns represent the throughput (number of operations per second) observed in the server. The first column contains the most recent value and the second column contains the average throughput since the test was initiated (i.e. the average of all values contained in column one).
The remaining columns represent response times with the third column being the most recent response time and the fourth column containing the average response time since the test was initiated. Columns five, six, and seven (represented by percentile headers) demonstrate how many operations fell within that range.
For instance, by the time we are at the 7th row, 99.9% of the operations are completed in 30.827 ms (5th column, 7th row), 99.99% are completed in 306.364 ms (6th column, 7th row), and 99.999% of them are completed within 306.364 ms (7th column, 7th row). The percentile rankings provide a good indication of the real system performance and can be interpreted as follows:
- 1 out of 1,000 search requests is exceeding 30 ms
- 1 one out of 100,000 requests is exceeding 306 ms
Note: The values contained in this search were performed on an untuned, limited resource test system. Results will vary depending on the amount of JVM memory, the system CPU(s), and the data contained in the directory. Generally, OpenDJ systems can achieve much better performance that the values shown above.
There are several factors that may need to be considered when tuning authentication and user profile databases. For instance, if you are using OpenDJ for your database you may need to modify your database cache, the number of worker threads, or even how indexing is configured in the server. If your constraint is operating system based, you may need to increase the size of the JVM or the number of file descriptors. If the hardware is the limiting factor, you may need to increase RAM, use high speed disks, or even faster network interfaces. No matter what the constraint, you should optimize the databases (and database servers) before moving up the stack to the OpenAM instance.
OpenAM Instance + Data Store(s)
Once you have optimized any data store(s) you can now begin testing directly against OpenAM as it is configured against those data store(s). Previous testing established a performance baseline and any degradation introduced at this point will be due to OpenAM or the environment (operating system, Java container) where it has been configured.
But how can you test an OpenAM instance without introducing the application that it is protecting? One way is to generate a series of authentications and authorizations using direct interfaces such as the OpenAM API or REST calls. I prefer to use REST calls as this is the easiest to implement.
There are browser based applications such as Postman that are great for functional testing, but these are not easily scriptable. As such, I lean towards a shell or Perl script containing a loop of cURL commands.
Note: You should use the same authentication and search operations in your cURL commands to be sure that you are making a fair comparison between the standalone database testing and the introduction of OpenAM.
You should expect some decrease in performance when the OpenAM server is introduced, but it should not be too drastic. If you find that it falls outside of your requirements, however, then you should consider updating OpenAM in one of the following areas:
- LDAP Configuration Settings (i.e. connections to the Configuration Server)
- Session Settings (if you are hitting limitations)
- JVM Settings (pay particular attention to garbage collection)
- Cache Settings (size and time to live)
Details behind each of these areas can be found in the OpenAM Administration Guide.
You may also find that OpenAM’s interaction with the database(s) introduces searches (or other operations) that you did not previously test for. This may require you to update your database(s) to account for this and restart your performance testing.
Note: Another tool I have started playing with is the Java Application Monitor (aka JAMon). While this tool is typically used to monitor a Java application, it provides some useful information to help determine bottlenecks working with databases, file IO, and garbage collection.
Application + OpenAM Instance + Data Store(s)
Once you feel comfortable with the performance delivered by OpenAM and its associated data store(s), it is time to introduce the final component – the protected application, itself.
This will differ quite a bit based on how you are protecting your application (for instance, policy agents will behave differently from OAuth2/OpenID Connect or SAML2) but this does provide you with the information you need to determine if you can meet your performance requirements in a production deployment.
If you have optimized everything up to this point, then the combination of all three components will provide a full end to end test of the entire system. In this case, then an impact due to network latency will be the most likely factor in performance testing.
To perform a full end to end test of all components, I prefer to use Apache JMeter. You configure JMeter to use a predefined set of credentials, authenticate to the protected resource, and look for specific responses from the server. Once you see those responses, JMeter will act according to how you have preconfigured it to act. This tool allows you to generate a load against OpenAM from login to logout and anything in between.
Other Considerations
Keep in mind that any time that you introduce a monitoring tool into a testing environment, the tool (itself) can impact performance. So while the numbers you receive are useful, they are not altogether acurate. There may be some slight performance degradation (due to the introduction of the tool) that your users will never see.
You should also be aware that the client machine (where the load generation tools are installed) may become a bottleneck if you are not careful. You should consider distributing your performance testing tools across multiple client machines to minimize this effect. This is another way of ensuring that the client environment does not become the limiting factor.
Summary
Like many other areas in our field, performance testing an OpenAM deployment may be considered as much of an art as it is a science. There may be as many methods for testing as there are consultants and each varies based on the tools they use. The information contained here is just one approach performance testing – one that I have used successfully in our deployments.
What methods have you used? Feel free to share in the comments, below.
OpenDJ Indexes Explained
Suppose that you have an OpenDJ directory server with 300,000 entries. And further suppose that the space consumed on your disk for said directory is 1.2 GB and made up of 114 database (*.jdb) files. Suppose that you didn’t plan correctly and you are now running out of space on your hard drive. What should you do? Run to your local System Administrator and beg for him to increase the size of your partition? Before promising to buy him lunch for the next year or offering your first born child to mow his lawn, look to see if you actually need that much space in the first place.
In general, the size of your database is based on three things:
- The number of entries in your database
- The size of an average entry
- Your indexing strategy
The first two items are relatively straight forward as you probably have a good idea of your data profile, but an improper indexing strategy can take you by surprise and may actually cause more harm than good. Indexes are used to increase search performance based on application search filters. Lack of necessary indexes can impact performance, increase aggravation, and lead to calls in the middle of the night. But maintaining indexes that are never used can unnecessarily increase disk space and impact the performance of write operations.
Default Indexes
OpenDJ comes with the following default indexes:
|
Attribute |
Presence |
Equality |
Substring |
Order |
|
| aci |
x |
operational | |||
| cn |
x |
x |
standard | ||
| ds-sync-conflict |
x |
operational | |||
| ds-sync-hist |
x |
operational | |||
| entryUUID |
x |
operational | |||
| givenName |
x |
x |
standard | ||
|
x |
x |
standard | |||
| member |
x |
standard | |||
| objectclass |
x |
standard | |||
| sn |
x |
x |
standard | ||
| telephonenumber |
x |
x |
standard | ||
| uid |
x |
standard | |||
| uniquemember |
x |
standard |
Indexes on operational attributes are necessary to make OpenDJ run efficiently. You should never modify these unless instructed to do so by ForgeRock support. Standard attributes, however, are used to increase external application search performance and should reflect the types of searches being performed by your own applications. Default attributes (and index types) are based on ForgeRock’s observations of what most of its customers use, but you may not be like most of their customers and while maintaining some index types can be relatively benign, others (like SUBSTRING) may have a more dramatic effect.
Using Indexes to Increase Search Performance
From a high level perspective, indexes are used to identify likely candidates that might be found as a result of an application’s search filter. Assume, for instance, that you have a simple phone book application that allows you to search for phone numbers based on first name and last name. A filter to locate all entries that have a first name (givenname) of “Bill” would be:
(givenname=Bill)
But not every entry in your OpenDJ server has a givenname attribute with a value of “Bill” so looking at every entry to see if it matches may take a lot of time. But how can you avoid looking at every entry? The answer is simple; you create an index for the givenname attribute to narrow down your search. Simply add an EQUALITY index for the givenname attribute and OpenDJ will associate all entries in its database with those that have a particular value. The following is a conceptual representation of how OpenDJ will make this association:
givenname=Bill: 1,3,9,22
givenname=Ralph: 2,11
givenname=Wally: 4,5,6,7,8,10
givenname=Wild Bill: 12
givenname=Billy: 13,15,21
givenname=Silly: 14
….
This demonstrates that the givenname value for entries 1, 3, 9, and 22 are all “Bill”. When OpenDJ receives a search for all entries that have a first name of “Bill”, it immediately knows that a match is found in records 1, 3, 9 and 22. It doesn’t even look at the other entries. In a database that contains hundreds of thousands of entries, this can drastically increase search performance.
This is all well and fine, but how can indexes actually impact us?
How Unnecessary Indexes Can Hurt You
Imagine that you met a coworker at a party last night and you didn’t quite get his name. You seem to remember his name was Bill, but you heard people call him Bill, Billy, Wild Bill, and even Bill-O-Rama. You want to look him up, but you can’t because you really aren’t sure about his first name. Hopefully your same phone book application allows you to search for all entries that contain the string, “Bill”. If so, an EQUALITY index would not work as you really don’t know the specifics of what you are looking for. In this case you would create a SUBSTRING index for the givenname attribute. In so doing, OpenDJ will associate substrings with entries as follows:
givenname=*Bill: 1,3,9,12,13,15,21,22
givenname=*illy: 13,14,15,21
givenname=*Wild: 12
givenname=*ild : 12
givenname=*ld B: 12
givenname=*d Bi: 12
givenname=*Ralp: 2,11
givenname=*alph: 2,11
givenname=*ally: 4,5,6,7,8,10
….
| Note: OpenDJ created entries for substrings consisting of four or more characters. These include the beginning of string (^) and end of string ($) characters; the shorter the string, the fewer entries that are created. Imagine how many entries would be generated if the attribute contained a value of ‘supercalifragilisticexpialidocious’! |
There are times when maintaining indexes may actually be more costly than if you were to perform an unindexed search (i.e. evaluate every entry in the directory server). To prevent this, OpenDJ provides the ds-cfg-index-entry-limit configuration parameter that allows you to define an upper limit on the number of indexes maintained for an attribute. There is a global (default) value of 4000 for this parameter, but it may also be configured on a per indexed attribute basis. A value of 4000 means that OpenDJ will stop generating index values once it reaches 4000 index entries. A minor problem is that you can maintain up to 4000 index entries for attributes that are never included in a search filter. A bigger problem, however, is that each time a write operation is performed that includes the indexed attribute, the indexes for that attribute are rebuilt. If your OpenDJ server is subject to extensive write operations, then you may be constantly writing and rewriting your database files which may impact write performance and ultimately overall server performance. (See “Unlocking the Mystery behind the OpenDJ User Database” for more information on how, when, and why the database files are rewritten on change operations.)
Determining Whether an Index is Necessary or Not
A recommendation is to maintain only those indexes for attributes that are included in your application search filters. The types of indexes selected should reflect the manner of searches being performed by your application. To determine this, you can review your LDAP-enabled applications and attempt to ascertain the types of filters it may be producing; but this may not be so obvious.
A more realistic approach is to come up with a “best guess” and then monitor your server to see if your guess was accurate or not. You can then add, delete, or modify attribute indexes based on your findings.
When to Add Indexes
You should monitor your access logs for searches that take a long time and consider adding indexes for search times that you find unacceptable. This can be seen in the etime (or elapsed time) value which is displayed in milliseconds (by default). This is subject to your own SLAs, but etimes greater than 5 milliseconds may be considered unacceptable. If you see etimes in the order of seconds (as shown below) then you definitely need to investigate further.
[31/Dec/2013:18:07:21 +0000] SEARCH RES conn=2231288 op=6 msgID=502 result=0 nentries=1 unindexed etime=5836
This access log entry indicates that the search took 5.8 seconds to complete. One reason why it took so long was that it was an unindexed search (as noted by the “unindexed” tag in the entry). To determine the filter associated with this search, you need to search backwards in the access log and find the corresponding SEARCH REQ for this connection (conn=2231288) and this operation (op=6).
[31/Dec/2013:18:07:15 +0000] SEARCH REQ conn=2231288 op=6 msgID=502 base="ou=people,dc=example,dc=com" scope=wholeSubtree filter="(&(&(exampleGUID=88291000818)(objectclass=inetorgperson)))" attrs="*"
This access log entry indicates that the filter used to perform the search is
(&(&(exampleGUID=88291000818)(objectclass=inetorgperson)))
OpenDJ contains a default EQUALITY index for objectclass so assuming that you have not modified the default indexes, then the unindexed attribute causing the problem is exampleGUID. Now that you have identified the culprit, should you run right out and create an EQUALITY index for this attribute? Not necessarily. It really depends on how often you see searches of this type appear in the access logs and what their impact might be. You don’t want to maintain exampleGUID indexes if your application only searches on this attribute once in a blue moon. If, however, you see this type of search on a consistent basis, you might want to consider adding an index.
When to Remove Unnecessary Indexes
It is relatively straightforward to determine when to add indexes, but how do you know when you are maintaining unnecessary indexes? Unfortunately, OpenDJ does not include utilities to tell you this, but it is possible to determine unused indexes by once again, reviewing the search filters in the access logs. One approach to accomplishing this would be to perform the following:
- Determine attribute names included in the search filter.
- Determine type of search being performed (EQUALITY, SUBSTRING, PRESENCE, etc.)
- Determine the frequency of the searches.
- Compare the searches to the already configured indexes.
- Remove unnecessary indexes (if desired).
It is pretty easy to write a script to perform these steps and fortunately one has already been written by Chris Ridd to perform steps 1 through 4. His topfilters script can be found here. Once armed with the information from his script you would simply compare it to what you already have configured for OpenDJ.
How to Determine Current Indexes and Index Types
Current indexes are reflected beneath the cn=config suffix of your OpenDJ server. You can either query this suffix as the rootDN user or you can simply view the contents of the config.ldif file to see what indexes have been configured.
dn: ds-cfg-attribute=givenName,cn=Index,ds-cfg-backend-id=userRoot,cn=Backends,cn=config
objectClass: top
objectClass: ds-cfg-local-db-index
ds-cfg-index-type: equality
ds-cfg-index-type: substring
ds-cfg-attribute: givenName
Another method is to use the dbtest command to obtain a more detailed analysis on each index. The dbtest command can be found in the bin directory of your OpenDJ installation. An example execution of this command might be:
/opt/opendj/bin/dbtest list-index-status -b "dc=example,dc=com" -n userRoot
Execution of this command will return each index, its type, the database it is associated with, whether the index is valid or not, and the number of records associated with the index. It will also detail the undefined index keys that are not maintained due to the ds-cfg-index-entry-limit being reached for that attribute.
You can take the data returned from Chris’ script, compare it with the data found for those indexes you are currently maintaining and make an intelligent decision as to whether you want to modify your indexes in any way.
Should you delete any indexes that you believe are not being used? Again, not necessarily. Your access logs only reflect a point in time and may not provide a comprehensive listing of application search filters. You should always carefully consider removing existing indexes, but if you find that you have made a mistake, you can always monitor the access log for searches that are taking an unacceptably long time – or wait for that 3:00 am phone call to let you know.
Configuring Indexes
If you do decide it is necessary to update your indexes, then the best approach is to do so using the OpenDJ Control Panel or the dsconfig command line tool. You should never update the config.ldif file directly.
The following provides an overview of how to add a new index for the exampleGUID attribute. The index type is set to EQUALITY.
/opt/opendj/bin/dsconfig create-local-db-index --port 4444 --hostname ldap1.example.com --bindDN "cn=Directory Manager" --bindPassword password
--backend-name userRoot --index-name exampleGUID --set index-type:equality
--trustAll
The following provides an overview of how to remove an existing EQUALITY index type from an existing mail index.
/opt/opendj/bin/dsconfig set-local-db-index-prop --port 4444 --hostname ldap1.example.com --bindDN "cn=Directory Manager" --bindPassword password
--backend-name userRoot --index-name mail --remove index-type:equality
--trustAll
If you would rather remove the entire mail index, use the following command, instead.
/opt/opendj/bin/dsconfig delete-local-db-index --port 4444 --hostname ldap1.example.com --bindDN "cn=Directory Manager" --bindPassword password
--backend-name userRoot --index-name mail --trustAll
Rebuilding Indexes
OpenDJ automatically updates indexes on LDAP operations that update the database. Adding or deleting an index or an index value is a configuration change, however, and does not affect index values already in the database. If you delete an index type, existing index values will remain in the database until you rebuild the index. The same is true if you add a new index or index type. Indexes will not be added for existing database entries until you rebuild the index.
As such, any configuration changes to indexes should be followed by a rebuilding of the appropriate index. The following provides an overview of how to rebuild the mail index once its configuration has changed.
/opt/opendj/bin/rebuild-index -p 4444 -D "cn=Directory Manager" -w password -b "dc=example,dc=com" --index mail --start 0 --trustAll
| Note: It is not necessary to stop the OpenDJ instance before performing this task. It has been my experience, however, that if you are able to stop the server you might want to consider doing so. If so, then you do not need to specify a start time, bind credentials, or the trust acceptance. These are not necessary as you will be initiating the connection immediately and directly to the database. |
Debugging Index Problems
There are times when you may see performance problems that indicate that you are performing an unindexed search, but when you look at the indexes, you find that the appropriate index has been configured.
| Note: This problem typically occurs when you do not rebuild the index after you have configured it. Essentially, there is already data in the database when the indexes were applied. In such cases, OpenDJ will not attempt to update the index until an initial rebuild-index has been performed. |
One method of debugging this problem is to use the debugsearchindex capability in OpenDJ.
If you perform your search and request that the debugsearchindex attribute be returned as follows:
/opt/opendj/bin/ldapsearch -D "cn=Directory Manager" -w cangetin -b "ou=people,dc=example,dc=com" -s sub "(&(&( exampleGUID=88291000818)(objectclass=inetorgperson)))" debugsearchindex
OpenDJ will emulate the search, but will not actually perform it against the database. Instead, it will tell you how the search is to be performed and whether or not the values are indexed or not as follows:
dn: cn=debugsearch
debugsearchindex: filter=(&(&( exampleGUID=88291000818)[NOT-INDEXED](object
Class=inetorgperson)[INDEX:objectClass.equality][LIMIT-EXCEEDED])[NOT-INDEXED])
[NOT-INDEXED] scope=wholeSubtree[LIMIT-EXCEEDED:30] final=[NOT-INDEXED]
If you see something like this but your configuration tells you that the indexes have been configured, then it is time to send your LDAP administrator to training.
Summary
As with most middleware products knowing when and how to configure indexes can be as much of an art as it is a science. You should follow best practices where possible, but as with other products you should monitor your server to see if those practices apply to you and react where appropriate.
The Case of the Mysteriously Creeping Database
While teaching a recent ForgeRock OpenDJ class, a student of mine observed an interesting behavior that at first seemed quite odd. While rebuilding his attribute indexes, the student found that the overall database size seemed to grow each time he performed a reindex operation. What seems obvious to me now sure made me scratch my head as I scrambled for an answer. I am sharing my findings here in the hopes that others will either a) find this information useful or b) find comic relief as to my misfortune.
| Note: If you are unclear about the information contained in OpenDJ’s database files, then I highly recommend that you read my posting entitled, Unlocking the Mystery behind the OpenDJ User Database. In that article I describe the overall structure of the Berkeley DB Java Edition database used by OpenDJ and how both entries (and indexes) are maintained in the same database. |
In OpenDJ, the rebuild-index command is used to update any attribute indexes contained in the OpenDJ database. This is necessary after you make a configuration change that affects indexes (such as modifying the index entry limit). Indexes are database specific and you can elect to rebuild a single attribute index or rebuild all attribute indexes for a particular database.
The following syntax is used to rebuild ALL indexes associated with the dc=example,dc=com suffix and its use is what caused the frantic head-scratching to occur:
$ rebuild-index -h ldap.example.com -p 4444 -D "cn=Directory Manager" -w password -b dc=example,dc=com --rebuildAll
Mystery
The student observed (and questioned) that every time he rebuilt the indexes, the aggregated size of the *.jdb files actually increased by some factor. In the case of a rebuild-all, it was about 18 MB each time he ran the command; in the case of rebuilding a single index, it was only about 3 MB each time. But the increase was consistent each time he rebuilt the index(es). This continued to occur until it reached a certain size at which time the consumption fell back to its original size (in our observations this occurred at roughly 200 MB when using the rebuild-all option).
The following details the output of the du -sh command on the userRoot database each time the rebuild-index command was run:
- 124 MB
- 142 MB (+ 18MB)
- 160 MB (+ 18MB)
- 178 MB (+ 18MB)
- 200 MB (+ 22MB)
- 123 MB (- 77MB)
This trend was consistent over several iterations.
We continued testing and observed that in addition to the increasing size, the database files on the file system (*.jdb) were changing as well. What was once 000000001.jdb and 000000002.jdb now became 000000002.jdb file and 000000003.jdb and later became 000000003.jdb file and 000000004.jdb. This occurred at the same time that we dropped back down to the 123 MB size and was the clue that unlocked the mystery.
Solution
Unlike the Berkeley Sleepycat database used in OpenDJ’s forefathers, when data is modified in the OpenDJ database, it is not immediately removed from the database. Instead it is marked for removal and the record essentially becomes inactive. Updated records are then appended to the end of the database in a log file fashion.
This process continues until OpenDJ cleaner threads detect that a database file contains less than 50% active records. Once that occurs, the cleaner threads migrate all active records from the file and append them to the end of the last file in the OpenDJ database (a new file is created if necessary). Once migrated, the cleaner threads delete the database file containing the stale entries.
During the rebuild process, old index values in each of the *.jdb files are marked as inactive and new indexes are added to the database. Simply marking these indexes as inactive does not eliminate their existence in the database and they continue to consume disk space. This process continues until the point where the cleaner threads detect that old indexes account for > 50% of the database entries. At this point, the migration process occurs, new *.jdb files are created to store the new indexes, old stale *.jdb files are deleted (hence the *.jdb file name changes), and the disk space is returned.
Word from ForgeRock’s OpenDJ Product Manager, Ludo Poitou confirmed my suspicions in the following response:
When an index is rebuilt, the whole btree is marked as deleted. But since it actually represents specific records of the database files, they will only be collected when the file itself reaches the threshold that triggers recollection.
With small databases, you will see the behavior you’re describing. With larger databases, this will be less noticeable as the amount of index records will be larger and cleanup point may be reached faster.
So there you go, mystery solved!
What do OpenDJ and McDonald’s Have in Common?
The OpenDJ directory server is highly scalable and can process all sorts of requests from different types of clients over various protocols. The following diagram provides an overview of how OpenDJ processes these requests. (See The OpenDJ Architecture for a more detailed description of each component.)
Note: The following information has been taken from ForgeRock’s OpenDJ Administration, Maintenance and Tuning Class and has been used with the permission of ForgeRock.
Client requests are accepted and processed by an appropriate Connection Handler. The Connection Handler decodes the request according to the protocol (LDAP, JMX, SNMP, etc.) and either responds immediately or converts it into an LDAP Operation Object that is added to the Work Queue.
Analogy: I like to use the analogy of the drive-through window at a fast food restaurant when describing this process. You are the client making a request of the establishment. The Connection Handler is the person who takes your order; they take your request and enter it into their ordering system (the Work Queue). They do not prepare your food; their jobs are simply to take the order as quickly and efficiently as possible.
Worker Threads monitor and detect items on the Work Queue and respond by processing them in a first in, first out fashion. Requests may be routed or filtered based on the server configuration and then possibly transformed before the appropriate backend is selected.
Analogy: Continuing with the fast food analogy, the Worker Threads are similar to the people who prepare your food. They monitor the order system (Work Queue) for any new orders and process them in a first in, first out fashion.
Note: OpenDJ routing is currently limited to the server’s determination of the appropriate backend. In future versions, this may take on more of a proxy or virtual directory type of implementation.
The result is returned to the client by the Worker Threads using the callback method specified by the Connection Handler.
Analogy: Once your order is completed, the food (or the results of your request) is given to you by one of the Worker Threads who has been tasked with that responsibility. This is the only place where the analogy somewhat breaks down. In older fast food restaurants (ones with only one window) this may sometimes be the person who took your order in the first place. In our analogy, however, the Connection Handler never responds to your request. This model is more closely attuned to more recent fast food establishments where they have two windows and there is a clear delineation of duties between the order taker (Connection Handler) and the one who provides you with your food (the Worker Thread).
Other services such as access control processing (ACIs), Logging, and Monitoring provide different access points within the request processing flow and are used to control, audit, and monitor how the requests are processed.
So, what do OpenDJ and McDonald’s have in common? They are both highly efficient entities that have been streamlined to process requests in the most efficient manner possible.
Check out ForgeRock’s website for more information on OpenDJ or click here if you are interested in attending one of ForgeRock’s upcoming training classes.
The OpenDJ Architecture
An understanding of the components that make up the OpenDJ Architecture is useful for administering, configuring, or troubleshooting the OpenDJ server.
The following information has been taken from ForgeRock’s OpenDJ Administration, Maintenance and Tuning Class and has been used with the permission of ForgeRock.
The OpenDJ server has been developed using a modular architecture in which most or all components are written to a well-defined specification. This image above provides an overview of these components. The following sections provide a brief description of some of the more prevalent components shown in this image.
Configuration Handler
The OpenDJ Configuration Handler is responsible for managing configuration information within OpenDJ’s configuration files (i.e. config.ldif). Configuration information may impact one or more components; as such, the Configuration Handler is responsible for notifying appropriate components when a configuration change occurs.
Connection Handlers
Connection and request handlers manage all interaction with LDAP clients. This includes accepting new connections and reading and responding to client. Connection handlers are responsible for any special processing that might be required for this communication, including managing encryption or performing protocol translation. It is possible to have multiple concurrent implementations active at any given time and as such, OpenDJ includes connection handlers which support various forms of communication that clients use to interact with the server (JMX, LDAP, LDAPS, LDIF, SNMP). Administrators have the ability to enable or disable these connection handlers to support their client environment.
Note: ForgeRock is currently working on REST and JSON interfaces to provide direct access to directory server data.
Backend Databases
Connection handlers place client requests onto OpenDJ’s Work Queue. Worker threads detect requests placed on the work queue and are responsible for performing the processing necessary to respond to the request. Today’s directory servers must be able to handle a tremendous number of requests in a short period of time; as such, OpenDJ’s Work Queue has been built to be both highly efficient and provide high performance.
A backend database serves as a repository for searching, retrieving, and storing directory data. OpenDJ supports multiple backends including those considered typical databases (such as Oracle, MySql, and Berekely DB) as well as file-based and memory-based backends. There can be multiple backend databases active at any given time, each of which handle mutually exclusive subsets of data (selection of the appropriate database is based on the root suffix specified in the operation). OpenDJ facilitates interaction with these backends and provides tools for enabling, disabling, creating, removing, backing up, and restoring the databases independently from each other without impacting other backends.
Note: Backends may consist of local or remote repositories (i.e. the database is stored on a remote machine). This can be found in cases where the backend interacts with a proxy or a virtual server. Support for proxy and virtual server backends are scheduled for a future release.
Loggers
OpenDJ has a robust logging capability that allows server information to be retained in various repositories. The most common loggers are as follows:
- Access Logger – stores server operations (binds, searches, modifications, etc.)
- Error Logger – stores warnings, errors, and significant events that occur with the server
- Debug Logger – records debug information when the server is run with debugging enabled and Java assertions are active.
Multiple loggers can be configured for each of these and each logger may be actively storing different information (filtered or not) in different formats in different repositories.
Note: Some error loggers can be used as an alerting mechanism to actively notify administrators of potential problems.
SASL Handlers
The LDAP protocol supports two methods that clients may use to authenticate to the server:
- LDAP simple authentication
- Simple Authentication and Security Layer (SASL)
SASL is an authentication framework that supports multiple authentication mechanisms including ANONYMOUS, CRAM-MD5, DIGEST-MD5, EXTERNAL, GSSAPI, and PLAIN.
OpenDJ includes a set of handlers that implement each of these SASL mechanisms in order to determine the identity of the client.
Access Control
OpenDJ contains an access control module that is used to determine if a client is permitted to perform a particular request or not.
Password Storage
OpenDJ includes several password storage modules that can be used to obscure user passwords using a reversible or one-way algorithm. Password storage schemes encode new passwords provided by users so that they are stored in an encoded manner. This makes it difficult or impossible for someone to determine the clear-text passwords from the encoded values. They can also be used to determine whether a clear-text password provided by a client matches the encoded value stored in the server.
Password Complexity
OpenDJ includes a series of modules that define logic used to determine whether a user’s password meets minimum requirements or not.
Syntax and Matching Rules
Attributes must follow a particular syntax and search filters determine matches based on a set of matching rules. OpenDJ contains a set of syntaxes and matching rules that define the logic for dealing with different kinds of attributes.
Database Cache
Interacting with data in memory is much faster than interacting with data on disk. As such, OpenDJ includes a database caching module that loads directory data into memory.
Check out ForgeRock’s website for more information on OpenDJ or click here if you are interested in attending one of ForgeRock’s upcoming training classes.
 |  |  |  |  |
Easy Identity 