X.500 Standard status
X.509 Related activities
How to be involved
Tutorial section 1
Tutorial section 2
Other PKI organizations
This whitepaper has been cross-posted from the Isode website.
This white paper looks at issues related to replication, when building a highly distributed and replicated directory. It argues that X.500 DISP (Directory Information Shadowing Protocol) is the best solution to this problem. This paper looks particularly at military directory, which has strong requirements for highly replicated directory. The paper is also applicable to other environments.
ACP 133 Directory is a key component of most modern military communication and messaging systems. ACP 133 defines a complex military schema, based on X.500, which provides support for both applications and information services. This international core specification is used as the basis for national and NATO directories. Military directories have two characteristics that are key for this paper, that apply to both tactical and strategic directories:
These fundamental requirements drive procurement and solutions. There are two further considerations that are important:
X.500 DISP (Directory Information Shadowing Protocol) is the standard replication protocol that is a part of X.500 and of ACP 133. It is the only open standard for directory replication. DISP has a number of important features:
Most directory deployments use directories from a single vendor. They will use the directory replication supported by that directory vendor. There are many complex and highly replicated single vendor directory solutions using DISP. Most organizations choosing to procure directory choose to have a single vendor.
Dealing with multi-vendor solutions always adds complexity. Experience has shown that there is one fundamental reason why multi-vendor directory replication is hard, and that is schema consistency. In a single vendor deployment, schema will generally be configured in the same way for each server. In a multi-vendor deployment, there will tend to be differences including:
When data is being read by a directory client, such inconsistencies do not usually cause significant problems. Where data is being replicated, minor schema inconsistencies between the data being replicated and the schema definitions in the directory server into which the data is being replicated can lead to problems. A directory, like any other database, needs to have the data it stores in line with its schema. In order to make things work, schema needs to be aligned, and information not needed by the consumer filtered out.
While this is the hardest problem for multi-vendor deployments, Isode believes that it is quite practical to address it in an operational environment.
At Joint Warrior Interoperability Demonstration (JWID) 2004, there was a multi-vendor demonstration of directory replication using X.500 DISP. While this was a significant success in demonstrating DISP interoperability, the effort to make it work was much higher than it should really have been. There are two key lessons that vendors supporting DISP should take away from JWID 2004.
Basic DISP protocol interoperability was very good. A http://www.x500standard.com/pmwiki.php?n=Participate.Replicated?action=edittypical problem scenario was that data would be sent correctly, but would not be accepted by the server. It would often take significant detective work to find out why the data was not accepted. Once the reason was determined, it was usually straightforward to correct the problem. Good error reporting on DISP errors is extremely important, and some products are lacking in this area.
Jon Postel, who designed many of the core Internet protocols, had an implementer's dictum "Be strict in what you send and be tolerant in what you receive". This excellent guideline is particularly appropriate for DISP. While schema consistency is important, overly fussy DISP Consumers (products receiving a DISP update) make deployment harder.
DISP advocates can argue the JWID 2004 showed that DISP is the right way forward. DISP detractors can argue that JWID 2004 showed that DISP is too complex to deploy. I think that there two important lessons for customers:
The difficulties showed clearly that this deployment is not something that is widely done in the field. Customers need to remember that most directory and DISP deployments are single vendor and that multi-vendor deployments have special requirements. In order to get multi-vendor systems to work, customers need to procure in a manner which will achieve this. It is naive for customers to procure independent (single vendor) implementations and then expect them to work together. Requirements for interworking with external directories using DISP need to be included in procurements.
There are many in the directory industry who are critical of DISP, and in particular some major directory vendors whose products do not support DISP. This section addresses some of the "myths" that have been propagated about DISP.
This is nonsense. DISP needs implementing with care, and some early implementations were very poor. DISP's incremental update makes very efficient use of bandwidth, and Isode customers use DISP every day for replicating multi-million entry directories.
DISP is quite complex, but most of its features are very useful for configuring and managing replication in non-trivial configurations. The IETF attempted to standardize replication in its LDUP initiative. They rejected DISP as too complex and proceeded to develop specifications of much greater complexity. The LDUP activity was eventually abandoned.
DISP contains useful security features. These are not widely implemented, primarily because they are less important for single vendor deployment.
JWID 2004 shows that this is not true.
It has been argued that the Total Update feature of DISP is bad design. It is true that it has operational issues with very large replicated areas (millions of entries). Isode provide an out of band update mechanism for this type of deployment. For smaller replicated areas (hundreds of thousands of entries and less) it is a very useful feature. It enables automatic setup of replication, and for automatic reset in the event of certain types of failure and corruption.
Dealing with schema inconsistency in a multi-vendor replication configuration is a problem that cannot be avoided. There are two basic approaches, independent of the replication mechanism used.
This approach leaves the two directories as they are, and then uses an integration mechanism, typically directory synchronization (meta directory), which deals with the inconsistencies. An example of where this would be a good approach is a commercial organization where two business units have developed directories using different products. Reasons:
This approach appears best in the military environment. Reasons:
The strategy of fixing the inconsistency is independent of the replication mechanism used, and the work to make the fixes will be independent of the mechanism used.
Most alternatives that have been proposed to DISP are based on LDIF (LDAP Data Interchange Format). They typically use an out of band data transfer mechanism, coupled with periodic, supplier initiated incremental replication of a single subtree, without mechanisms for detecting updated loss. The basic replication approach is much less flexible than that offered by DISP. This restriction may be acceptable, but much more serious problems are expected in relation to the underlying LDIF format:
It is naive to believe that wide vendor support for LDIF makes it a good format for replication.
In looking at approaches to replication, it is important to consider the driving requirements for replication in a military directory:
DISP is an bandwidth efficient protocol, with many configuration and management features. The key benefit of DISP for meeting military requirements is that it is a part of an ACP 133 directory server product. Configuring replication is integrated with the directory server, and there are no third party components sitting between a pair of replicating directory servers. This clean architecture is key to meeting requirements 1 and 3.
If directory replication uses third party products or custom integration scripting, it is going to be more complex to configure and will introduce a number of failure points and additional components to be managed. Any architecture along these lines is going to be grossly inferior to using integrated direct replication.
Data filtering is important for military directory deployments. When data is replicated, it is important to optimize performance over slow links, and to distribute data on a need to know basis and in line with policy.
Data filtering is a key capability offered by DISP. While sophisticated data transformation is best done by a directory synchronization product (meta directory), standard directory replication is an ideal place to perform data filtering. Robustness is improved by controlling data filtering without the need to integrate a third party product. As well as standard DISP capabilities, such as attribute and object class filtering, it also makes sense for vendors to extend directory filtering to support specific markets. For example, support of the UK Military "Publish To" attribute, which controls entry level replication fits well with DISP replication.
This paper has set out why DISP is the best approach for replicating data in a highly distributed and replicated directory. It also explains key requirements, so that those building replicated directories can procure necessary functionality from directory vendors.