Individual Submission to ldup working group R. Harrison Internet Draft J. Sermersheim Document: draft-rharrison-lburp-01.txt Novell, Inc. Category: Proposed Standard March, 2000 LDAP Bulk Update/Replication Protocol Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. 1. Abstract The protocol described in this document allows an LDAP client (a genuine client or an LDAP server acting as a client) to perform a bulk update to a replica on an LDAP server. The protocol groups a set of update operations using the LDAP framed protocol requests defined in [FRAMING] to notify the client that the update operations in the framed set are related. The update operations within the framed set are LDAP v3 extended operations each containing a sequence number and one or more update LDAP v3 update operations. The sequence number allows the server to process the update operations in the proper order even when they are sent asynchronously by the client, and the update operations can be grouped within the extended request to maximize the efficiency of client-server communication. The protocol may be used to initialize all of the entries in an LDAP replica or to incrementally change the existing entries in an LDAP replica. It is suitable for client utilities that need to efficiently initialize a replica with many entries or efficiently make a substantial set of update changes to a replica. It is also suitable as a protocol for replication between a single master replica and its slave replicas. Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 1 LDAP Bulk Update/Replication Protocol March, 2000 2. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [ReqsKeywords]. The term supplier applies to an LDAP client or and LDAP server (acting as a client) that supplies a set of update operations to a consumer. The term consumer applies to an LDAP server that consumes (processes) the update operations sent to it by a supplier. 3. Motivation for protocol This protocol arose from the need to allow LDAP clients to efficiently present large quantities of updates to an LDAP server and have the LDAP server efficiently process them. This protocol introduces a minimum of new operational functionality to the LDAP protocol since the update requests sent by the client encapsulate standard LDAP [LDAPv3] update operations. However, this protocol greatly facilitates this process by allowing the client to present the update operations asynchronously and still allow the server to maintain proper ordering of the operations. It also allows the server to recognize the clientÆs intent to perform a potentially large set of update operations and then to change its processing strategy to be more efficient than it otherwise could be. In effect, this protocol gives a hint to the server that the LDAP operations framed within it can be treated in a special way because they are related to each other. The server may then take actions that would not otherwise be practical to speed the processing of the updates. Examples of such actions include refusing to perform operations for other clients during the update sequence and grouping update operations into a single transaction rather than applying them to the DIT singly. Additionally, this protocol deals with a common interoperability problem in this space caused by implementations having different requirements and abilities for handling linear and circular dependencies as entries are created. A common application of this protocol is anticipated to be the initialization or update of an LDAP replica from a set of records represented in an [LDIF] file. Due to the abilities of various implementations, these files often contain "out-of-order" records where an entry is created before its parent or it is made a member of a group before the member's entry has been created. Some implementations create temporary holding objects to deal with these issues, but others do not. This protocol allows the server to reorder update operations in a limited way to deal with such cases. Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 2 LDAP Bulk Update/Replication Protocol March, 2000 4. Overview of protocol The bulk replication/update protocol utilizes framing described in [FRAMING] to group a set of update operations to be applied to a naming context. The update operations are sent via LDAP v3 extended operations, each of which contains a sequence number and a list of one or more update operations to be performed by the consumer. Except for the fact that they are grouped together as part of a larger LDAP v3 extended request, the update operations in each subset are normal LDAP v3 update operations and use the encoding specified in [LDAPv3]. 4.1. Update Initiation The protocol is initiated when a supplier sends a StartFramedProtocolRequest extended operation to a consumer to notify it that the following stream of LDAP update operations are to be treated as a unit of update information. The consumer responds to the StartFramedProtocolRequest with a StartFramedProtocolResponse. 4.2. Update Stream After the consumer responds with a StartFramedProtocolResponse extended operation, the supplier sends a stream of LBURPOperationRequest extended operations to the consumer. This stream MAY be sent asynchronously to maximize the efficiency of the transfer. Except in certain circumstances allowed by the protocol to deal with linear and circular dependency issues, the consumer processes each LBURPOperationRequest in the order it was sent by the supplier by applying the LDAP v3 update operations contained within it to the DIT. (The sequence number in each request is utilized to ensure that proper ordering between LBURPOperationRequests is maintained even when the requests are sent asynchronously.) As each LBURPOperationRequest is completed, the consumer sends a LBURPOperationResponse to the supplier indicating the success or failure of the operation. 4.3. Update Termination When the supplier has sent all LBURPOperationRequests, it sends an EndFramedProtocolRequest extended operation to the consumer to terminate the update stream. The consumer responds with an EndFramedProtocolResponse extended operation, and the update is complete. 4.4. Update Styles Two styles of update--full and incremental--are defined. Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 3 LDAP Bulk Update/Replication Protocol March, 2000 Full update creates a completely new set of entries in the target replica; any existing entries in that replica at the start of the StartFramedProtocolRequest operation are removed before the LDAP requests in the replication stream are processed. The only LDAP operation allowed in a full update stream is add. After a server receives and acknowledges a StartFramedProtocolRequest for the LBURPFullUpdateProtocol, the LDAP server MAY choose to not service LDAP requests other than those contained in the update stream for the replica which is being initialized until that update stream is completely processed. Incremental update performs a series of incremental changes to the replica; existing entries in the replica are only affected if explicitly modified in some way by the update operations contained in the update stream. All update operations defined in [LDAPv3]-- add, modify, delete, and moddn--are allowed in the incremental update stream. When a server is processing an incremental update stream, the replica that is being modified MAY choose to not service LDAP requests other than those contained in the update stream until that update stream is completely processed. 4.5. Applicability of Protocol No attempt is made to deal with the issues associated with multiple- master replicas or to maintain state information such as modification times of entries or attribute values in such a way that updates to the same entry on multiple master replicas can be correctly ordered. For this reason convergence of data between all replicas can only be assured in a single-master replication environment. 5. High-level Description of Protocol Flow The following section provides a high-level overview of the replication protocol. Throughout this section, the client or server acting as supplier is indicated by the letter "S", and the server acting as consumer is indicated by the letter "C". The construct "S -> C" indicates that the supplier is sending an LDAPv3 operation to the consumer, and "C -> S" indicates that the consumer is sending an LDAPv3 operation to the supplier. S -> C: LDAP bind operation (identity and credentials used are implementation-defined) C -> S: Bind response S -> C: StartFramedProtocolRequest LDAPv3 extended operation. The parameters are: 1) OID of LBURP incremental or full update Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 4 LDAP Bulk Update/Replication Protocol March, 2000 replication protocol as defined in this document. C -> S: StartFramedProtocolResponse LDAPv3 extended operation. The parameters are: 1) A suggested transactionSize (see section 6.2.1) S -> C: The supplier may send zero or more LBURPOperationRequest LDAPv3 extended operations. The requests MAY be sent asynchronously. The parameters are: 1) A sequence number which specifies the order of the operation with respect to other operations in the replication stream. 2) A list of one or more LDAP v3 update operations. C -> S: The consumer processes each of the LDAP v3 update operations contained within an LBURPOperationRequest. If it was able to successfully complete all of the update operations in the request, the consumer sends an LBURPOperationResponse with a result code of success. If one or more of the LDAP v3 update operations was not completed successfully, the consumer sends an LBURPOperationResponse with a non- success result code and includes a sequence of LDAPResult elements for each of the failed update operations which indicate the reason for failure. S -> C: After all required updates have been sent to the consumer, the supplier sends an EndFramedProtocolRequest LDAPv3 extended operation. The parameters are: 1) A sequence number which is one greater than the sequence number of the last operation in the replication stream. C -> S: The consumer responds by sending an EndFramedProtocolResponse LDAPv3 extended operation. 6. Elements of Protocol The LDAP Bulk Update/Replication protocol works within the framework of the Replication Update Protocol [LDUP RUP]. Unless otherwise Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 5 LDAP Bulk Update/Replication Protocol March, 2000 stated, those elements of protocol taken from the Replication Update Protocol framework are to be used precisely as described there. 6.1. StartFramedProtocolRequest Extended Operation Section 4.1 of [FRAMING] defines the StartFramedProtocolRequest extended operation in terms of the [LDAPv3] ExtendedRequest as follows: ExtendedRequest ::= [APPLICATION 23] SEQUENCE { requestName [0] LDAPOID, requestValue [1] OCTET STRING OPTIONAL } The requestName portion of the StartFramedProtocolRequest must be the OID "2.16.840.1.113719.1.142.100.1". The requestValue of the StartFramedProtocolRequest must be set to the BER-encoding of the following: StartFramedProtocolRequestValue ::= SEQUENCE { framedProtocolOID LDAPOID, framedProtocolPayload OPTIONAL OCTET STRING } 6.1.1. framedProtocolOID The framedProtocolOID is an OID that uniquely identifies the protocol framed by this operation. The framedProtocolOID for the LBURP Incremental Update Protocol is 2.16.840.1.113719.1.142.1.4.1. The framedProtocolOID for the LBURP Full Update Protocol is 2.16.840.1.113719.1.142.1.4.2. 6.1.2. framedProtocolPayload The framedProtocolPayload is an octet string that contains protocol- specific information. For LBURP, there is no framedProtocolPayload element for the StartFramedProtocolRequest extended operation. 6.2. StartFramedProtocolResponse Section 4.2 of [FRAMING] defines the StartFramedProtocolResponse extended operation in terms of the [LDAPv3] ExtendedResponse as follows: ExtendedResponse ::= [APPLICATION 24] SEQUENCE { COMPONENTS of LDAPResult, responseName [10] LDAPOID OPTIONAL, Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 6 LDAP Bulk Update/Replication Protocol March, 2000 response [11] OCTET STRING OPTIONAL } The responseName of the StartFramedProtocolResponse must be the OID "2.16.840.1.113719.1.142.100.2". For LBURP, the response of a StartFramedProtocolResponse must be set to the BER-encoding of the following: LBURPStartFramedProtocolResponse ::= SEQUENCE { transactionSize INTEGER, } 6.2.1 transactionSize The transactionSize is sent by the consumer to tell the supplier the number of update operations per UpdateOperationList (see section 6.3.2) that it would like the supplier to send. 6.3. LBURPOperationRequest The LBURPOperationRequest extended operation is used to send a set of one or more LDAP v3 update operations from the supplier to the consumer along with sequencing information that enables the consumer to maintain the proper sequencing of among multiple asynchronous requests. An LDAPv3 Extended Request is defined in [LDAPv3] as follows: ExtendedRequest ::= [APPLICATION 23] SEQUENCE { requestName [0] LDAPOID, requestValue [1] OCTET STRING OPTIONAL } The responseName of the LBURPOperationRequest must be the OID "2.16.840.1.113719.1.142.100.6". The requestValue of an LBURPOperationRequest extended operation must be set the BER-encoding of the following: LBURPOperationRequestValue ::= SEQUENCE { sequenceNumber INTEGER (1 .. maxInt), updateOperationList UpdateOperationList } 6.3.1. sequenceNumber The sequenceNumber is used to specify the ordering of LBURPOperationRequests. This enables the consumer to know the order in which LBURPOperationRequests must be processed even if it receives them in a sequence different from that in which they were Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 7 LDAP Bulk Update/Replication Protocol March, 2000 sent from the supplier. The supplier MUST set the value of sequenceNumber of the first LBURPOperationRequest to 1, and MUST increment the value of sequenceNumber for each succeeding LBURPOperationRequest. 6.3.2. UpdateOperationList The OpList is a list of one or more standard LDAP update requests and is defined as follows: UpdateOperationList ::= SEQUENCE of CHOICE { addRequest AddRequest, modifyRequest ModifyRequest, delRequest DelRequest, modDNRequest ModifyDNRequest, } AddRequest, ModifyRequest, DelRequest, and ModifyDNRequest are standard LDAP update requests as defined in sections 4.6, 4.7, 4.8, and 4.9 of [LDAPv3]. For the LBURP Incremental Update Protocol, all of the choices listed in OpList are valid and can be freely intermixed within an LBURP Incremental Update Protocol replication stream For the LBURP Full Update Protocol, the only valid choice is addRequest. Clients MUST not include any other choice in LBURPOperationRequests sent as part of an LBURP Full Update Protocol replication stream. 6.4. LBURPOperationResponse An LDAPv3 Extended Response is defined in [LDAPv3] as follows: ExtendedResponse ::= [APPLICATION 24] SEQUENCE { COMPONENTS of LDAPResult, responseName [10] LDAPOID OPTIONAL, response [11] OCTET STRING OPTIONAL } The responseName of the LBURPOperationResponse must be the OID "2.16.840.1.113719.1.142.100.7". The response of an LBURPOperationRequest extended operation must be set the BER-encoding of the following: LBURPOperationResponseValue ::= SEQUENCE of OperationResult OperationResult ::= SEQUENCE { Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 8 LDAP Bulk Update/Replication Protocol March, 2000 operationNumber INTEGER, ldapResult LDAPResult } If all of the update operations contained in the LBURPOperationRequest are successfully processed, the resultCode of the LBURPOperationResponse is set to success and no LBURPOperationResponseValue is included in the LBURPOperationResponse. If any of the update operations contained in the LBURPOperationRequest fails, an LBURPOperationResponseValue is included in the LBURPOperationResponse. For each update operation that fails, an OperationResult is included in the LBURPOperationResponseValue. 6.4.1. operationNumber The operationNumber identifies the operation that failed. Operations are numbered beginning at 1. 6.4.2 ldapResult The ldapResult included in the OperationResult is the same ldapResult that would be sent for the update operation that failed if it had failed while being processed as a normal LDAP v3 update operation. 6.5. EndFramedProtocolRequest Section 4.3 of [FRAMING] defines the EndFramedProtocolRequest extended operation in terms of the [LDAPv3] ExtendedRequest as follows: ExtendedRequest ::= [APPLICATION 23] SEQUENCE { requestName [0] LDAPOID, requestValue [1] OCTET STRING OPTIONAL } The requestName of the EndFramedProtocolRequest must be the OID "2.16.840.1.113719.1.142.100.4". For LBURP, the requestValue of the EndFramedProtocolRequest must be set to the BER-encoding of the following: LBURPEndFramedProtocolRequestValue::= SEQUENCE { sequenceNumber INTEGER (1 .. maxInt) } 6.5.1. sequenceNumber Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 9 LDAP Bulk Update/Replication Protocol March, 2000 The value in sequenceNumber is one greater than the last LBURPOperationRequest sequence number in the replication stream. It allows the server to know when it has received all outstanding asynchronous LBURPOperationRequests. 6.6. EndFramedProtcolResponse Section 4.4 of [FRAMING] defines the EndFramedProtocolResponse extended operation in terms of the [LDAPv3] ExtendedResponse as follows: ExtendedResponse ::= [APPLICATION 24] SEQUENCE { COMPONENTS of LDAPResult, responseName [10] LDAPOID OPTIONAL, response [11] OCTET STRING OPTIONAL } The responseName of the EndFramedProtocolResponse must be the OID "2.16.840.1.113719.1.142.100.5". For LBURP, there is no response element for the EndFramedProtocolResponse extended operation. 7. Semantics of Full and Incremental Update Protocols 7.1. Semantics Common to Both Full and Incremental Update Protocols Since both the full and incremental update protocols use sequences of standard LDAP v3 operations to transmit update information, no attempt is made by the protocol to include the information needed to support full multi-master replication as defined by [LDUP RUP]. Although entries and their associated attributes and attribute values can be synchronized using this protocol, no CSNs are included in the update stream; updates made using this replication protocol are treated the same as other LDAP operations wherein they are deemed to occur at the present. Operational attributes such as modifiersTimeStamp MAY be included in the replication stream, and if they are, they SHOULD be put into the entry. The LBURPOperationRequests that form the replication stream MAY be sent asynchronously by the supplier to the consumer. This means that the supplier need not wait for a LBURPOperationResponse from one LBURPOperationRequest before sending the next. The LBURPOperationRequests in the replication stream, plus the initial state of entries in the consumer replica in the case of incremental update, collectively represent the desired final state of the consumer replica. The consumer server may take any action required to efficiently achieve this desired final state of the replica. Examples of such actions include reordering LDAP requests to ensure that linear dependencies within the replication stream are Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 10 LDAP Bulk Update/Replication Protocol March, 2000 correctly ordered, breaking an LDAP modification request into two or more separate modification requests to adequately resolve a circular dependency within the replication stream, and aggregating LDAP update operations to process them more efficiently. When taking actions of this sort, the server MUST NOT reorder LDAP requests in such a way that the desired final state is different from that which would have been achieved before the action was taken. While a consumer server is processing a an LBURP update stream, the it MAY choose to not service LDAP requests for the replica involved in the replication operation other than those contained in the replication stream including LDAP requests on other connections. This requirement is designed to allow implementers the freedom to implement highly-efficient methods of handling the replication stream without being constrained by the need to maintain a live, working DIT database while doing so. 7.2. Semantics of the Full Replication Protocol Full replication creates a completely new set of entries in that replica; any existing entries in that replica at the start of the replication operation are removed before the LDAP requests in the replication stream are processed. The only LDAP operation allowed in a full update stream is add. Clients MUST NOT send any other type of update operation in the full update stream. If any other LDAP operation is received as part of the update stream, the server MUST return unwillingToPerform as the response for the operation. If the full update sequence fails for some reason (e.g. a connection is abruptly broken in the middle of an otherwise-successful update sequence) the server MUST remove any entries created during the update sequence, thereby leaving the replica in a known state. 7.3. Semantics of the Incremental Replication Protocol Incremental replication performs a series of incremental changes to the replica; existing entries in the replica are only affected if explicitly modified in some way by the update operations contained in the replication stream. All LDAP v3 update operations--add, modify, delete, and moddn--are allowed in the incremental replication stream. If a non-update operation is received as part of the replication stream, the server MUST return unwillingToPerform as the response for the operation. 8. Notes To Implementers Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 11 LDAP Bulk Update/Replication Protocol March, 2000 It is RECOMMENDED that, if possible, the consumer LDAP server refer non-replication requests for the replica being updated to another server containing that replica during the time that consumer LDAP server has the replica off-line for replication. Implementations MAY choose to perform the operations in the replication stream with special permissions to improve performance. 9. Security Considerations Implementations should ensure that a supplier making a Bulk Update/Replication request is bound with appropriate permissions. There is a potential for loss of data, especially with the full update protocol, which removes all entries in a replica as part of its operation. Second, forcing the removal of all entries in a replica may consume large amounts of server resources. Third, unlike other replication protocols, no existing replication agreement between supplier and consumer is required. These risks increase if the consumer server also processes the replication stream with special permissions to improve performance. For these reasons, implementers should carefully consider which identities or permissions are required to perform Bulk Update/Replication Protocol operations and take steps to ensure that only connections bound with appropriate permissions are allowed to perform them. The data contained in the replication stream may contain passwords and other sensitive data. Care should be taken to properly safeguard this information while in transit between supplier and consumer. 10. Document Revision History 10.1. draft-rharrison-lburp-00.txt Initial Draft 10.2. draft-rharrison-lburp-01.txt Adjusted LBURP protocol to use extended requests for all operations. LDAP update operations are now encapsulated within the LBURP Operation Request for two reasons: (1) To allow the inclusion of operation ordering information. This allows LDAP servers to maintain the proper ordering of updates even in cases where multi-threaded stacks present update operations to the server out-of-sequence. (2) To allow multiple update operations to be sent from client to server in a single request. This was a natural evolution of the changes made for (1) and allows the protocol to make more efficient use of network bandwidth, Converted references to LDUP extended operations to use a new LDAP Framed Operations Protocol. Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 12 LDAP Bulk Update/Replication Protocol March, 2000 Specified OIDs used for the protocol and extended operations. Changed requirement that a server "MUST NOT" service non-LBURP requests during a full update to a "MAY choose to not" service non- LBURP requests during a full update. This gives implementers the option to do what is needed without imposing a requirement that may not be needed by some implementations. 11. References [FRAMING] Stokes, Ellen, and Gordon Good, ôExtended Operations for Framing LDAP Operationsö, UNPUBLISHED INTERNET-DRAFT: ldup- framing.txt. [LDAPv3] Wahl, M., Howes, T., and S. Kille, "Lightweight Directory Access Protocol (v3)", RFC 2251, December 1997. [LDIF] Gordon Good, "LDAP Data Interchange Format (LDIF)", INTERNET- DRAFT: draft-good-ldap-ldif-04.txt, 22 June 1999. [ReqsKeywords] Scott Bradner. "Key Words for use in RFCs to Indicate Requirement Levels". RFC 2119. 12. Author's Addresses Roger Harrison Novell, Inc. 122 E. 1700 S. Provo, UT 84606 +1 801 861 2642 roger_harrison@novell.com Jim Sermersheim Novell, Inc. 122 E. 1700 S. Provo, UT 84606 +1 801 861 3088 jimse@novell.com Full Copyright Statement "Copyright (C) The Internet Society (date). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implmentation may be prepared, copied, published Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 13 LDAP Bulk Update/Replication Protocol March, 2000 and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Individual Submission - Expires September 10, 2000 Harrison & Sermersheim 14