Internet Draft R. Harrison Document: J. Sermersheim Intended Category: Proposed Standard Novell, Inc. June, 1999 LDAP v3 Extensions for Bulk Transfer of LDIF Data Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. 1. Abstract This document defines an LDAP extension to efficiently import LDIF into an LDAP server. The extension utilizes a protocol that maximizes the speed with which LDIF data can be presented to the LDAP server and yet allows the LDAP client to be kept apprised of errors and the progress of its request. 2. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [ReqsKeywords]. An LDIF stream is a data stream which conforms to the LDAP Data Interchange Format (LDIF) [LDIF] with the additional restriction that a LDIF record which defines an object MUST come before any LDIF records which define children of that object. This document describes the transmission of LDIF data from a client to a server and the server's behavior of accepting or consuming the LDIF data as well as processing that data. In talking about server behavior, the terms consume and consuming refer to the server accepting the LDIF stream. The terms process, and processing refer to the server applying the changes held in the LDIF records to the DIT. Harrison & Sermersheim Expires December, 1999 1 LDAP v3 Bulk LDIF Transmission June, 1999 3. Elements of Protocol The protocol of this extension is described using Abstract Syntax Notation 1 (ASN.1) [ASN1] and is typically transferred using a subset of ASN.1 Basic Encoding Rules [BER]. 4. The Bulk LDIF Transmission Extension Server support of the extension used for the bulk transmission of LDIF streams is advertised by the presence of its OID, "2.16.840.1.113719.1.1.102.1.0" in the supportedExtension attribute of a server's root DSE. 4.1 Bulk LDIF Transmission Request This extended request is sent in an LDAPMessage as described in 4.12 of [LDAPv3]. It is used by a client to initiate, transmit and complete the transfer of LDIF data to the server. It optionally allows specification of progress reports and which errors the server should halt on. 4.1.1 Bulk LDIF Transmission ASN.1 Definition The requestName is "2.16.840.1.113719.1.1.102.1.0". The requestValue is an OCTET STRING, whose value is the BER encoding of the following type: BulkLDIFTransmission ::= SEQUENCE { fileID [0] INTEGER DEFAULT 0, transactionSizeHint [1] INTEGER OPTIONAL, progressResponses [2] INTEGER DEFAULT 0, criticalErrors [3] CriticalErrorMask OPTIONAL, completedState [4] BOOLEAN DEFAULT TRUE, data [5] OCTET STRING } CriticalErrorMask ::= SEQUENCE { allExcept BOOLEAN DEFAULT TRUE, SET OF ENUMERATED { operationsError (1), sizeLimitExceeded (4), strongAuthRequired (8), referral (10), adminLimitExceeded (11), confidentialityRequired (13), noSuchAttribute (16), undefinedAttributeType (17), inappropriateMatching (18), Harrison & Sermersheim Expires December, 1999 2 LDAP v3 Bulk LDIF Transmission June, 1999 constraintViolation (19), attributeOrValueExists (20), invalidAttributeSyntax (21), noSuchObject (32), aliasProblem (33), invalidDNSyntax (34), inappropriateAuthentication (48), insufficientAccessRights (50), busy (51), unavailable (52), unwillingToPerform (53), loopDetect (54), namingViolation (64), objectClassViolation (65), notAllowedOnNonLeaf (66), notAllowedOnRDN (67), entryAlreadyExists (68), objectClassModsProhibited (69), affectsMultipleDSAs (71), other (80) } } 4.1.2 Bulk LDIF Transmission fileID description The fileID field is used to synchronize the transmission of LDIF data when more than one Bulk LDIF Transmission requests are used. When a client sends the first BulkLDIFTransmission request, this MUST be set to 0. If subsequent Bulk LDIF Transmission requests are sent to complete the transmission of the LDIF data, this value MUST contain the value of the fileID field returned by the server in the File ID Response. Servers use this value to distinguish between possible asynchronous Bulk LDIF Transmission requests from the same client. 4.1.3 Bulk LDIF Transmission transactionSizeHint description The transactionSizeHint field specifies the number of LDIF records the client would like applied to the LDAP server's DIT. This option is provided to allow the server to improve the performance of loading data. If no value is specified, the server is free to choose any value it wishes. If the server does not support transactions of the size requested by the client, it can ignore this value. 4.1.4 Bulk LDIF Transmission progressResponses description The progressResponses field specifies the frequency with which the server sends BulkLDIFProgress responses. These are asynchronous responses, which the client may consume in order to discover the number of records that the server has processed. If this is set to 0 (the default), no progress responses are sent by the server as it processes the LDIF data. The value specifies the number of records Harrison & Sermersheim Expires December, 1999 3 LDAP v3 Bulk LDIF Transmission June, 1999 to be processed between sending BulkLDIFProgress responses. If the server completes the processing of LDIF data before the non-zero number has been reached, it sends a final BulkLDIFProgress response. 4.1.5 BulkLDIFTransmission criticalErrors description The criticalErrors field allows the client to specify which errors will cause the server to stop processing and consuming LDIF data. For instance, a client may want the server to ignore entryAlreadyExists errors and continue to consume and process the LDIF data. If this field is not specified, all errors will be treated as critical. If the allExcept field is TRUE it means that all errors are critical except those listed. When a critical error occurs, the server MUST stop processing the LDIF stream and send a Bulk LDIF Error response with the critical error field set to TRUE. It MUST also refuse any subsequent Bulk LDIF Transmission requests that use the same fileID value. There is one error, malformedLdifData (91), which is always treated as critical. 4.1.6 Bulk LDIF Transmission completedState description The completedState is set to TRUE if there are to be no more Bulk LDIF Transmission requests sent for this LDIF stream. If a request is sent with this value set to FALSE, the server MUST send a Bulk LDIF FileID Response to the client with a unique value in the fileID field before processing any LDIF data. If a request is sent with this value set to TRUE, the server sends a Bulk LDIF Consumed Response to the client, it finishes processing the LDIF stream and then sends a Bulk LDIF Processed Response to the client. 4.1.7 Bulk LDIF Transmission data description The data field contains part or all of the LDIF data as it is contained in the LDIF file. Each octet represents one 8 bit character of the file. Clients may wish to transmit an LDIF stream in small portions to achieve a performance gain. Consider the scenario where a very large LDIF file is being processed. The client can send a manageable piece of data to the server. While the server is processing the first set of data, the client can asynchronously send remaining portions. If the data from the LDIF file is not specified in its entirety, the completedState MUST be set to FALSE. The data may be broken at any place in the LDIF stream. Server implementations MUST NOT rely on a client sending a complete LDIF record or even a complete line of data in a transmission. The Harrison & Sermersheim Expires December, 1999 4 LDAP v3 Bulk LDIF Transmission June, 1999 resulting concatenation of all transmissions of a set of LDIF data MUST conform to the definition of an LDIF stream in section 2. This allows a client to break up and transmit a large LDIF file without knowledge of the LDIF format. 4.2 Bulk LDIF response This extended response is sent in an LDAPMessage as described in 4.12 of [LDAPv3]. A server sends this response for several reasons which are enumerated below. 4.2.1 Bulk LDIF response ASN.1 Definition BulkLDIFResponse ::= { fileID INTEGER, CHOICE { bulkLdifFileID [0] NULL, bulkLdifProgress [1] BulkLdifProgress, bulkLdifError [2] BulkLdifError, bulkLdifConsumed [3] INTEGER, bulkLdifProcessed [4] INTEGER } } BulkLdifProgress ::= SEQUENCE { recordNumber [0] INTEGER, lastProcessedDN [1] LDAPDN } BulkLdifError ::= SEQUENCE { errorCode [0] BulkLdifErrors, criticalError [1] BOOLEAN DEFAULT FALSE, errorRecordNumber [2] INTEGER, errorDN [3] LDAPDN } BulkLdifErrors ::= ENUMERATED { operationsError (1), sizeLimitExceeded (4), strongAuthRequired (8), referral (10), adminLimitExceeded (11), confidentialityRequired (13), noSuchAttribute (16), undefinedAttributeType (17), inappropriateMatching (18), constraintViolation (19), attributeOrValueExists (20), invalidAttributeSyntax (21), noSuchObject (32), Harrison & Sermersheim Expires December, 1999 5 LDAP v3 Bulk LDIF Transmission June, 1999 aliasProblem (33), invalidDNSyntax (34), inappropriateAuthentication (48), insufficientAccessRights (50), busy (51), unavailable (52), unwillingToPerform (53), loopDetect (54), namingViolation (64), objectClassViolation (65), notAllowedOnNonLeaf (66), notAllowedOnRDN (67), entryAlreadyExists (68), objectClassModsProhibited (69), affectsMultipleDSAs (71), other (80), malformedLdifData (91) } The fileID field in all responses is a server-generated value that it uses to uniquely identify this bulk LDIF stream. 4.2.1.1 Bulk LDIF File ID response This response is sent by the server after the first Bulk LDIF Transmission request is received. The client must use the file ID received in this response in all subsequent bulk transmission requests for this LDIF stream. 4.2.1.2 Bulk LDIF Progress response This response is sent periodically by the server as requested by the client in the first Bulk LDIF Transmission request. When this response is sent by the server, the LDAPResult field MUST contain a value of success (0). The recordNumber contains a count of the records processed by the server, and the lastProcessedDN contains the DN of the last record processed by the server. The Bulk LDIF Progress response does not contain error notifications. Error notifications are sent via the Bulk LDIF Error response. 4.2.1.3 Bulk LDIF Error response This response is sent by the server whenever an error in consuming or processing an LDIF record is encountered. When this response is sent, the LDAPResult field MUST contain a value of operationsError (1). Harrison & Sermersheim Expires December, 1999 6 LDAP v3 Bulk LDIF Transmission June, 1999 The errorCode should contain the actual error value generated by the server in attempting to consume or process the LDIF record. If the criticalError flag is FALSE, it indicates that this error was ignored by the server and the server continued processing the LDIF stream at the next LDIF record. If the criticalError flag is TRUE, it indicates that the server has stopped processing LDIF records, any consumed but unprocessed LDIF data has been discarded, and the client MUST NOT attempt to send further LDIF data for this LDIF stream. The errorRecordNumber is the record number in the LDIF stream in which the error occurred, and the errorDN is the DN contained in that record. In the case of data corruption, the DN may be empty. 4.3.1.4 Bulk LDIF Consumed response This response is sent by the server to acknowledge receipt of all LDIF records within an entire stream. When this response is sent, the LDAPResult field MUST contain a value of success (0). The bulkLdifConsumed field indicates the total number of records counted while consuming the LDIF stream. 4.3.1.5 Bulk LDIF Processed response This response is sent by the server to acknowledge that all LDIF records within an entire stream have been applied to the server's DIT. When this response is sent, the LDAPResult field MUST contain a value of success (0). The bulkLdifProcessed field indicates the total number of records applied to the DIT from the LDIF stream. 5. Security Considerations The LDIF data format may be used to textually represent all information in a directory including sensitive information such as passwords, credit card numbers, etc. This protocol makes no provision for security of the LDIF stream transmitted to an LDAP server. Clients SHOULD take appropriate measures to ensure that sensitive LDIF data is not transmitted in cleartext form by using encrypted connections or other mechanisms. 6. References [LDIF] Gordon Good, "LDAP Data Interchange Format (LDIF)", INTERNET- DRAFT: draft-good-ldap-ldif-04.txt, 22 June 1999. [ASN1] ITU-T Rec. X.680, "Abstract Syntax Notation One (ASN.1) - Specification of Basic Notation", 1994. [BER] ITU-T Rec. X.690, "Specification of ASN.1 encoding rules: Basic, Canonical, and Distinguished Encoding Rules", 1994. Harrison & Sermersheim Expires December, 1999 7 LDAP v3 Bulk LDIF Transmission June, 1999 [LDAPv3] Wahl, M., Howes, T., and S. Kille, "Lightweight Directory Access Protocol (v3)", RFC 2251, December 1997. [ReqsKeywords] Scott Bradner. "Key Words for use in RFCs to Indicate Requirement Levels". RFC 2119. 7. Author's Addresses Roger Harrison Novell, Inc. 122 E. 1700 S. Provo, UT 84606 +1 801 861 2642 roger_harrison@novell.com Jim Sermersheim Novell, Inc. 122 E. 1700 S. Provo, UT 84606 +1 801 861 3088 jimse@novell.com Full Copyright Statement "Copyright (C) The Internet Society (date). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implmentation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into Harrison & Sermersheim Expires December, 1999 8