Internet DRAFT - draft-black-pnfs-block

draft-black-pnfs-block



NFSv4 Working Group                                      David L. Black 
Internet Draft                                         Stephen Fridella 
Expires: April 2006                                     EMC Corporation 
                                                       October 21, 2005 
                                    
 
                                      
                         pNFS Block/Volume Layout 
                       draft-black-pnfs-block-02.txt 


Status of this Memo 

   By submitting this Internet-Draft, each author represents that       
   any applicable patent or other IPR claims of which he or she is       
   aware have been or will be disclosed, and any of which he or she       
   becomes aware will be disclosed, in accordance with Section 6 of       
   BCP 79. 

   Internet-Drafts are working documents of the Internet Engineering 
   Task Force (IETF), its areas, and its working groups.  Note that 
   other groups may also distribute working documents as Internet-
   Drafts. 

   Internet-Drafts are draft documents valid for a maximum of six months 
   and may be updated, replaced, or obsoleted by other documents at any 
   time.  It is inappropriate to use Internet-Drafts as reference 
   material or to cite them other than as "work in progress." 

   The list of current Internet-Drafts can be accessed at 
        http://www.ietf.org/ietf/1id-abstracts.txt 

   The list of Internet-Draft Shadow Directories can be accessed at 
        http://www.ietf.org/shadow.html 

   This Internet-Draft will expire in April 2006. 

Abstract 

   Parallel NFS (pNFS) extends NFSv4 to allow clients to directly access 
   file data on the storage used by the NFSv4 server.  This ability to 
   bypass the server for data access can increase both performance and 
   parallelism, but requires additional client functionality for data 
   access, some of which is dependent on the class of storage used.  The 
   main pNFS operations draft specifies storage-class-independent 
   extensions to NFS; this draft specifies the additional extensions 
   (primarily data structures) for use of pNFS with block and volume 
   based storage. 
 
 
 
Black                     Expires April 2006                   [Page 1] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

Conventions used in this document 

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 
   document are to be interpreted as described in RFC-2119 [RFC2119]. 

Table of Contents 

    
   1. Introduction...................................................3 
   2. Background and Architecture....................................3 
      2.1. Data Structures: Extents and Extent Lists.................4 
         2.1.1. Layout Requests and Extent Lists.....................6 
         2.1.2. Client Copy-on-Write Processing......................7 
         2.1.3. Extents are Permissions..............................8 
      2.2. Volume Identification....................................10 
   3. Operations Issues.............................................11 
      3.1. Layout Operation Ordering Considerations.................12 
         3.1.1. Client Side Considerations..........................12 
         3.1.2. Server Side Considerations..........................13 
      3.2. Recall Callback Completion and Robustness Concerns.......14 
      3.3. Crash Recovery Issues....................................15 
      3.4. Additional Features - Not Needed or Recommended..........16 
   4. Security Considerations.......................................17 
   5. Conclusions...................................................17 
   6. Revision History..............................................18 
   7. Acknowledgments...............................................18 
   8. References....................................................18 
      8.1. Normative References.....................................18 
      8.2. Informative References...................................18 
   Author's Addresses...............................................19 
   Intellectual Property Statement..................................19 
   Disclaimer of Validity...........................................19 
   Copyright Statement..............................................20 
   Acknowledgment...................................................20 
    
   NOTE: This is an early stage draft.  It's still rough in places, with 
   significant work to be done. 









 
 
Black                     Expires April 2006                   [Page 2] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

    
1. Introduction 

   Figure 1 shows the overall architecture of a pNFS system: 

       +-----------+                                
       |+-----------+                                 +-----------+ 
       ||+-----------+                                |           | 
       |||           |        NFSv4 + pNFS            |           | 
       +||  Clients  |<------------------------------>|   Server  | 
        +|           |                                |           | 
         +-----------+                                |           | 
              |||                                     +-----------+ 
              |||                                           | 
              |||                                           | 
              |||                +-----------+              | 
              |||                |+-----------+             | 
              ||+----------------||+-----------+            | 
              |+-----------------|||           |            | 
              +------------------+||  Storage  |------------+ 
                                  +|  Systems  | 
                                   +-----------+ 
    
                        Figure 1 pNFS Architecture 

   The overall approach is that pNFS-enhanced clients obtain sufficient 
   information from the server to enable them to access the underlying 
   storage (on the Storage Systems) directly.  See [PNFS] for more 
   details.  This draft is concerned with access from pNFS clients to 
   Storage Systems over storage protocols based on blocks and volumes, 
   such as the SCSI protocol family (e.g., parallel SCSI, FCP for Fibre 
   Channel, iSCSI, SAS).  This class of storage is referred to as 
   block/volume storage.  While the Server to Storage System protocol is 
   not of concern for interoperability here, it will typically also be a 
   block/volume protocol when clients use block/volume protocols. 

2. Background and Architecture 

   The fundamental storage abstraction supported by block/volume storage 
   is a storage volume consisting of a sequential series of fixed size 
   blocks.  This can be thought of as a logical disk; it may be realized 
   by the Storage System as a physical disk, a portion of a physical 
   disk or something more complex (e.g., concatenation, striping, RAID, 
   and combinations thereof) involving multiple physical disks or 
   portions thereof. 


 
 
Black                     Expires April 2006                   [Page 3] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   A pNFS layout for this block/volume class of storage is responsible 
   for mapping from an NFS file (or portion of a file) to the blocks of 
   storage volumes that contain the file.  The blocks are expressed as 
   extents with 64 bit offsets and lengths using the existing NFSv4 
   offset4 and length4 types.  Clients must be able to perform I/O to 
   the block extents without affecting additional areas of storage 
   (especially important for writes), therefore extents MUST be aligned 
   to 512-byte boundaries, and SHOULD be aligned to the block size used 
   by the NFSv4 server in managing the actual filesystem (4 kilobytes 
   and 8 kilobytes are common block sizes).  This block size is 
   available as an NFSv4 attribute - see Section 7.4 of [PNFS]. 

   This draft relies on the pNFS client indicating whether a requested 
   layout is for read use or read-write use.  A read-only layout may 
   contain holes that are read as zero, whereas a read-write layout will 
   contain allocated, but uninitialized storage in those holes (read as 
   zero, can be written by client).  This draft also supports client 
   participation in copy on write by providing both read-only and 
   uninitialized storage for the same range in a layout.  Reads are 
   initially performed on the read-only storage, with writes going to 
   the uninitialized storage.  After the first write that initializes 
   the uninitialized storage, all reads are performed to that now-
   initialized writeable storage, and the corresponding read-only 
   storage is no longer used. 

   This draft draws extensively on the authors' familiarity with the the 
   mapping functionality and protocol in EMC's HighRoad system.  The 
   protocol used by HighRoad is called FMP (File Mapping Protocol); it 
   is an add-on protocol that runs in parallel with filesystem protocols 
   such as NFSv3 to provide pNFS-like functionality for block/volume 
   storage.  While drawing on HighRoad FMP, the data structures and 
   functional considerations in this draft differ in significant ways, 
   based on lessons learned and the opportunity to take advantage of 
   NFSv4 features such as COMPOUND operations.  The support for client 
   participation in copy-on-write is based on contributions from those 
   with experience in that area, as HighRoad does not currently support 
   client participation in copy-on-write. 

2.1. Data Structures: Extents and Extent Lists 

   A pNFS layout is a list of extents with associated properties. Each 
   extent MUST be at least 512-byte aligned. 

    

    

 
 
Black                     Expires April 2006                   [Page 4] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   struct extent { 

     offset4      file_offset;/* the logical location in the file */ 

     length4      extent_length; /* the size of this extent in file and 
                                    and on storage */ 

     pnfs_deviceid4  volume_ID;  /* the logical volume/physical device 
                                    that this extent is on */ 

     offset4      storage_offset;/* the logical location of 
                                 this extent in the volume */ 

     extentState4    es;   /* the state of this extent */ 

   }; 

   enum extentState4 { 

     READ_WRITE_DATA  = 0, /* the data located by this extent is valid 
                              for reading and writing. */ 

     READ_DATA = 1, /* the data located by this extent is valid for   
                        reading only; it may not be written. */ 

     INVALID_DATA = 2,  /* the location is valid; the data is invalid. 
                           It is a newly (pre-) allocated extent. 
                           There is physical space. */ 

     NONE_DATA = 3,  /* the location is invalid. It is a hole in the  
                        file. There is no physical space. */ 

     }; 

   The file_offset, extent_length, and es fields for an extent returned 
   from the server are always valid. The interpretation of the 
   storage_offset field depends on the value of es as follows:  

   o  READ_WRITE_DATA means that storage_offset is valid, and points to 
      valid/initialized data that can be read and written. 

   o  READ_DATA means that storage_offset is valid and points to valid/ 
      initialized data which can only be read.  Write operations are 
      prohibited; the client may need to request a read-write layout. 



 
 
Black                     Expires April 2006                   [Page 5] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   o  INVALID_DATA means that storage_offset is valid, but points to 
      invalid uninitialized data. This data must not be physically read 
      from the disk until it has been initialized.  A read request for 
      an INVALID_DATA extent must fill the user buffer with zeros. Write 
      requests must write whole blocks to the disk with bytes not 
      initialized by the user must be set to zero.  Any write to storage 
      in an INVALID_DATA extent changes the written portion of the 
      extent to READ_WRITE_DATA; the pNFS client is responsible for 
      reporting this change via LAYOUTCOMMIT. 

   o  NONE_DATA means that storage_offset is not valid, and this extent 
      may not be used to satisfy write requests. Read requests may be 
      satisfied by zero-filling as for INVALID_DATA. NONE_DATA extents 
      are returned by requests for readable extents; they are never 
      returned if the request was for a writeable extent. 

   The volume_ID field for an extent returned by the server is used to 
   identify the logical volume on which this extent resides, see Section 
   2.2. 

   The extent list lists all relevant extents in increasing order of the 
   file_offset of each extent; any ties are broken by increasing order 
   of the extent state (es). 

   typedef extent extentList<MAX_EXTENTS>;  /* MAX_EXTENTS = 256; */ 

   TODO: Define the actual layout and layoutupdate data structures as 
   extent lists. 

   TODO: Striping support.  Layout independent striping will not be 
   added to [PNFS], but it can help compact layout representations when 
   the filesystem is striped across block/volume storage. 

2.1.1. Layout Requests and Extent Lists 

   Each request for a layout specifies at least three parameters: 
   offset, desired size, and minimum size (the desired size is missing 
   from the operations draft - see Section 3).  If the status of a 
   request indicates success, the extent list returned must meet the 
   following criteria:  







 
 
Black                     Expires April 2006                   [Page 6] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   o  A request for a readable (but not writeable layout returns only 
      READ_WRITE_DATA, READ_DATA or NONE_DATA extents (but not 
      INVALID_DATA extents).  A READ_WRITE_DATA extent MAY be returned 
      by a pNFS server in a readable layout in order to avoid a 
      subsequent client request for writing (ISSUE: Is that a good idea?  
      It involves server second-guessing client, and the downside is the 
      possible need for a recall callback). 

   o  A request for a writeable layout returns READ_WRITE_DATA or 
      INVALID_DATA extents (but not NONE_DATA extents).  It may also 
      return READ_DATA extents only when the offset ranges in those 
      extents are also covered by INVALID_DATA extents to permit writes. 

   o  The first extent in the list MUST contain the starting offset. 

   o  The total size of extents in the extent list MUST cover at least 
      the minimum size and no more than the desired size.  One exception 
      is allowed: the total size MAY be smaller if only readable extents 
      were requested and EOF is encountered. 

   o  Extents in the extent list MUST be logically contiguous for a 
      read-only layout.  For a read-write layout, the set of writable 
      extents (i.e., excluding READ_DATA extents) MUST be logically 
      contiguous.  Every READ_DATA extent in a read-write layout MUST be 
      covered by an INVALID_DATA extent.  This overlap of READ_DATA and 
      INVALID_DATA extents is the only permitted extent overlap. 

   o  Extents MUST be ordered in the list by starting offset, with 
      READ_DATA extents preceding INVALID_DATA extents in the case of 
      equal file_offsets. 

2.1.2. Client Copy-on-Write Processing 

   Distinguishing the READ_WRITE_DATA and READ_DATA extent types 
   combined with the allowed overlap of READ_DATA extents with 
   INVALID_DATA extents allows copy-on-write processing to be done by 
   pNFS clients. In classic NFS, this operation would be done by the 
   server.  Since pNFS enables clients to do direct block access, it 
   requires clients to participate in copy-on-write operations. 

   When a client wishes to write data covered by a READ_DATA extent, it 
   MUST have requested a writable layout from the server; that layout 
   will contain INVALID_DATA extents to cover all the data ranges of 
   that layout's READ_DATA extents. More precisely, for any file_offset 
   range covered by one or more READ_DATA extents in a writable layout, 
   the server MUST include one or more INVALID_DATA extents in the 
   layout that cover the same file_offset range. The client MUST 
 
 
Black                     Expires April 2006                   [Page 7] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   logically copy the data from the READ_DATA extent for any partial 
   blocks of file_offset and range, merge in the changes to be written, 
   and write the result to the INVALID_DATA extent for the blocks for 
   that file_offset and range. That is, if entire blocks of data are to 
   be overwritten by an operation, the corresponding READ_DATA blocks 
   need not be fetched, but any partial-block writes must be merged with 
   data fetched via READ_DATA extents before storing the result via 
   INVALID_DATA extents.  Storing of data in an INVALID_DATA extent 
   converts the written portion of the INVALID_DATA extent to a 
   READ_WRITE_DATA extent; all subsequent reads MUST be performed from 
   this extent; the corresponding portion of the READ_DATA extent MUST 
   NOT be used after storing data in an INVALID_DATA extent. 

   In the LAYOUTCOMMIT operation that normally sends updated layout 
   information back to the server, for writable data, some INVALID_DATA 
   extents may be committed as READ_WRITE_DATA extents, signifying that 
   the storage at the corresponding storage_offset values has been 
   stored into and is now to be considered as valid data to be read. 
   READ_DATA extents need not be sent to the server. For extents that 
   the client receives via LAYOUTGET as INVALID_DATA and returns via 
   LAYOUTCOMMIT as READ_WRITE_DATA, the server will understand that the 
   READ_DATA mapping for that extent is no longer valid or necessary for 
   that file. 

   ISSUE: This assumes that all block/volume pNFS clients will support 
   copy-on-write.  Negotiating this would require additional server code 
   to cope with clients that don't support this, which doesn't seem like 
   a good idea. 

2.1.3. Extents are Permissions 

   Layout extents returned to pNFS clients grant permission to read or 
   write; READ_DATA and NONE_DATA are read-only (NONE_DATA reads as 
   zeroes), READ_WRITE_DATA and INVALID_DATA are read/write, 
   (INVALID_DATA reads as zeros, any write converts it to 
   READ_WRITE_DATA).  This is the only client means of obtaining 
   permission to perform direct I/O to storage devices; a pNFS client 
   MUST NOT perform direct I/O operations that are not permitted by an 
   extent held by the client.  Client adherence to this rule places the 
   pNFS server in control of potentially conflicting storage device 
   operations, enabling the server to determine what does conflict and 
   how to avoid conflicts by granting and recalling extents to/from 
   clients.   

   Block/volume class storage devices are not required to perform read 
   and write operations atomically.  Overlapping concurrent read and 
   write operations to the same data may cause the read to return a 
 
 
Black                     Expires April 2006                   [Page 8] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   mixture of before-write and after-write data.  Overlapping write 
   operations can be worse, as the result could be a mixture of data 
   from the two write operations; this can be particularly nasty if the 
   underlying storage is striped and the operations complete in 
   different orders on different stripes.  A pNFS server can avoid these 
   conflicts by implementing a single writer XOR multiple readers 
   concurrency control policy when there are multiple clients who wish 
   to access the same data.  This policy SHOULD be implemented when 
   storage devices do not provide atomicity for concurrent read/write 
   and write/write operations to the same data. 

   A client that makes a layout request that conflicts with an existing 
   layout delegation will be rejected with the error 
   NFS4ERR_LAYOUTTRYLATER.  This client is then expected to retry the 
   request after a short interval.  During this interval the server 
   needs to recall the conflicting portion of the layout delegation from 
   the client that currently holds it.  This reject-and-retry approach 
   does not prevent client starvation when there is contention for the 
   layout of a particular file.  For this reason a pNFS server SHOULD 
   implement a mechanism to prevent starvation.  One possibility is that 
   the server can maintain a queue of rejected layout requests.  Each 
   new layout request can be checked to see if it conflicts with a 
   previous rejected request, and if so, the newer request can be 
   rejected. Once the original requesting client retries its request, 
   its entry in the rejected request queue can be cleared, or the entry 
   in the rejected request queue can be removed when it reaches a 
   certain age. 

   NFSv4 supports mandatory locks and share reservations.  These are 
   mechanisms that clients can use to restrict the set of I/O operations 
   that are permissible to other clients.  Since all I/O operations 
   ultimately arrive at the NFSv4 server for processing, the server is 
   in a position to enforce these restrictions.  However, with pNFS 
   layout delegations, I/Os will be issued from the clients that hold 
   the delegations directly to the storage devices that host the data.  
   These devices have no knowledge of files, mandatory locks, or share 
   reservations, and are not in a position to enforce such restrictions.  
   For this reason the NFSv4 server must not grant layout delegations 
   that conflict with mandatory locks or share reservations.  Further, 
   if a conflicting mandatory lock request or a conflicting open request 
   arrives at the server, the server must recall the part of the layout 
   delegation in conflict with the request before processing the 
   request.   




 
 
Black                     Expires April 2006                   [Page 9] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

2.2. Volume Identification 

   Storage Systems such as storage arrays can have multiple physical 
   network ports that need not be connected to a common network, 
   resulting in a pNFS client having simultaneous multipath access to 
   the same storage volumes via different ports on different networks.  
   The networks may not even be the same technology - for example, 
   access to the same volume via both iSCSI and Fibre Channel is 
   possible, hence network address are difficult to use for volume 
   identification.  For this reason, this pNFS block layout identifies 
   storage volumes by content, for example providing the means to match 
   (unique portions of) labels used by volume managers.  Any block pNFS 
   system using this layout MUST support a means of content-based unique 
   volume identification that can be employed via the data structure 
   given here. 

   A volume is content-identified by a disk signature made up of extents 
   within blocks and contents that must match. 

   block_device_addr_list - A list of the disk signatures for the 
   physical volumes on which the file system resides. This is list of 
   variable number of diskSigInfo structures.  This is the 
   device_addr_list<> as returned by GETDEVICELIST in [PNFS] 

   typedef diskSigInfo block_device_addr_list<MAX_DEVICE>; 
            /* disksignature info */ 

   where diskSigInfo is: 

   struct diskSigInfo {       /* used in DISK_SIGNATURE */ 
      diskSig        ds;      /* disk signature */ 

      pnfs_deviceid4 volume_ID;  /* volume ID the server will use in 
                                    extents. */ 

   }; 

   where diskSig is defined as: 

   typedef sigComp diskSig<MAX_SIG_COMPONENTS>; 

   struct sigComp {        /*  disk signature component */ 

      offset4  sig_offset; /* byte offset of component */ 
    
      length4  sig_length; /* byte length of component */ 

 
 
Black                     Expires April 2006                  [Page 10] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

      sigCompContents contents;  /* contents of this component of the 
                                    signature (this is opaque) */ 

   }; 

   sigCompContents MUST NOT be interpreted as a zero-terminated string, 
   as it may contain embedded zero-valued octets.  It contains 
   sig_length octets.  There are no restrictions on alignment (e.g., 
   neither sig_offset nor sig_length are required to be multiples of 4). 

3. Operations Issues 

   NOTE: This section and its subsections are preserved for 
   historical/review purposes only, as the [PNFS] draft has addressed 
   all of these issues.  The section and all subsections will be deleted 
   in the next version of this draft. 

   This section collects issues in the operations draft encountered in 
   writing this block/volume layout draft.  Most of these issues are 
   expected to be resolved in draft-welch-pnfs-ops-03.txt .  

   1. RESOLVED: LAYOUTGET provides minimum and desired (max) lengths to 
      server. 

   2. RESOLVED: Layouts are managed by offset and range; they are no 
      longer treated as indivisible objects. 

   3. RESOLVED: There is a callback for the server to convey a new EOF 
      to the client. 

   4. RESOLVED: HighRoad supports three types of layout recalls beyond 
      range recalls: "everything in a file", "everything in a list of 
      files", "everything in a filesystem".  The first and third are 
      supported in [PNFS] (set offset to zero and length to all 1's for 
      everything in a file - [PNFS] implies this, but isn't explicit 
      about it), and the second one can probably be done as a COMPOUND 
      with reasonable effectiveness.  LAYOUTRETURN supports return of 
      everything in a file in a similar fashion (offset of zero, length 
      of all 1's). 

   5. RESOLVED: Access and Modify time behavior.  LAYOUTCOMMIT operation 
      sets both Access and Modify times.   LAYOUTRETURN cannot set 
      either time - use a SETATTR in a COMPOUND to do this (Q: Can this 
      inadvertently make time run backwards?). 



 
 
Black                     Expires April 2006                  [Page 11] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   6. RESOLVED: The disk signature approach to volume identification 
      appears to be supportable via the opaque pnfs_devaddr4 union 
      element. 

   7. RESOLVED: The LAYOUTCOMMIT operation has no LAYOUTRETURN side 
      effects in -03.  If it ever did, they were not intended. 

3.1. Layout Operation Ordering Considerations 

   This deserves its own subsection because there is some serious 
   subtlety here. 

   In contrast to NFSv4 callbacks that expect immediate responses, 
   HighRoad layout callback responses are delayed to allow the client to 
   perform any required commits, etc. prior to responding to the 
   callback.  This allows the reply to the callback to serve as an 
   implicit return of the recalled range or ranges and tell the server 
   that all callback related processing has been completed by the 
   client.  For consistency, pNFS should use the NFSv4 callback approach 
   in which immediate responses are expected.  As a result all returns 
   of layout ranges MUST be explicit. 

3.1.1. Client Side Considerations 

   Consider a pNFS client that has issued a LAYOUTGET and then receives 
   an overlapping recall callback for the same file.  There are two 
   possibilities, which the client cannot distinguish when the callback 
   arrives: 

   1. The server processed the LAYOUTGET before issuing the recall, so 
      the LAYOUTGET response is in flight, and must be waited for 
      because it may be carrying layout info that will need to be 
      returned to deal with the recall callback. 

   2. The server issued the callback before receiving the LAYOUTGET. The 
      server will not respond to the LAYOUTGET until the recall callback 
      is processed. 

   This can cause deadlock, as the client must wait for the LAYOUTGET 
   response before processing the recall in the first case, but that 
   response will not arrive until after the recall is processed in the 
   second case.  The deadlock is avoided via a simple rule: 

      RULE: A LAYOUTGET MUST be rejected with an error if there's an 
      overlapping outstanding recall callback to the same client.  The 
      client MUST process the outstanding recall callback before 
      retrying the LAYOUTGET. 
 
 
Black                     Expires April 2006                  [Page 12] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   Now the client can wait for the LAYOUTGET response because it will 
   come in both cases.  This RULE also applies to the callback to send 
   an updated EOF to the client.   

   The resulting situation is still less than desired, because issuance 
   of a recall callback indicates a conflict and potential contention at 
   the server, so recall callbacks should be processed as fast as 
   possible by clients.  In the second case, if the client knows that 
   the LAYOUTGET will be rejected, it is beneficial for the client to 
   process the recall immediately without waiting for the LAYOUTGET 
   rejection.  To do so without added client complexity, the server 
   needs to reject the LAYOUTGET even if it arrives at the server after 
   the client operations that process the recall callback; if the client 
   still wants that layout, it can reissue the LAYOUTGET. 

   HighRoad uses the equivalent of a per-file layout stateid to enable 
   this optimization.  The layout stateid increments on each layout 
   operation completion and callback issuance, and the current value of 
   the layout stateid is sent in every operation response and every 
   callback.  If the initial layout stateid value is N, then in the 
   first case above, the recall callback carries stateid N+2 indicating 
   that the LAYOUTGET response is carrying N+1 and hence has to be 
   waited for.  In the second case above, the recall callback carries 
   layout stateid N+1 indicating that the LAYOUTGET will be rejected 
   with a stale layout stateid (N where N+1 or greater is current) 
   whenever it arrives, and hence the callback can be processed 
   immediately.  This per-file layout stateid approach entails 
   prohibiting concurrent callbacks to the for the same file to the same 
   client, as server issuance of a new callback could cause stale layout 
   stateid errors for operations that the client is performing to deal 
   with an earlier recall callback. 

   ISSUE: Does restricting all pNFS client operations on the same file 
   to a single session help? 

3.1.2. Server Side Considerations 

   Consider a related situation from the pNFS server's point of view.  
   The server has issued a recall callback and receives an overlapping 
   LAYOUTGET for the same file before the LAYOUTRETURN(s) that respond 
   to the recall callback.  Again, there are two cases: 

   1. The client issued the LAYOUTGET before processing the recall 
      callback.  The LAYOUTGET MUST be rejected according to the RULE in 
      the previous subsection. 


 
 
Black                     Expires April 2006                  [Page 13] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   2. The client issued the LAYOUTGET after processing the recall 
      callback, but it arrived before the LAYOUTRETURN that completed 
      that processing. 

   The simplest approach is to apply the RULE and always reject the 
   overlapping LAYOUTGET.  The client has two ways to avoid this result 
   - it can issue the LAYOUTGET as a subsequent element of a COMPOUND 
   containing the LAYOUTRETURN that completes the recall callback, or it 
   can wait for the response to that LAYOUTRETURN. 

   This leads to a more general problem; in the absence of a callback if 
   a client issues concurrent overlapping LAYOUTGET and LAYOUTRETURN 
   operations, it is possible for the server to process them in either 
   order.  HighRoad forbids a client from doing this, as the per-file 
   layout stateid will cause one of the two operations to be rejected 
   with a stale layout stateid.  This approach is simpler and produces 
   better results by comparison to allowing concurrent operations, at 
   least for this sort of conflict case, because server execution of 
   operations in an order not anticipated by the client may produce 
   results that are not useful to the client (e.g., if a LAYOUTRETURN is 
   followed by a concurrent overlapping LAYOUTGET, but executed in the 
   other order, the client will not retain layout extents for the 
   overlapping range). 

3.2. Recall Callback Completion and Robustness Concerns 

   The discussion of layout operation ordering implicitly assumed that 
   any callback results in a LAYOUTRETURN or set of LAYOUTRETURNs that 
   match the range in the callback.  This envisions that the pNFS client 
   state for a file match the pNFS server state for that file and client 
   regarding layout ranges and permissions.  That may not be the best 
   design assumption because: 

   1. It may be useful for clients to be able to discard layout 
      information without calling LAYOUTRETURN.  If conflicts that 
      require callbacks are rare, and a server can use a multi-file 
      callback to recover per-client resources (e.g., via a multi-file 
      recall operation based on some sort of LRU), the result may be 
      significantly less client-server pNFS traffic. 

   2. It may be similarly useful for servers to enhance information 
      about what layout ranges are held by a client beyond what a client 
      actually holds.  In the extreme, a server could manage conflicts 
      on a per-file basis, only issuing whole-file callbacks even though 
      clients may request and be granted sub-file ranges. 


 
 
Black                     Expires April 2006                  [Page 14] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   3. The synchronized state assumption is not robust to minor errors.  
      A more robust design would allow for divergence between client and 
      server and the ability to recover.  It is vital that a client not 
      assign itself layout permissions beyond what the server has 
      granted and that the server not forget layout permissions that 
      have been granted in order to avoid errors.  OTOH, if a server 
      believes that a client holds an extent that the client doesn't 
      know about, it's useful for the client to be able to issue the 
      LAYOUTRETURN that the server is expecting in response to a recall. 

   At a minimum, in light of the above, it is useful for a server to be 
   able to issue callbacks for layout ranges it has not granted to a 
   client, and for a client to return ranges it does not hold.  This 
   leads to a couple of requirements: 

      A pNFS client's final operation in processing a recall callback 
      SHOULD be a LAYOUTRETURN whose range matches that in the callback. 
      If the pNFS client holds no layout permissions in the range that 
      has been recalled, it MUST respond with a LAYOUTRETURN whose range 
      matches that in the callback. 

   This avoids any need for callback cookies (server to client) that 
   would have to be returned to indicate recall callback completion. 

   For a callback to set EOF, the client MUST logically apply the new 
   EOF before issuing the response to the callback, and MUST NOT issue 
   any other pNFS operations before responding to the callback. 

   ISSUE: HighRoad FMP also requires that LAYOUTCOMMIT operations be 
   stalled at the server while an EOF callback is outstanding. 

3.3. Crash Recovery Issues 

   Client recovery for layout delegations works in much the same way as 
   NFSv4 client recovery for other lock/delegation state.  When an NFSv4 
   client reboots, it will lose all information about the layout 
   delegations that it previously owned.  There are two methods by which 
   the server can reclaim these resources and begin providing them to 
   other clients.  The first is through the expiry of the client's 
   lock/delegation lease.  If the client recovery time is longer than 
   the lease period, the client's lock/delegation lease will expire and 
   the server will know to reclaim any state held by the client.  On the 
   other hand, the client may recover in less time than it takes for the 
   lease period to expire.  In such a case, the client will be required 
   to contact the server through the standard SETCLIENTID protocol.  The 
   server will find that the client's id matches the id of the previous 
   client invocation, but that the verifier is different.  The server 
 
 
Black                     Expires April 2006                  [Page 15] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   uses this as a signal to reclaim all the state associated with the 
   client's previous invocation. 

   The server recovery case is slightly more complex.  In general, the 
   recovery process will again follow the standard NFSv4 recovery model:  
   the client will discover that the server has rebooted when it 
   receives an unexpected STALE_STATEID or STALE_CLIENTID reply from the 
   server; it will then proceed to try to reclaim its previous 
   delegations during the server's recovery grace period.  However there 
   is an important safety concern associated with layout delegations 
   that does not come into play in the standard NFSv4 case.  If a 
   standard NFSv4 client makes use of a stale delegation, the 
   consequence could be to deliver stale data to an application.  
   However, the pNFS layout delegation enables the client to directly 
   access the file system storage---if this access is not properly 
   managed by the NFSv4 server the client can potentially corrupt the 
   file system data or meta-data. 

   Thus it is vitally important that the client discover that the server 
   has rebooted as soon as possible, and that the client stops using 
   stale layout delegations before the server gives the delegations away 
   to other clients.  To ensure this, the client must be implemented so 
   that layout delegations are never used to access the storage after 
   the client's lease timer has expired.  This prohibition applies to 
   all accesses, especially the flushing of dirty data to storage.  If 
   the client's lease timer expires because the client could not contact 
   the server for any reason, the client MUST immediately stop using the 
   layout delegation until the server can be contacted and the 
   delegation can be officially recovered or reclaimed. 

3.4. Additional Features - Not Needed or Recommended 

   This subsection is a place to record things that existing SAN or 
   clustered filesystems do that aren't needed or recommended for pNFS: 

   o  Callback for write-to-read downgrade.  Writers tend to want to 
      remain writers, so this feature may not be very useful. 

   o  HighRoad FMP implements several frequently used operation 
      combinations as single RPCs for efficiency; these can be 
      effectively handled by NFSv4 COMPOUNDs.  One subtle difference is 
      that a single RPC is treated as a single operation, whereas NFSv4 
      COMPOUNDs are not atomic in any sense.  This can result in 
      operation ordering subtleties, e.g., having to set the new EOF 
      *before* returning the layout extent that contains the new EOF, 
      even within a single COMPOUND. 

 
 
Black                     Expires April 2006                  [Page 16] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

   o  Queued request support.  The HighRoad FMP protocol specification 
      allows the server to return an "operation blocked" result code 
      with a cookie that is later passed to the client in a "it's done 
      now" callback.  This has not proven to be of great use vs. having 
      the client retry with some sort of back-off.  Recommendations on 
      how to back off should be added to the ops draft. 

   o  Additional client and server crash detection mechanisms.  As a 
      separate protocol, HighRoad FMP had to handle this on its own.  As 
      an NFSv4 extension, NFSv4's SETCLIENTID, STALE CLIENTID and STALE 
      STATEID mechanisms combined with implicit lease renewal and (per-
      file) layout stateids should be sufficient for pNFS. 

4. Security Considerations 

   Certain security responsibilities are delegated to pNFS clients.  
   Block/volume storage systems generally control access at a volume 
   granularity, and hence pNFS clients have to be trusted to only 
   perform accesses allowed by the layout extents it currently holds 
   (e.g., and not access storage for files on which a layout extent is 
   not held).  This also has implications for some NFSv4 functionality 
   outside pNFS.  For instance, if a file is covered by a mandatory 
   read-only lock, the server can ensure that only read-layout-
   delegations for the file are granted to pNFS clients.  However, it is 
   up to each pNFS client to ensure that the read layout delegation is 
   used only to service read requests, and not to allow writes to the 
   existing parts of the file.  Since block/volume storage systems are 
   generally not capable of enforcing such file-based security, in 
   environments where pNFS clients cannot be trusted to enforce such 
   policies, block/volume-based pNFS SHOULD NOT be used. 

   <TBD: Need discussion about security for block/volume protocol vis-a-
   vis NFSv4 security.  Client may not even use same identity for both 
   (e.g., for Fibre Channel, same identity as NFSv4 is impossible).  
   Need to talk about consistent security protection of data via NFSv4 
   vs. direct block/volume access.  Some of this extends discussion in 
   previous paragraph about client responsibility for security as part 
   of overall system.> 

5. Conclusions 

   <TBD: Add any conclusions> 

6. IANA Considerations 

   There are no IANA considerations in this document.  All pNFS IANA 
   Considerations are covered in [PNFS]. 
 
 
Black                     Expires April 2006                  [Page 17] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

7. Revision History 

   -00: Initial Version 

   -01: Rework discussion of extents as locks to talk about extents 
   granting access permissions.  Rewrite operation ordering section to 
   discuss deadlocks and races that can cause problems.  Add new section 
   on recall completion.  Add client copy-on-write based on text from 
   Craig Everhart. 

   -02: Fix glitches in extent state descriptions.  Describe most issues 
   as RESOLVED.  Most of Section 3 has been incorporated into the [PNFS] 
   draft, add NOTE to that effect and say that it will be deleted in the 
   next version of this draft (which should be a draft-ietf-nfsv4 
   draft).  Cleaning up a number of things have been left to that draft 
   revision, including the interlocks with the types in [PNFS], layout 
   striping support, and finishing the Security Considerations section. 

8. Acknowledgments 

   This draft draws extensively on the authors' familiarity with the the 
   mapping functionality and protocol in EMC's HighRoad system.  The 
   protocol used by HighRoad is called FMP (File Mapping Protocol); it 
   is an add-on protocol that runs in parallel with filesystem protocols 
   such as NFSv3 to provide pNFS-like functionality for block/volume 
   storage.  While drawing on HighRoad FMP, the data structures and 
   functional considerations in this draft differ in significant ways, 
   based on lessons learned and the opportunity to take advantage of 
   NFSv4 features such as COMPOUND operations.  The design to support 
   pNFS client participation in copy-on-write is based on text and ideas 
   contributed by Craig Everhart of IBM. 

9. References 

9.1. Normative References 

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 
             Requirement Levels", BCP 14, RFC 2119, March 1997. 

   [PNFS]    Goodson, G., et. al. "NFSv4 pNFS Extensions", draft-ietf-
             nfsv4-pnfs-00.txt, Work in Progress, October 2005. 

   TODO: Need to reference RFC 3530. 

9.2. Informative References 

   OPEN ISSUE: HighRoad and/or SAN.FS references? 
 
 
Black                     Expires April 2006                  [Page 18] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

Author's Addresses 

   David L. Black 
   EMC Corporation 
   176 South Street 
   Hopkinton, MA 01748 
       
   Phone: +1 (508) 293-7953 
   Email: black_david@emc.com 
    

   Stephen Fridella 
   EMC Corporation 
   32 Coslin Drive 
   Southboro, MA  01772 
       
   Phone: +1 (508) 305-8512 
   Email: fridella_stephen@emc.com 
    

Intellectual Property Statement 

   The IETF takes no position regarding the validity or scope of any 
   Intellectual Property Rights or other rights that might be claimed to 
   pertain to the implementation or use of the technology described in 
   this document or the extent to which any license under such rights 
   might or might not be available; nor does it represent that it has 
   made any independent effort to identify any such rights.  Information 
   on the procedures with respect to rights in RFC documents can be 
   found in BCP 78 and BCP 79. 

   Copies of IPR disclosures made to the IETF Secretariat and any 
   assurances of licenses to be made available, or the result of an 
   attempt made to obtain a general license or permission for the use of 
   such proprietary rights by implementers or users of this 
   specification can be obtained from the IETF on-line IPR repository at 
   http://www.ietf.org/ipr. 

   The IETF invites any interested party to bring to its attention any 
   copyrights, patents or patent applications, or other proprietary 
   rights that may cover technology that may be required to implement 
   this standard.  Please address the information to the IETF at ietf-
   ipr@ietf.org. 




 
 
Black                     Expires April 2006                  [Page 19] 

Internet-Draft         pNFS Block/Volume Layout            October 2005 
    

Disclaimer of Validity 

   This document and the information contained herein are provided on an 
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 
   ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 
   INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 
   INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 

Copyright Statement 

   Copyright (C) The Internet Society (2005). 

   This document is subject to the rights, licenses and restrictions 
   contained in BCP 78, and except as set forth therein, the authors 
   retain all their rights. 

Acknowledgment 

   Funding for the RFC Editor function is currently provided by the 
   Internet Society. 

 























 
 
Black                     Expires April 2006                  [Page 20]