Internet DRAFT - draft-benjamin-extendedcallbackinfo

draft-benjamin-extendedcallbackinfo



Network Working Group				       	   Matt Benjamin
Internet-Draft 					   Linux Box Corporation
Intended status: Informational		               December 10, 2011
Expires: June 10, 2012


AFS Callback Extensions (Draft 14)

draft-benjamin-extendedcallbackinfo-02

Abstract

AFS cache-control strategy is callback (invalidate) based. The 
AFS callback design allows a client to know when an object it 
has cached is no longer consistent, but the callback 
notification message itself provides no specific information 
about the triggering event. This is a protocol inefficiency, as 
in several scenarios it results in unnecessary round-trips to 
file servers to verify file status information, file access 
information, or to fetch file data which has not changed. We 
propose an extension of the callback mechanism to provide 
information about the event(s) triggering a callback, in the 
payload of the callback notification message itself. The 
proposed mechanism eliminates most or all unnecessary 
round-trips imposed by the current callback mechanism, and 
simultaneously allows AFS implementations to (efficiently) 
provide correct semantics in several scenarios involving 
multiple writers (ie, where AFS currently provides incorrect 
semantics).

Status of this Memo

This Internet-Draft is submitted in full conformance with the 
provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet 
Engineering Task Force (IETF), its areas, and its working 
groups. Note that other groups may also distribute working 
documents as Internet-Drafts. 

Internet-Drafts are draft documents valid for a maximum of six 
months and may be updated, replaced, or obsoleted by other 
documents at any time. It is inappropriate to use 
Internet-Drafts as reference material or to cite them other 
than as "work in progress."

Benjamin            November 29, 2011                  [Page 2]

The list of current Internet-Drafts can be accessed at 
http://www.ietf.org/1id-abstracts.html

The list of Internet-Draft Shadow Directories can be accessed 
at http://www.ietf.org/shadow.html

Copyright Notice

Copyright (c) 2011 IETF Trust and the persons identified as the 
document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal 
Provisions Relating to IETF Documents 
(http://trustee.ietf.org/license-info) in effect on the date of 
publication of this document. Please review these documents 
carefully, as they describe your rights and restrictions with 
respect to this document. 


Table of Contents

Abstract
Status of this Memo
Copyright Notice
    1 Introduction
    2 Conventions Used in this Document
    3 The AFS Callback Mechanism
        3.1 Description
        3.2 Analysis
    4 Extended Callback Interface
        4.1 Backward Compatibility
        4.2 Interface Changes
            4.2.1 Procedures
            4.2.2 Constants
                All Sequences
                AFSXCBInvocation
                AFSExtendedCallBack
                AFSXCBRInvocation
                AFSExtendedCallBackResult
                Sequence Types
            4.2.3 Data Types
                HostIdentifier
                AFSXCBInvocation
                Fid
                Flags
                lowDV
                highDV
                ExpirationTime
                CallBacks_Array
                AFSExtendedCallBack
                Flags and ExtraFlags
                Origin
                NCoalesced
                DataVersion
                Data
                AFSXCBRInvocation
                AFSExtendedCallBackResult
                AFSCBFileStatus
                AFSCBDirStatus
                AFSCB_NotificationData
        4.3 Semantic Changes
            4.3.1 DataVersion Rule
            4.3.2 Callback Synchrony and Callback Bracketing
            4.3.3 Callback Workload Considerations
        4.4 Callback Invocations
            4.4.1 AFSXCBInvocation
                Origin
                ExpirationTime
            4.4.2 AFSExtendedCallBack
                Flags
                ExtraFlags
                DataVersion
                Data
            4.4.3 AFSXCBRInvocation
                Xcb_Result_Array
            4.4.4 AFSExtendedCallBackResult
                Flags
                ExtraFlags
                Data
            4.4.5 ExtendedCallBack Procedure
            4.4.6 Callback Coalescing
                Call Consolidation (Sequences of Notifications)
                Coalescing of Equivalent Notifications
                Implementation Note
            4.4.7 AFSCB_Event_Cancel
                Reasons for Cancellation
                AFSCB_Cancel_Shutdown
                AFSCB_Cancel_CallbackGC
                AFSCB_Cancel_VolumeOffline
                AFSCB_Cancel_VolumeMoved
                AFSCB_Cancel_LostMyMind
                AFSCB_Cancel_IHateYou
            4.4.8 AFSCB_Event_StoreData
            4.4.9 AFSCB_Event_StoreACL
            4.4.10 AFSCB_Event_StoreStatus
            4.4.11 AFSCB_Event_CreateFile
            4.4.12 AFSCB_Event_MakeDir
            4.4.13 AFSCB_Data_Symlink
            4.4.14 AFSCB_Event_Link
            4.4.15 AFSCB_Event_RemoveFile
            4.4.16 AFSCB_Event_RemoveDir
            4.4.17 AFSCB_Event_Rename
            4.4.18 AFSCB_Event_Deleted
            4.4.19 AFSCB_Event_ReleaseLock
        4.5 Callbacks And Read-Only Volume Replicas
            4.5.1 Constants
                AFSCB_Flag_Release
                AFSCB_IFlag_Release
                AFSCB_Release_WholeVolumeCancel
            4.5.2 Semantic Changes
    5 Security Considerations
                Edinburgh Consensus
    6 IANA Considerations
    7 Acknowledgements 
    8 Appendix A: XDR Grammar
    9 Informative References
    Author's Address


1 Introduction

The AFS protocol provides a comprehensive framework for 
scalable, secure, wide-area file sharing over IP networks. The 
AFS system has historically distinguished itself through its 
emphasis on client-side caching[3, 7]. File data, file and 
directory metadata, and access control information may all be 
cached. Cache consistency is maintained through client 
registration and an associated asynchronous notification 
mechanism known as the callback. 

The current AFS consistency model (which is of larger scope 
than the callback mechanism, eg, it includes AFS sync-on-close 
semantics) has allowed AFS to scale to large numbers of clients 
(tens of thousands today), and to perform well under the 
workloads for which AFS was originally designed.

However, AFS does not perform efficiently under other 
conditions, such as when more than one client is interested in 
a file which is changing--even if the file has only one writer, 
and many readers[footnote:
NFSv4.1 in particular efficiently supports this scenario with 
byte-range delegation, see[9]. 
]. In general, the AFS protocol arguably (still, considering 
improvements made between AFS-2 and AFS-3) places too little 
emphasis on efficient caching of mutable data. The current AFS 
consistency model is insufficient to correctly support 
single-file, multiple-writer scenarios, including those 
required for POSIX semantics, and therefore is insufficient to 
support many applications which may be run correctly on 
competing distributed file systems (e.g., CIFS, Novell Netware, 
or NFSv4).

Benjamin            November 29, 2011                  [Page 3]

The efficiency of the current AFS cache management algorithm 
can be substantially improved if specific triggering event 
information and current status are included in the payload of 
the callback notifications sent to clients. In particular, 
inclusion of the current DataVersion number and affected byte 
ranges in response to StoreData operations significantly 
reduces the need for cache revalidation and reconstruction 
traffic in response to callbacks--in many cases, altogether. 
These changes allow efficient support for single-writer updates 
on a file with multiple readers. More importantly, they permit 
AFS to correctly and efficiently support multiple writers 
updating disjoint ranges on a single file, a prerequisite for 
supporting granular file locking (and applications which 
require it) in future.

2 Conventions Used in this Document

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL 
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and 
"OPTIONAL" in this document are to be interpreted as described 
in RFC 2119.

3 The AFS Callback Mechanism

3.1 Description

When an AFS-3 client contacts a file server to perform any of 
several operations on a file, or explicitly to fetch its 
status, the file server includes in its RPC response an 
AFSCallBack structure, representing the server's promise to 
call back the client "if any modifications are made to the data 
in the file." (A key paper on AFS-2 has "before allowing a
modification by any other workstation".)  The AFSCallBack
structure contains the callback expiration time, and two integer
values treated as invariants.

When any client executes an operation which would change a file 
(e.g., StoreData), and in a variety of other situations, the 
file server invalidates the client's cached copy by executing a 
call to the CallBack procedure in the client's RPC interface. (The
call includes in its arguments an AFSCallBack structure for each file
being invalidated. However, the value of the passed AFSCallBack is 
unused [e.g., afs/afs_callback.c:643 ff., openafs-1.5.54]). 
Between the time of issue and either expiry or receipt of a 
callback, the client may consider any information it has cached 
on a file to be consistent with the file server's on-disk copy. 
Conversely, on receipt of a callback, the client must consider 
that it knows nothing about the file. Thus the client must 
re-establish a relationship with the file at the file server 
before executing any further operations on it.

Benjamin            November 29, 2011                  [Page 4]

The AFS callback mechanism obviates the need for clients to 
send frequent cache validation requests before performing 
operations on their locally cached copies of objects, reducing 
network traffic as well as file server workload[3, 7]. The 
callback innovation has been since taken up, with variations, 
by other distributed file system protocols[4, 6].

3.2 Analysis

The AFS callback mechanism reliably notifies clients when 
information they may have cached becomes invalidated, but omits 
to send information it trivially knows, ie, the triggering 
event, that could certainly be used by the client to more 
efficiently manage cache state.

For example, consider the case where 2 clients A and B are 
interested in a file F, each having read chunks 1-15 into 
cache. Now another client C initiates a change in the file, 
writing a new state to chunk 45. This event increments the 
dataversion of the file, and triggers a callback to A and B. 
(C, because it initiated the change, is not called back.) On 
receipt of the callback, A and B must issue FetchStatus 
requests on F to acquire its current status information, 
including its current data version. Since the data version of F 
has increased, any chunks of F which A or B has cached are 
invalidated, including 30 chunks correctly cached. Should A or 
B remain interested, it must refetch these chunks (up to 2 
megabytes of data, in this case). This scenario will occur 
reasonably often in environments where mutable data is common, 
and a related scenario involving directory entries (omitted for 
brevity) is much more common. In these cases, an AFS callback 
mechanism capable of sending triggering event information with 
the callback would have facilitated a more efficient result, at 
small marginal cost. In another set of scenarios where a client 
A has changed data in a file invalidated by non-overlapping 
stores by B, a revised mechanism would be capable of delivering 
a correct result, whereas a correct result would be impossible 
with the mechanism in AFS today. (In the AFS-3 callback model, 
either As or Bs changes must be rejected. In the extended 
callback model, the range-based invalidate mechanism means that 
As and Bs changes will be merged, as they are disjoint.)

The justification for sending minimal information with the 
callback is presumably to minimize the execution cost of the 
callback procedure. The increased cost of sending a limited but 
informative callback notification to clients, relative to 
sending an uninformative one, is small. Analysis of the OpenAFS 
file server code reveals that the file server always has the 
information that would logically be sent as extended callback 
information in response to file operations (e.g., file ranges 
affected by StoreData operations, or changed entries for 
various directory modification operations). 

Benjamin            November 29, 2011                  [Page 5]

For these reasons, enhancement of the AFS callback interface to 
supply triggering event information seems likely to improve 
both correctness and performance of AFS implementations, and 
experimental implementation and profiling appear justified.

4 Extended Callback Interface

4.1 Backward Compatibility

AFS clients will indicate their preference to receive extended 
callback notifications through a new client capability flag:

const CLIENT_CAPABILITY_EXT_CALLBACK = 0x0002;

4.2 Interface Changes

4.2.1 Procedures

We propose a new procedure ExtendedCallBack in the client's RPC 
interface. The ExtendedCallBack procedure arguments consist of 
a HostIdentifier containing the UUIDs of the sending fileserver 
and of its cell, and a (variable-length) sequence of 
AFSXCBInvocation structures. And AFSXCBInvocation represents a 
(variable-length) sequence of AFSExtendedCallBack events on one 
AFSFid at Server. One invocation of the ExtendedCallBack 
procedure can thus deliver up to AFSXCBMAX event notifications 
on each of up to AFSXCBMAX fids. An OUT-direction sequence of 
variant AFSExtendedCallBackResult structures is added for 
future callback notification styles (e.g., locks, delegations) 
which may return structured data on receipt of notifications:

proc ExtendedCallBack(

    IN HostIdentifier *Server,

       AFSXCBInvocationSeq *Invocations_Array,

    OUT AFSXCBRInvocationSeq *Result_Array

) multi = 65540;



4.2.2 Constants<sub:Constants>

  All Sequences

The AFSXCBMAX constant is the maximum allowed length for 
AFSXCBInvocation and AFSExtendedCallBack sequences:

Benjamin            November 29, 2011                  [Page 6]

const AFSXCBMAX = 512;

  AFSXCBInvocation

As detailed in section [sub:Data-Types], an AFSXCBInvocation is 
a structure representing a sequence of XCB events on one Fid. 
The following constants are flag values are used as flag values 
on AFSXCBInvocation instances:

const AFSCB_IFlag_SOrigin = 1; /* callbacks on this invocation 
have a single origin host */

const AFSCB_IFlag_Release = 2; /* this invocation was triggered 
by a volume release */

  AFSExtendedCallBack

As detailed in section [sub:Data-Types], an AFSExtendedCallBack 
is a structure representing an XCB event, and is principally 
constituted by an instance of an XDR union, discriminated on 
the callback event type. The following callback event types are 
defined:

const AFSCB_Event_Cancel = 1; /* extended break callback */

const AFSCB_Event_StoreData = 2; /* data in file changed */

const AFSCB_Event_StoreACL = 3; /* ACL changed on vnode */

const AFSCB_Event_StoreStatus = 4; /* status stored on vnode */

const AFSCB_Event_CreateFile = 5; /* file created in directory 
vnode */

const AFSCB_Event_MakeDir = 6; /* dir created in directory 
vnode */

const AFSCB_Event_Symlink = 7; /* symlink created in directory 
vnode */

const AFSCB_Event_Link = 8; /* hard link created in directory 
vnode */

const AFSCB_Event_RemoveFile = 9; /* file removd from directory 
vnode */

const AFSCB_Event_RemoveDir = 10; /* dir removed from directory 
vnode */

Benjamin            November 29, 2011                  [Page 7]

const AFSCB_Event_Rename = 11; /* object renamed (moved) */

const AFSCB_Event_Deleted = 12; /* object no longer exists, ex 
object */

const AFSCB_Event_ReleaseLock = 13; /* traditional AFS lock 
released */

A flag constant is provided to indicate callback cancellation 
along with an extended notification message of any of the above 
types:

const AFSCB_Flag_Cancel = 1; /* Callback promise is cancelled 
*/

The following constants indicate reasons for cancellation, when 
(Flags & AFSCB_Flag_Cancel):

const AFSCB_Cancel_Shutdown = 1;

const AFSCB_Cancel_CallbackGC = 2;

const AFSCB_Cancel_VolumeOffline = 3;

const AFSCB_Cancel_VolumeMoved = 4;

const AFSCB_Cancel_LostMyMind = 5;

const AFSCB_Cancel_IHateYou = 6; 

The following flag constants (Flags) indicate the beginning or 
end of a callback bracketing sequence (each is orthogonal to 
each other and to AFSCB_Flag_Cancel):

const AFSCB_Flag_OpenBracket = 2; /* Callback syncrhony is 
interrupted */

const AFSCB_Flag_CloseBracket = 4; /* Callback syncrhony is 
restored */

The following constants indicate direction (from or to called 
back FID) in the atomic AFSCB_Event_Rename notification:

const AFSCB_Rename_From = 1;

const AFSCB_Rename_To = 2;

  AFSXCBRInvocation

Benjamin            November 29, 2011                  [Page 8]

As detailed in section [sub:Data-Types], an AFSRXCBInvocation 
is a structure representing the sequence of results of XCB 
events delivered in one AFSXCBInvocation.

  AFSExtendedCallBackResult

As detailed in section [sub:Data-Types], an 
AFSExtendedCallBackResult is a structure describing the result 
of an XCB event. The following constants a provided as 
descriminator for the AFSCB_ResultData union:

const AFSCB_Result_NoResult = 1;

const AFSCB_Result_Diag = 2; 

const AFSCB_Result_Generic = 3; /* all currently defined 
messages */

  Sequence Types

The following sequences are defined, and are used to construct 
the input and output arguments for the ExtendedCallBack 
procedure:

typedef AFSXCBInvocation AFSXCBInvocationSeq<AFSXCBMAX>;

typedef AFSExtendedCallBack AFSExtendedCallBackSeq<AFSXCBMAX>;

typedef AFSXCBRInvocation AFSXCBRInvocationSeq<AFSXCBMAX>;

typedef AFSExtendedCallBackResult 
AFSExtendedCallBackRSeq<AFSXCBMAX>;

4.2.3 Data Types<sub:Data-Types>

  HostIdentifier

A HostIdentifier structure contains the unique server and cell 
UUIDs of a specific host in some AFS cell.

struct HostIdentifier {

    afsUUID ServerUuid;

    afsUUID CellUuid;

};

  AFSXCBInvocation

Benjamin            November 29, 2011                  [Page 9]

The AFSXCBInvocation data type represents a sequence of 0 or 
more callback events on one fid. The enclosed 
AFSExtendedCallBack objects MUST be in DataVersion order.

  Fid

Fid is the fid object of the callback sequence.

  Flags

Flags provide specializing information about the invocation.

  lowDV

The lowest data version of Fid at all events in the sequence.

  highDV

The highest data version of Fid at all events in the sequence.

  ExpirationTime

ExpirationTime indicates a new expiration time for the 
receiving client's callback on fid. And ExpirationTime of 0 
indicates no change in ExpirationTime.

  CallBacks_Array

A sequence of 0 or more AFSExtendedCallBack notifications on 
FID.

struct AFSXCBInvocation {

    AFSFid Fid;

    afs_uint32 Flags;

    afs_uint64 lowDV; /* lowest DV at invocation */

    AFSTimestamp highDV; /* highest */

    AFSTimestamp ExpirationTime; /* new expiration, or 0 if 
unchanged */

    AFSExtendedCallBackSeq CallBacks_Array;

};

  AFSExtendedCallBack

Benjamin            November 29, 2011                  [Page 10]

The AFSExtendedCallBack data type represents a single callback 
event on some fid, that of its containing AFSXCBInvocation when 
sent with an ExtendedCallBack RPC. 

  Flags and ExtraFlags

Flags and ExtraFlags (added for future expansion) provide 
possibly event-specific information. 

  Origin

Origin is the AFS UUID of the host or server which originated 
the event, ie, the client whose operation on fid triggered some 
event, in the typical case. If the origin is unknown to the 
server or would not be meaningful, it MAY send the null UUID. 

  NCoalesced

As specified later in this document, certain operations (ie, 
StoreData, StoreStatus) MAY be regarded by the file server as 
idempotent and sent as one callback. NCoalesced indicates the 
number of equivalent or combined operations coalesced on the 
event, or 0 if the event is singular. 

  DataVersion

DataVersion is the (possibly updated) data version of fid at 
the completion of the operation which triggered the event. 
Considering coalescing, DataVersion is the data version at the 
completion of the first event in the coalesced sequence. 

  Data

Data is an object of the discriminated union type 
AFSCB_NotificationData:

struct AFSExtendedCallBack {

    afs_uint32 Flags;

    afs_uint32 ExtraFlags;

    afsUUID Origin; /* originator of changes */

    afs_uint32 NCoalesced; /* calls combined on this */

    afs_uint64 DataVersion;

    AFSCB_NotificationData Data;

Benjamin            November 29, 2011                  [Page 11]

};

A non-zero value in Flags for the AFSCB_Flag_Cancel bit 
indicates cancellation of the callback upon receipt of the 
message. In that event, a non-zero value of ExtraFlags 
indicates the reason for the cancellation.

  AFSXCBRInvocation

An AFSXCBRInvocation is a structure describing the result of an 
XCB invocation (a sequence of extended callback events at one 
Fid).

struct AFSXCBRInvocation {

    AFSExtendedCallBackRSeq Xcb_Result_Array;

};

  AFSExtendedCallBackResult

An AFSExtendedCallBackResult is a structure describing the 
result of a single XCB event.

struct AFSExtendedCallBackResult {

    afs_uint32 Flags;

    afs_uint32 ExtraFlags;

    AFSCB_ResultData Data;

}; 

  AFSCBFileStatus

The AFSCBFileStatus structure is a reduced-footprint 
AFSFetchStatus replacement intended to communicate changed 
vnode information in response to StoreData operations:

struct AFSCBFileStatus {

    afs_uint32 LinkCount;

    afs_uint64 ClientModTime;

};

  AFSCBDirStatus

Benjamin            November 29, 2011                  [Page 12]

The AFSCBDirStatus structure is a reduced-footprint 
AFSFetchStatus replacement intended to communicate changed 
vnode information in response to directory change operations:

struct AFSCBDirStatus {

    afs_uint32 LinkCount;

    afs_uint64 ClientModTime;

};

  AFSCB_NotificationData

AFSCB_NotificationData is a union discriminated by callback 
event type, ie, its value may be any of the constants defined 
in section [sub:Constants].

ext-union AFSCB_NotificationData switch (afs_uint32 Event_Type) 
{

case AFSCB_Event_StoreData:

    AFSCB_Data_StoreData u_store_data;

case AFSCB_Event_StoreACL:

    void;

case AFSCB_Event_StoreStatus:

    AFSCB_Data_StoreStatus u_store_status;

case AFSCB_Event_CreateFile:

    AFSCB_Data_CreateFile u_create_file;

case AFSCB_Event_MakeDir:

    AFSCB_Data_MakeDir u_make_dir;

case AFSCB_Event_Symlink:

    AFSCB_Data_Symlink u_symlink;

case AFSCB_Event_Link:

    AFSCB_Data_Link u_link;

Benjamin            November 29, 2011                  [Page 13]

case AFSCB_Event_RemoveFile:

    AFSCB_Data_RemoveFile u_remove_file;

case AFSCB_Event_RemoveDir:

    AFSCB_Data_RemoveDir u_remove_dir;

case AFSCB_Event_Rename:

    AFSCB_Data_Rename u_rename;

case AFSCB_Event_Deleted:

    void;

case AFSCB_Event_ReleaseLock:

    AFSCB_Data_Lock u_lock;

case AFSCB_Event_Cancel:

    void;

};

The types for the variant member u_data are enumerated and 
discussed in detail in section [sub:Callback-Invocations]. 

4.3 Semantic Changes

A file server MAY send traditional callback messages, with 
traditional semantics, to any AFS client in response to any 
event. A file server MAY send extended callback notifications 
to any client which has announced the capability to use the 
extended interface, with the following semantics:

* extended callback notification messages, in general, preserve 
  the file server's callback promise to send further 
  notifications for the called-back FID

* the file server MAY cancel a client's registration (callback 
  promise) with any extended callback notification message, by 
  setting AFSCB_Flag_Cancel in the Flags member of the 
  AFSExtendedCallBack structure

* the AFSCB_Event_Cancel message is similar to a traditional 
  AFS callback, breaking the callback promise, and requesting 
  the client not request further status on the FID

Benjamin            November 29, 2011                  [Page 14]

* the file server MAY suspend its obligation to deliver 
  callback notifications synchronously without revoking an 
  object's registration, by setting AFSCB_Flag_OpenBracket in 
  the Flags member of the AFSExtendedCallBack structure

* a file server which as previously suspended synchronous 
  notification on an object (with AFSCB_Flag_OpenBracket) MAY 
  restore it by sending AFSCB_Flag_CloseBracket in Flags in a 
  subsequent message

  * callback bracketing, if enabled, terminates automatically 
    on ordinary cancellation or expiration of the corresponding 
    callback registration

4.3.1 DataVersion Rule

The various extended callback notification messages include 
information a client may use to selectively invalidate or 
reconstruct its cache. In interpreting each message, the client 
MUST observe the dataversion rule, which states:

If the client's cached DataVersion is DataVersion or 
(DataVersion-1), the client MAY invalidate or update its cache 
using the type-dependent information contained in the message. 
In all other cases, the client MUST regard the message as 
equivalent to a traditional AFS callback.

The semantics of specific callback events are enumerated in 
section [sub:Callback-Invocations].

4.3.2 Callback Synchrony and Callback Bracketing

The AFS prototol regards client data and metadata operations as 
atomic and synchronous. A legacy AFS callback registration 
establishes a domain of synchrony between the registering file 
server(s) and each registered client. A client operating on any 
object in the file server interface is assured that each 
eligible client has been notified of its operation when the 
operation completes.

The extended callback mechanism carries forward these semantics 
in full, with the following modification, referred to as 
callback bracketing: A synchronous notification may suspend the 
sending server's obligation to deliver synchronous 
notifications without otherwise removing the receiving client's 
registration on a Fid. The motivation for callback bracketing 
is to permit more efficient processing of sequential operations 
at one or several clients, through callback coalescing, without 
violating any client's expectation of synchrony. Since an 
AFSCB_Flag_OpenBracket indication is delivered synchronously to 
any client which would be assured to receive an ordinary legacy 
or extended callback notification with synchrony, in any 
circumstance in which a server would be obligated to send 
callback breaks or notifications, the callback bracketing 
mechanism does not violate any client's expectation of 
synchrony.

Benjamin            November 29, 2011                  [Page 15]

4.3.3 Callback Workload Considerations

The preference to preserve the fileserver's callback promises 
to clients across file operations is a significant behavioral 
change. In particular, file servers configured with traditional 
small callback databases may be vulnerable to callback 
exhaustion, which in turn may lead to thrashing or other 
undesirable behavior. Sites are recommended to explicitly 
increase provisioned callbacks, likely to at least 1 million. 
The OpenAFS viced implementation will increase the number of 
callbacks in the "large" configuration (-L) to a new, large 
value. In general, a file server is expected take appropriate 
action to shed workload (e.g., break callbacks) whenever 
appropriate.

4.4 Callback Invocations<sub:Callback-Invocations>

The various extended callback notification types generally 
respond to specific events at the file server, but present a 
view of it relevant to a specific callback promise at one 
client. In one case (ie, AFSCB_Event_Rename), the file server 
is sending notification of an event which effects two FIDs, 
either or both of which may be cached by the receiving client. 
A structure of type AFSExtendedCallBack is sent with each 
extended callback notification message, as noted above. Unless 
otherwise noted, FID is the FID of the object that is the 
subject of the callback. 

4.4.1 AFSXCBInvocation

  Origin

A file server MAY omit to send extended callback notifications 
triggered by a file operation to the client host which 
originated the change. (Omission to send such callbacks has 
been the general behavior of AFS file servers.) A client MUST 
be prepared to appropriately process (or ignore) callbacks for 
which its own UUID is the Origin.

  ExpirationTime

The new expiration time asserted for the server's callback 
promise, not necessarily different from the existing expiration 
cached by the client.

4.4.2 AFSExtendedCallBack

The members of the AFSExtendedCallBack structures are to be 
interpreted as follows:

Benjamin            November 29, 2011                  [Page 16]

  Flags

If AFSCB_Flag_Cancel is set, the notification effects a 
callback break. The client may make use of the information sent 
with the message. If AFSCB_Flag_OpenBracket is set, the 
notification (if actionable with respect to DataVersion) 
suspends the server's obligation to send synchronous 
notifications on FID. The client may make use of the 
information sent with the message. The new state persists until 
restored by a subsequent AFSCB_Flag_OpenBracket notification or 
client-initiated callback-granting operation.

  ExtraFlags

If (Flags & AFSCB_Flag_Cancel), a non-zero value for ExtraFlags 
indicates the reason for cancellation.

  DataVersion

The value of DataVersion at completion of the event of which 
the client is being notified. Considering coalescing, the new 
data version after completion of all events summarized at this 
callback is (DataVersion+NCoalesced).

  Data

The message-specific data for this notification.

4.4.3 AFSXCBRInvocation

The AFSXCBRInvocation structure contains only an array of 
AFSExtendedCallBackResult.

  Xcb_Result_Array

A sequence of AFSExtendedCallBackResult objects dimensioned by 
the matching Callbacks_Array argument of an ExtendedCallBack 
invocation.

4.4.4 AFSExtendedCallBackResult

An AFSExtendedCallBackResult is a structure describing the 
result of a single XCB event.

  Flags

Provided for future extension.

  ExtraFlags

Benjamin            November 29, 2011                  [Page 17]

Provided for future extension.

  Data

The result, expressed as a union of nil, diagnostic/string, and 
generic result types. The generic result type (or future 
extension) should be used in general, as it allows for 
per-message acknowledgement.

4.4.5 ExtendedCallBack Procedure

Extended callbacks are delivered through a new ExtendedCallBack 
procedure.

proc ExtendedCallBack(

    IN HostIdentifier *Server,

       AFSXCBInvocationSeq *Invocations_Array,

    OUT AFSXCBRInvocationSeq *Result_Array

) multi = 65540;

ExtendedCallBack provides for flexible event notification, 
including bulk notification support by Fid and per Fid, 
supports per-message acknowledgement, and uniquely identifies 
the issuing server host.

4.4.6 Callback Coalescing

A server implementation MAY coalesce sequences of 
effectively-simultaneous notifications to a single client, in 
accordance with rules of composition of specific notifications, 
and provided doing so would not violate any client's 
expectation of synchrony.

  Call Consolidation (Sequences of Notifications)

A server implementation MAY coalesce any sequence of 
effectively simultaneous notifications into sequences of 
AFSExtendedCallBack objects enclosed in one AFSXCBInvocation 
object, provided doing so would not violate any client's 
expectation of synchrony. Any number of such callbacks may be 
combined, up to the limit of AFSXCBMAX.

  Coalescing of Equivalent Notifications

A server implementation MAY coalesce a sequence of effectively 
simultaneous and equivalent notifications to the same client 
into a single callback in a notification message, provided 
doing so would not violate any client's expectation of 
synchrony. The following combinations of operations are 
explicitly permitted:

Benjamin            November 29, 2011                  [Page 18]

* sequences of AFSCB_EventStoreAcl notifications on FID from a 
  single Origin MAY be delivered as a single notification

* sequences of AFSCB_EventStoreStatus notifications on FID from 
  a single Origin MAY be delivered as the single notification 
  of the most recently stored status

* sequences of AFSCB_Event_StoreData notifications on FID from 
  a single Origin at adjacent or overlapping byte ranges MAY 
  deliver a single notification at the consolidated range

  Implementation Note

"Effectively simultaneous" is left intentionally unspecified. 
An adaptive window expanding from 100ms to a small number of 
seconds appears to work well with commonly available switched 
networks and multi-core fileservers, in 2008. The current 
OpenAFS implementation uses an adaptive per-Fid window, which 
extends on repeated events, and closes when client contention 
is detected. Peer RTT or other considerations may be added to 
the windowing algorithm in future.

4.4.7 AFSCB_Event_Cancel

The AFSCB_Event_Cancel notification indicates that the client's 
callback promise on the corresponding Fid is cancelled. It is 
therefore equivalent to a legacy AFS break call back 
indication, but uses the extended interface. A cancel 
indication may include an optional reason for cancellation in 
the Flags member of the corresponding AFSExtendedCallBack 
message.

  Reasons for Cancellation

The following reasons for cancellation are defined:

  AFSCB_Cancel_Shutdown

The server or service is shutting down.

  AFSCB_Cancel_CallbackGC

Callback has been disposed during periodic garbage collection.

  AFSCB_Cancel_VolumeOffline

The volume associated with FID is now offline.

  AFSCB_Cancel_VolumeMoved

Benjamin            November 29, 2011                  [Page 19]

The volume associated with FID has moved.

  AFSCB_Cancel_LostMyMind

The server may be having problems related to provisioning an 
insufficient number of callback structures.

  AFSCB_Cancel_IHateYou

Callback has been administratively revoked.

4.4.8 AFSCB_Event_StoreData

The notification is sent in response to a successful StoreData 
RPC on FID. A structure of type AFSCB_Data_StoreData is sent 
with the message.

struct AFSCB_Data_StoreData {

    afs_uint64 StoreOffset;

    afs_uint64 StoreLength;

    afs_uint64 Length;

    AFSCBFileStatus FileStatus;

};

StoreLength bytes were stored starting at position StoreOffset 
in FID. Length is the current file length and FileStatus 
contains the modification time of FID following the operation. 
The client must regard cached file data in the range 
[StoreOffset, StoreOffset+StoreLength) as invalidated, and may 
regard data outside that range as up-to-date. The client MUST 
discard undirtied cached data in the invalidated range. The 
client MAY send dirtied data in the invalidated range to the 
file server prior to discarding (as allowed in current AFS 
semantics).

4.4.9 AFSCB_Event_StoreACL

ACL and/or access information cached by the client for FID, if 
any, is invalidated.

4.4.10 AFSCB_Event_StoreStatus

A StoreStatus RPC was successfully executed on FID. A structure 
of type AFSFetchStatus is sent with the message.

Benjamin            November 29, 2011                  [Page 20]

struct AFSCB_Data_StoreStatus {

    struct AFSFetchStatus Status;

};

Status is the new AFSFetchStatus of FID, ie, the message 
communicates the current status information of FID.[footnote:
This is changed from earlier drafts.
]

4.4.11 AFSCB_Event_CreateFile

A file has been created in the vnode corresponding to FID. A 
structure of type AFSCB_Data_CreateFile is sent with the 
message.

struct AFSCB_Data_CreateFile {

    string Name<AFSNAMEMAX>;

    AFSFid Fid;

    AFSFetchStatus FidStatus;

    AFSCBDirStatus DirStatus;

};

Name and Fid are, respectively, the name and FID of the created 
file. FidStatus is the AFSFetchStatus of the created file, and 
DirStatus the current modification time and link count of FID, 
at the completion of the call.

4.4.12 AFSCB_Event_MakeDir

A directory has been created in the vnode corresponding to FID. 
A structure of type AFSCB_Data_MakeDir is sent with the 
message.

struct AFSCB_Data_MakeDir {

    string Name<AFSNAMEMAX>;

    AFSFid Fid;

    AFSFetchStatus FidStatus;

    AFSCBDirStatus DirStatus;

};;

Benjamin            November 29, 2011                  [Page 21]

Name and Fid are, respectively, the name and FID of the created 
directory. FidStatus is the AFSFetchStatus of the created 
directory, and DirStatus the current modification time and link 
count of FID, at the completion of the call.

4.4.13 AFSCB_Data_Symlink

A symbolic link has been created in the vnode corresponding to 
FID. A structure of type AFSCB_Data_Symlink is sent with the 
message.

struct AFSCB_Data_Symlink {

    string Name<AFSNAMEMAX>;

    AFSFid Fid;

    string LinkContents<AFSPATHMAX>;

    AFSFetchStatus FidStatus;

    AFSCBDirStatus DirStatus;

};

Name is the name of the symbolic link. Fid is its AFSFid. The 
link points to LinkContents. FidStatus is the AFSFetchStatus of 
the created symbolic link, and DirStatus the current 
modification time and link count of FID, at the completion of 
the call.

4.4.14 AFSCB_Event_Link

A hard link has been created in the vnode corresponding to FID. 
A structure of type AFSCB_Data_Link is sent with the message.

struct AFSCB_Data_Link {

    string Name<AFSNAMEMAX>;

    AFSFid LinkTarget;

    AFSFetchStatus FidStatus;

    AFSCBDirStatus DirStatus;

};

Name is the name of the link. The link is a synonym for 
LinkTarget. FidStatus is the AFSFetchStatus of the created 
link, and DirStatus the current modification time and link 
count of FID, at the completion of the call.

Benjamin            November 29, 2011                  [Page 22]

4.4.15 AFSCB_Event_RemoveFile

A file has been removed from the vnode corresponding to FID. A 
structure of type AFSCB_Data_RemoveFile is sent with the 
message.

struct AFSCB_Data_RemoveFile {

    string Name<AFSNAMEMAX>;

    AFSCBDirStatus DirStatus;

};

Name indicates the removed entry. DirStatus the current 
modification time and link count of FID, at the completion of 
the call.

4.4.16 AFSCB_Event_RemoveDir

A directory has been removed from the vnode corresponding to 
FID. A structure of type AFSCB_Data_RemoveDir is sent with the 
message.

struct AFSCB_Data_RemoveDir {

    string Name<AFSNAMEMAX>;

    AFSCBDirStatus DirStatus;

};

Name indicates the removed entry. DirStatus the current 
modification time and link count of FID, at the completion of 
the call.

4.4.17 AFSCB_Event_Rename

A file or directory has been renamed, ie moved, from or to the 
vnode corresponding to FID. A structure of type 
AFSCB_Data_RemoveDir is sent with the message.

const AFSCB_Rename_From = 1;

const AFSCB_Rename_To = 2;



struct AFSCB_Data_Rename {

Benjamin            November 29, 2011                  [Page 23]

    afs_uint32 Direction;

    string OldName<AFSNAMEMAX>;

    string NewName<AFSNAMEMAX>;

    AFSCBDirStatus FromStatus;

    AFSCBDirStatus ToStatus;

};

Direction indicates whether FID is the source or the 
destination directory of the move. OldName is the name of the 
object in its old location, NewName the name of the object in 
its new location. FromStatus is the current modification time 
and link count of the source directory vnode, and ToStatus is 
the current modification time and link count of the destination 
directory vnode, and FidStatus the at the completion of the 
call. 

To preserve atomicity, the AFSCB_Data_Rename message is 
constructed so that changes to cached copies of both the source 
and directory vnodes may be recovered from a single 
notification. If a client owns callbacks for both the source 
and destination FIDs, a file server MAY elect to send only one 
notification, for either the source or the destination FID.

4.4.18 AFSCB_Event_Deleted

The object corresponding to FID not longer exists, and so may 
no longer be cached. It is an ex-object. (I.e., the client MUST 
discard any information it has cached about FID.)

4.4.19 AFSCB_Event_ReleaseLock

A traditional AFS whole-file lock has been released on FID. A 
structure of type AFSCB_Data_Lock is sent with the message. 
LockType is the type of the lock released.

struct AFSCB_Data_Lock {

    afs_uint32 LockType;

};

Receipt of an AFSCB_Event_ReleaseLock notification does not 
imply that a lock on FID will be immediately available to the 
receiving client (i.e., it is not a reservation). Non-receipt 
of a notification of this type does not imply non-release of 
locks that may be (may have been) held on FID. A file server 
SHOULD send notifications of this type only to clients which 
have indicated probable interest in the event, e.g., by having 
recently requested a lock on FID.

Benjamin            November 29, 2011                  [Page 24]

4.5 Callbacks And Read-Only Volume Replicas

Callbacks associated with read-only volume replicas have 
traditionally been handled specially in AFS. When any file in 
an RO volume is accessed the AFS file server establishes a 
single callback promise considered to be on the entire volume. 
Any event which updates the replica (e.g., vos release) 
triggers a whole-volume callback break. The whole-volume 
callback optimization significantly reduced file server memory 
utilization, which was at a premium in 1988. However, the 
whole-volume callback is less of an optimization in OpenAFS in 
2008:

* modern AFS file servers have sufficient memory to track 
  millions of callbacks (and do track up to 1 million callbacks 
  at one site we know of, with up to 3 million callback 
  structures available)[8]

* whole-volume callback semantics require clients (and the file 
  server) to potentially expend considerable effort 
  re-establishing cache consistency, and so whole-volume 
  callbacks are necessarily a considerable protocol 
  inefficiency for sites relying heavily on AFS replication (in 
  particular, incremental replication now possible in OpenAFS)

For these reasons, we propose that the scope of extended 
callback information include notifications concerning changes 
that originate in the release of a volume. We provide the 
option for the file server to provide whole-volume or per file 
notifications, at its discretion. We provide the option for the 
file server to track client interest in specific files (ie, 
issue per-file callbacks on files in RO volumes), and speculate 
that this implementation would be preferred, but do not mandate 
it.

4.5.1 Constants

The following flag constants are added:

const AFSCB_Flag_Release = 2;

const AFSCB_IFlag_Release = 2;

const AFSCB_Release_WholeVolumeCancel = 1;

  AFSCB_Flag_Release

In an AFSExtendedCallBack instance, (Flags & 
AFSCB_Flag_Release) indicates a notification in response to the 
(possibly incremental) release of a read-only replica. 

Benjamin            November 29, 2011                  [Page 25]

  AFSCB_IFlag_Release

In an AFSXCBInvocation instance, (Flags & AFSCB_IFlag_Release) 
indicates a notification in response to the (possibly 
incremental) release of a read-only replica. 

  AFSCB_Release_WholeVolumeCancel

If additionally (ExtraFlags & AFSCB_Release_WholeVolumeCancel), 
then the callback invalidates the entire volume, otherwise it 
is a selective invalidation of just the FIDs in Fids_Array.

4.5.2 Semantic Changes

An AFS file server MAY send selective or whole-volume extended 
callback notifications. The file server MAY choose to regard 
files in RO volumes equivalently to files in RW volumes, ie, 
effectively maintain callback state on them. Alternatively it 
MAY send selective notifications on any FIDs changed, removed, 
or added in the volume without regard to client cache state. 
The AFS client must handle such notifications gracefully.

5 Security Considerations

Extended callback information messages that only invalidate 
information that may be cached at clients have equivalent 
security implications to AFS-3 callback messages. This class of 
messages includes AFSCB_Event_Cancel and probably 
AFSCB_Event_StoreData. The remaining extended callback 
information messages (most of them) contain explicit metadata 
information which could potentially be used by an attacker 
impersonating a file server to introduce malicious information 
into a client cache. Rx security extensions in development (eg, 
rxgk) include provisions for secure transmission of callback 
messages. 

  Edinburgh Consensus

Implementations should ensure that extended callbacks which 
send explicit metadata use a secure communication channel. 
Cancellation and StoreData messages may be sent over any 
channel.

6 IANA Considerations

This document makes no request of the IANA.

7 Acknowledgements 

Benjamin            November 29, 2011                  [Page 26]

Thanks to Jeffrey Altman, Tom Keiser, Jeffrey Hutzelman, 
Derrick Brashear, and Steven Jenkins for their feedback and 
suggested improvements from previous drafts. Thanks to 
participants at the 2009 Hackathon in Edinburgh and 2011 
Hackathon in Pittsburgh for their feedback and assistance.

8 Appendix A: XDR Grammar

/* Cache Manager Capability Flags */

const CLIENT_CAPABILITY_EXT_CALLBACK = 0x0002;



/* Host Tracking/Extended Information */

struct HostIdentifier {

    afsUUID ServerUuid;

    afsUUID CellUuid;

};



/* Extended Callback Information */

/* callback event types, predominantly events on the vnode for 

* which the callback is being made, but also (e.g., Deleted) 
side 

* effects of operations on related vnodes */

const AFSCB_Event_Cancel = 1;

const AFSCB_Event_StoreData = 2;

const AFSCB_Event_StoreACL = 3;

const AFSCB_Event_StoreStatus = 4;

const AFSCB_Event_CreateFile = 5;

const AFSCB_Event_MakeDir = 6;

const AFSCB_Event_Symlink = 7;

Benjamin            November 29, 2011                  [Page 27]

const AFSCB_Event_Link = 8;

const AFSCB_Event_RemoveFile = 9;

const AFSCB_Event_RemoveDir = 10;

const AFSCB_Event_Rename = 11;

const AFSCB_Event_Deleted = 12;

const AFSCB_Event_ReleaseLock = 13;



/* for use in AFSExtendedCallBack Flags */

const AFSCB_Flag_Cancel = 1;

const AFSCB_Flag_Release = 2;



/* intended for use in AFSExtendedCallBack ExtraFlags, 

 * when (flags & AFSCB_Flag_Cancel), to indicate reason for

 * cancellation */

const AFSCB_Cancel_Shutdown = 1;

const AFSCB_Cancel_CallbackGC = 2;

const AFSCB_Cancel_VolumeOffline = 3;

const AFSCB_Cancel_VolumeMoved = 4;

const AFSCB_Cancel_LostMyMind = 5; /* ran out of callbacks? */

const AFSCB_Cancel_IHateYou = 6; /* callback administratively 
revoked */



/* for use in AFSXCBInvocation Flags */

const AFSCB_IFlag_SOrigin = 1;

const AFSCB_IFlag_Release = 2;

Benjamin            November 29, 2011                  [Page 28]



/* flags intended for use in AFSExtendedCallBack ExtraFlags 

* to indicate RO volume callback events */

const AFSCB_Release_WholeVolumeCancel = 1;



/* callback result types */

const AFSCB_Result_NoResult = 1;

const AFSCB_Result_Diag = 2; 

const AFSCB_Result_Generic = 3; /* all currently defined 
messages */



/* differential status to be sent with StoreData msgs */

struct AFSCBFileStatus {

    afs_uint32 LinkCount;

    afs_uint64 ClientModTime;

};



/* differential status to be sent with directory change msgs */

struct AFSCBDirStatus {

    afs_uint32 LinkCount;

    afs_uint64 ClientModTime;

};



/* variant data types for AFSCB_Notification_Data */

struct AFSCB_Data_StoreData {

Benjamin            November 29, 2011                  [Page 29]

    afs_uint64 StoreOffset;

    afs_uint64 StoreLength;

    afs_uint64 Length;

    AFSCBFileStatus FileStatus;

};



struct AFSCB_Data_StoreStatus {

    struct AFSFetchStatus Status;

};



struct AFSCB_Data_CreateFile {

    string Name<AFSNAMEMAX>;

    AFSFid Fid;

    AFSFetchStatus FidStatus;

    AFSCBDirStatus DirStatus;

};



struct AFSCB_Data_MakeDir {

    string Name<AFSNAMEMAX>;

    AFSFid Fid;

    AFSFetchStatus FidStatus;

    AFSCBDirStatus DirStatus;

};



struct AFSCB_Data_Symlink {

Benjamin            November 29, 2011                  [Page 30]

    string Name<AFSNAMEMAX>;

    AFSFid Fid;

    string LinkContents<AFSPATHMAX>;

    AFSFetchStatus FidStatus;

    AFSCBDirStatus DirStatus;

};



struct AFSCB_Data_Link {

    string Name<AFSNAMEMAX>;

    AFSFid LinkTarget;

    AFSFetchStatus FidStatus;

    AFSCBDirStatus DirStatus;

};



struct AFSCB_Data_RemoveFile {

    string Name<AFSNAMEMAX>;

    AFSCBDirStatus DirStatus;

};



struct AFSCB_Data_RemoveDir {

    string Name<AFSNAMEMAX>;

    AFSCBDirStatus DirStatus;

};



const AFSCB_Rename_From = 1;

Benjamin            November 29, 2011                  [Page 31]

const AFSCB_Rename_To = 2;



struct AFSCB_Data_Rename {

    afs_uint32 Direction;

    string OldName<AFSNAMEMAX>;

    string NewName<AFSNAMEMAX>;

    AFSCBDirStatus FromStatus;

    AFSCBDirStatus ToStatus;

};



struct AFSCB_Data_Lock {

    afs_uint32 LockType;

};



union AFSCB_NotificationData switch (afs_uint32 Event_Type) {

case AFSCB_Event_StoreData:

    AFSCB_Data_StoreData u_store_data;

case AFSCB_Event_StoreACL:

    void;

case AFSCB_Event_StoreStatus:

    AFSCB_Data_StoreStatus u_store_status;

case AFSCB_Event_CreateFile:

    AFSCB_Data_CreateFile u_create_file;

case AFSCB_Event_MakeDir:

    AFSCB_Data_MakeDir u_make_dir;

Benjamin            November 29, 2011                  [Page 32]

case AFSCB_Event_Symlink:

    AFSCB_Data_Symlink u_symlink;

case AFSCB_Event_Link:

    AFSCB_Data_Link u_link;

case AFSCB_Event_RemoveFile:

    AFSCB_Data_RemoveFile u_remove_file;

case AFSCB_Event_RemoveDir:

    AFSCB_Data_RemoveDir u_remove_dir;

case AFSCB_Event_Rename:

    AFSCB_Data_Rename u_rename;

case AFSCB_Event_Deleted:

    void;

case AFSCB_Event_ReleaseLock:

    AFSCB_Data_Lock u_lock;

case AFSCB_Event_Cancel:

    void;

};



struct AFSExtendedCallBack {

    afs_uint32 Flags;

    afs_uint32 ExtraFlags;

    afsUUID Origin; /* originator of changes */

    afs_uint32 NCoalesced; /* calls [StoreData] combined on 
this */

    afs_uint64 DataVersion;

Benjamin            November 29, 2011                  [Page 33]

    AFSCB_NotificationData Data;

};



const AFSXCBMAX = 512;



struct AFSXCBInvocation {

    AFSFid Fid;

    afs_uint32 Flags;

    afs_uint64 lowDV; /* lowest DV at invocation */

    AFSTimestamp highDV; /* highest */

    AFSTimestamp ExpirationTime; /* new expiration, or 0 if 
unchanged */

    AFSExtendedCallBackSeq CallBacks_Array;

};



typedef AFSExtendedCallBack AFSExtendedCallBackSeq<AFSXCBMAX>;



/* Forward-looking union for callback results */

struct AFSCB_Result_Data_Generic {

    afs_int32 code;

};



union AFSCB_ResultData switch (afs_uint32 Result_Type) {

case AFSCB_Result_NoResult:

    void;

Benjamin            November 29, 2011                  [Page 34]



case AFSCB_Result_Diag:

    string msg<30>;



case AFSCB_Result_Generic:

    AFSCB_Result_Data_Generic u_generic;

};



/* extended callback result structure */

struct AFSExtendedCallBackResult {

    afs_uint32 Flags;

    afs_uint32 ExtraFlags;

    AFSCB_ResultData Data;

};



typedef AFSExtendedCallBackResult 
AFSExtendedCallBackRSeq<AFSXCBMAX>;



struct AFSXCBRInvocation {

    AFSExtendedCallBackRSeq Xcb_Result_Array;

};



typedef AFSXCBRInvocation AFSXCBRInvocationSeq<AFSXCBMAX>;



proc ExtendedCallBack(

Benjamin            November 29, 2011                  [Page 35]

    IN HostIdentifier *Server,

       AFSXCBInvocationSeq *Invocations_Array,

    OUT AFSXCBRInvocationSeq *Result_Array

) multi = 65540;

9 Informative References

References

[1] Bradner, S., "Key words for use in RFCs to Indicate 
Requirement Levels", BCP 14, RFC 2119, March 1997.

[3] Howard, J.H., Kazar, M.L., Menees, S.G., Nichols, D.A., 
Satyanarayanan, M., Sidebotham, R.N. and West, M. "Scale and 
Performance in a Distributed File System" ACM Transactions on 
Computer Systems, February 1988 

[4] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., 
Beame, C., Eisler, M., and D. Noveck, "Network File System 
(NFS) version 4 Protocol", RFC 3530, April 2003.

[5] Edward R Zayas, "AFS-3 Programmer's Reference: File 
Server/Cache Manager Interface", Transarc Corporation, 
FS-00-D162, 20th August 1991

[6] Paul J. Leach, Dilip C. Naik. A Common Internet File System 
(CIFS/1.0) Protocol 
[http://www.tools.ietf.org/html/draft-leach-cifs-v1-spec-01], 
1997.

[7] Kazar, Michael Leon, "Synchronization and Caching Issues in 
the Andrew File System," USENIX Conference Proceedings, USENIX 
Association, Berkeley, CA, Dallas Winter 1988, pages 27-36.

[8] Alistair Ferguson. OpenAFS and the Dawn of a New Era. AFS 
and Kerberos Best Practices Workshop, 2008.

[9] Trond Myklebust. Byte Range Delegations. 
[https://www3.ietf.org/proceedings/05nov/slides/nfsv4-3.pdf ], 
November 2006.

Benjamin            November 29, 2011                  [Page 36]

Author's Address

  Matt Benjamin
  Linux Box Corporation
  206 S. Fifth Ave, Ste 150
  Ann Arbor, MI 48104
  USA
  Phone: +1 734 761 4689 

  Email: matt@linuxbox.com