Internet DRAFT - draft-wolf-http-select


HTTP/1.1 200 OK
Date: Tue, 09 Apr 2002 12:13:00 GMT
Server: Apache/1.3.20 (Unix)
Last-Modified: Wed, 30 Jun 1999 16:16:47 GMT
ETag: "2e6ce5-5c0d-377a42ef"
Accept-Ranges: bytes
Content-Length: 23565
Connection: close
Content-Type: text/plain

    Internet Draft                                                K. H. Wolf
    Document: draft-wolf-http-select-00.txt               University of Ulm
    Category: Informational                                        June 1999

              Experiences with HTTP-SELECT and Asynchronous Updates

    Status of this Memo

       This document is an Internet-Draft and is in full conformance with
       all provisions of Section 10 of RFC2026 [1].

       Internet-Drafts are working documents of the Internet Engineering
       Task Force (IETF), its areas, and its working groups. Note that other
       groups may also distribute working documents as Internet-Drafts.
       Internet-Drafts are draft documents valid for a maximum of six months
       and may be updated, replaced, or obsoleted by other documents at any
       time. It is inappropriate to use Internet-Drafts as reference
       material or to cite them other than as "work in progress."

       The list of current Internet-Drafts can be accessed at
       The list of Internet-Draft Shadow Directories can be accessed at


       Many Internet services need asynchronous notifications. At the same
       time there is an increasing number of protocols which use HTTP or
       HTTP-alike request-response transactions.

       This document proposes an extension to application protocols which
       enables asynchronous notifications from server to client through
       client initiated request-response transactions. The extension applies
       the Unix select() pattern to TCP connections of HTTP-alike protocols.
       Hence it is called HTTP-SELECT.

    Structure of the Document

       The document starts by introducing the reason for this extension in
       general and in the specific case where the mechanism has been
       developed. It then gives implementation details, abstract syntax, and

    Table of Contents

    Wolf               Informational - Expires December 1999               1
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

    1. Reason .............................................................2
     1.1.  Notifications vs. Request-Response .............................2
     1.2.  The Virtual Presence Case ......................................3
    2. Implementation Details .............................................3
     2.1.  Introduction ...................................................3
     2.2.  Nameless Subscription Sets .....................................4
     2.3.  Typical Request Sequence .......................................4
    3. Extension Syntax ...................................................5
     3.1.  Basic Definitions ..............................................5
     3.2.  Messages .......................................................6
     3.3.  Example ........................................................7
     3.4.  Optmizations ...................................................9
     3.5.  Traffic .......................................................10
    4. Security Considerations ...........................................10
    5. References ........................................................10
    6. Acknowledgments ...................................................11
    7. Author's Addresses ................................................11

    1.  Reason

     1.1.  Notifications vs. Request-Response

       An increasing number of Internet services use HTTP as transport
       protocol, an HTTP-alike protocol, or another request-response based
       protocol. They have in common that TCP connections are established by
       the client to send the request from the client to the server. The
       response uses the same TCP connection from server to client. In the
       world of protected intranets and firewalls this is a requirement of
       many services.

       An increasing number of Internet services needs asynchronous
       notifications. Notifications from client to server are easily
       embedded into a client initiated request-response protocol. But
       asynchronous notifications from a server to clients do not fit well
       into such a protocol.

       Services which want to keep a strict request-response architecture
       have two options to implement server-client asynchronous
         - Notification as requests from server to client with a TCP
           connection established from the server. This is not feasible for
           services where servers are outside of the firewall.
         - Notification as a response to a (long time) outstanding request.
           This is compatible with firewalls. It is not firewall friendly,
           though not firewall hostile. However, this is the way chosen by
           this proposal.

    Wolf               Informational - Expires December 1999               2
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

       A summary of such services/protocols which need asynchronous
       notifications is available in [2].

     1.2.  The Virtual Presence Case

       The development was done while working on an Internet Presence
       protocol in combination with the Web (Virtual Presence). Internet
       Presence basically provides users with information about other users
       in a dynamic fashion. Peers subscribe for each other's online status
       and get notifications when the status changes, i.e. when a peer "goes
       online". Virtual Presence augments the user detail with information
       on the user's virtual locations (VPP [3]).

       Virtual Presence is a wide area, HTTP-related service. Hence, it must
       cope with the existing infrastructure of HTTP clients, servers,
       proxies, and firewalls. The developers decided to encapsulate VPP
       requests into HTTP to tunnel VPP through HTTP/1.1. VPP requests use
       HTTP-GET. The data is URL-encoded unless the request carries body
       data. In this case HTTP-POST is used for the body data. The
       encapsulation is strict HTTP/1.1 with request pipelining.

       VPP clients subscribe for asynchronous notifications. The server
       sends notifications when the value of objects changes. But as
       mentioned above, connection establishment from the server to the
       client is not an option. The chosen HTTP encapsulation of VPP uses
       long living TCP connections from client to server with outstanding
       (unanswered) requests. The protocol uses a Unix select-like

    2.  Implementation Details

     2.1.  Introduction

       The relevant methods of the protocol are SUBSCRIBE, SELECT, and GET.
         - The SUBSCRIBE method requests asynchronous notifications from the
           server for a single object maintained by the server.
         - The SELECT response carries the notifications.
         - GET requests the object's contents.

       Typically, a client sends a number of SUBSCRIBE requests followed by
       SELECT. Then the client waits for the SELECT response. The SELECT
       method works similar to the select() system call. The request blocks
       the connection and keeps the connection open until a timeout is
       reached or an event happens. The server responds with a set of events
       or an empty set. The client may then request the contents of these
       objects using the GET method.

    Wolf               Informational - Expires December 1999               3
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

       As in the case of Unix select(), the client creates a set of object
       identifiers for the SELECT request. The set is created by SUBSCRIBE
       requests which carry the name of the subscription set (set-id). The
       SELECT request contains the same name. It contains the set-id of the
       subscriptions for which it is supposed to return events. All events
       from subscriptions of a set are returned by a single SELECT response.

     2.2.  Nameless Subscription Sets

       The set-id may be omitted for nameless sets on a single TCP
       connection. Both SUBSCRIBE and SELECT requests MUST use the same TCP
       connection in this case. Subscriptions of nameless sets are lost
       (unsubscribed automatically) when the connection closes.

       Nameless sets do NOT work with proxies and/or HTTP requests made by
       HTTP clients implementations (Web browser, Java URLConnection), since
       this software may terminate the TCP connection at any time. They do
       work with socket connections. Implementations are available in C++,
       Java, and Delphi.

     2.3.  Typical Request Sequence

       Assumed a client wants notifications of contents changes of two
       objects (object1, object2). The client creates a subscription set
       (set1). It sends the subscriptions, a SELECT request, and waits for
       the response. If the SELECT response returns an empty set of events
       then it restarts the SELECT request. If the server returns an event,
       then the client examines the event. In the case shown here it just
       retrieves the changed contents of the object.

       C: client, S: server

       C: SUBSCRIBE request (object1, set1)
       S: SUBSCRIBE response (OK)
       C: SUBSCRIBE request (object2, set1)
       S: SUBSCRIBE response (OK)
       C: SELECT request (timeout, set1)   <--------+
       ...client waits for response...              |
       S: SELECT response ( (object1, CHANGED) )    |
       C: GET request (object1)                     |
       S: GET response (object1-data)               |
        |                                           |
        +-------------- restart SELECT -------------+

       If the client wants to cancel the subscriptions, it just sends an
       UNSUBSCRIBE request. The server first returns the SELECT response and
       then the UNSUBSCRIBE response (HTTP-pipelining).

    Wolf               Informational - Expires December 1999               4
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

       C: UNSUBSCRIBE request (object1, set1)
       S: SELECT response ()
       S: UNSUBSCRIBE response (OK)
       C: UNSUBSCRIBE request (object2, set1)
       S: UNSUBSCRIBE response (OK)

       The best practice for client implementations is to dedicate a thread
       of execution to maintain an unanswered SELECT request.
       - If the server responds, then the client evaluates the events. It
         might fetch the updated objects and finally issues another SELECT
       - If the server terminates the SELECT transaction due to a timeout
         then the client restarts the SELECT.
       - The same happens, if intermediate proxies cut the connection after
         some time.
       - Clients can issue other requests at any time, even on the SELECT-
         connection. In this case the server terminates the SELECT, responds
         to the other request, and the client automatically restarts the

    3.  Extension Syntax

       The following syntax specification uses the augmented Backus-Naur
       Form (BNF) as described in RFC-2234 [4].

       This is not a complete application protocol. It is a protocol neutral
       specification to add the SELECT mechanism to HTTP-alike application

     3.1.  Basic Definitions

       Requests consist of
         - object,
         - method,
         - request specific attributes.

       Responses consist of
         - responsecode,
         - attributes,
         - optional additional data.

       http-select-method       = SUBSCRIBE | UNSUBSCRIBE
                                  | NOTIFY | GET | PUT
       http-select-time         = HTTP-date | delta-seconds
       http-select-object       = *(VCHAR)
       http-select-reference    = *(VCHAR)
       http-select-eventtype    = UPDATED | DELETED
                                  | SUBSCRIBED | UNSUBSCRIBED
       http-select-responsecode = OK | NOT_FOUND | BAD_REQUEST

    Wolf               Informational - Expires December 1999               5
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

                                  | ACCESS_DENIED | SERVER_ERROR
       http-select-notifymode   = EVENT | DATA

     3.2.  Messages

      3.2.1. SUBSCRIBE

       SUBSCRIBE request:
        object       = http-select-object     ; REQUIRED
        method       = SUBSCRIBE              ; REQUIRED
        sub-id       = http-select-reference  ; REQUIRED (subscription id)
        timeout      = http-select-time       ; OPTIONAL (lifetime)
        eventtype    = http-select-eventtype  ; OPTIONAL (default: UPDATED)
        set-id       = http-select-reference  ; OPTIONAL (named set)
        notifymode   = http-select-notifymode ; OPTIONAL (default: EVENT)

       SUBSCRIBE response:
        responsecode = http-select-code       ; REQUIRED
        timeout      = http-select-time       ; OPTIONAL (if changed)

      3.2.2. GET

       GET request:
        object       = http-select-object     ; REQUIRED
        method       = GET                    ; REQUIRED

       GET response:
        responsecode = http-select-code       ; REQUIRED
        additional-data = http-data           ; OPTIONAL (in case of OK)

       http-data = <object contents, length, type>

      3.2.3. SELECT

       SELECT request:
        object       = "sys"                  ; REQUIRED (dummy object)
        method       = SELECT                 ; REQUIRED
        timeout      = http-select-time       ; OPTIONAL

    Wolf               Informational - Expires December 1999               6
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

        set-id       = http-select-reference  ; OPTIONAL (named set)

       SELECT response:
        responsecode = http-select-code       ; REQUIRED
        additional-data = http-select-data    ; REQUIRED

       http-select-data = *[ "<event>"
                             "<type>" http-select-eventtype "</type>"
                             "<object>" http-select-object "</object>"
                                 [ "<data>" http-data "</data>" ]
                             "</event>" ]

       The SELECT response carries events on multiple objects. If the client
       requests instantaneous delivery for UPDATE events (SUBSCRIBE
       attribute mode=data and attribute eventtype=UPDATED), then the event
       contains the object data.

       The encapsulation of multiple events with optional data into a single
       response is depends on the application protocol. VPP is based on
       ASCII object data, so XML with object data between tags is
       appropriate. Control characters are URL-encoded according to chapter
       2.2. of [5]. Other protocols might prefer MIME-multipart encoding to
       deliver multiple objects in a single SELECT response.

      3.2.4. UNSUBSCRIBE

       UNSUBSCRIBE request:
        object       = http-select-object     ; REQUIRED
        method       = UNSUBSCRIBE            ; REQUIRED
        sub-id       = http-select- reference ; REQUIRED (subscription id)
        eventtype    = http-select-eventtype  ; OPTIONAL (default: UPDATED)

       UNSUBSCRIBE response:
        responsecode = http-select-code       ; REQUIRED

     3.3.  Example

       The VPP protocol is encapsulated into HTTP. Requests are completely
       URL-encoded as query parameters (called HTTP-encapsulated). Other
       protocols might extend HTTP directly with new header fields (called
       HTTP-alike). Messages will be shown here in both variants.

    Wolf               Informational - Expires December 1999               7
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

       A leading "|" denotes a message line. Lines are separated by CRLF.
       "+" denotes a continued line.

      3.3.1. SUBSCRIBE

       Subscription request HTTP-encapsulated (adapted from VPP):
       | GET /vpp&object=
       + &method=SUBSCRIBE
       + &sub-id=subscription123
       + &eventtype=UPDATED
       + &set-id=set456
       + &notifymode=event HTTP/1.1

       Subscription request HTTP-alike:
       | SUBSCRIBE /user1/status
       | Host:
       | X-Select-sub-id: subscription123
       | X-Select-eventtype: UPDATED
       | X-Select-set-id: set456
       | X-Select-notifymode: event

      3.3.2. SELECT

       SELECT request HTTP-encapsulated (adapted from VPP):
       | GET /vpp&object=sys
       + &method=SELECT
       + &set-id= set456
       + &timeout=86400 HTTP/1.1

       SELECT request HTTP-alike:
       | SELECT sys
       | X-Select-set-id: set456
       | X-Select- timeout: 86400

       The response is delayed until the timeout is reached or one of the
       subscriptions of the set creates a notification.

       SELECT response HTTP-encapsulated (adapted from VPP):
       | HTTP/1.1 200 OK
       | Content-type: text/xml
       | <code> 200 </code><ver> 2.0 </ver>
       | <event>
       |   <type> updated </type>
       |   <object> </subject>
       | </event>
       | <event>

    Wolf               Informational - Expires December 1999               8
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

       |   <type> deleted </type>
       |   <subject> </subject>
       | </event>

       A client will then usually request the updated object:
       | GET /user1/status HTTP/1.1
       | Host:

       and restart the SELECT.

     3.4.  Optmizations

      3.4.1. Subscription Clustering

       Multiple subscriptions could be clustered and sent in a single
       SUBSCRIBE request. This reduces the traffic, but increases the
       complexity. Whether this optimization is necessary depends on the
       application and the traffic pattern.

       In the current VPP implementation, clients typically issue 4
       subscriptions. Subscription clustering is not necessary since the
       data volume of objects is much larger than the data volume of
       subscription requests.

      3.4.2. Subscription as Extended GET

       In many cases clients will need not only updates to the contents of
       an object, but also the initial value of an object. In this case the
       client will request the contents (GET) and will then SUBSCRIBE to the
       object. In order to avoid the gap between GET and SUBSCRIBE/SELECT in
       which events could be lost. Programmers tend to send the sequence
       SUBSCRIBE, SELECT, GET. However, since the server must keep the
       response order (HTTP/1.1) it terminates the SELECT to return the GET
       response. The client then automatically resends the SELECT. Actually
       the sequence becomes: SUBSCRIBE, SELECT, GET, SELECT. This gets even
       worse with more objects.

       If clients always combine SUBSCRIBE with GET, then SUBSCRIBE can be
       extended to cover the GET functionality, i.e. to return the initial
       contents of the object. In this case the typical sequence becomes
       just: SUBSCRIBE, SELECT.

      3.4.3. Instantaneous Notifications

       The basic mechanism always requires the client to fetch (GET) the
       contents after receiving the notification. The notification and the
       transfer of the object have been separated for flexibility reasons.
       But the application may require a quick delivery of the updated

    Wolf               Informational - Expires December 1999               9
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

       contents without the additional round-trip of SELECT response and GET

       For this reason the SELECT response (the notification) may carry the
       object's contents. Whether instantaneous delivery is desired by the
       client is specified in the subscription. The client adds an attribute
       (notifymode=DATA) to the SUBSCRIBE request to request instantaneous
       delivery for a single object.

     3.5.  Traffic

       The SELECT method with simple GET proved to be very useful and
       flexible. In usual scenarios the traffic generated by timeouts (empty
       responses and re-request) is negligible compared to the amount of
       object data

       Both, named subscription sets in multi-threaded clients and nameless
       sets with HTTP-pipelining on a single connection have been
       implemented. We suggest to use this method for other notification
       based protocols under similar conditions.

    4.  Security Considerations

       This document proposes an extension to application protocols. The
       proposed subscription mechanism is an extension to simple GET-access.
       Both methods involved, SUBSCRIBE and SELECT, need the same protection
       as HTTP-GET.

       Instantaneous delivery of object data by the SELECT response provides
       access to object data, if the name of the subscription set is
       disclosed. It is recommended to restrict the SELECT response to
       simple event notification and leave the transfer of object data to
       GET requests.

    5.  References

       1 Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9,
          RFC 2026, October 1996.
       2 Rohit Khare, Adam Rifkin , "Scenarios for an Internet-Scale Event
          Notification Service (ISENS)", INTERNET-DRAFT, <draft-khare-
          notification-00.txt>, Expires May 1999, November 17, 1998.
       3 K. H. Wolf, " VPP: Virtual Presence Protocol", INTERNET-DRAFT,
          Expires: July 5, 1999, <draft-wolf-vpp-01.txt>.
       4 Crocker, D. and Overell, P.(Editors), "Augmented BNF for Syntax
          Specifications: ABNF", RFC 2234, Internet Mail Consortium and
          Demon Internet Ltd., November 1997.

    Wolf               Informational - Expires December 1999              10
    INTERNET-DRAFT         Experiences with HTTP-SELECT            June 1999

       5 [URL] Berners-Lee T., Masinter L., McCahill M., "Uniform Resource
          Locators (URL)", RFC 1738, Dec 1994.

    6.  Acknowledgments

       Thanks to members of the my working group and the members of the
       CoBrow projects (EU Telematics for Research RE 1003/RE 4003) who
       helped to develop the model and implemented it.

    7.  Author's Addresses

       Klaus H. Wolf
       Distributed Systems Dept.
       University of Ulm
       89069 Ulm
       Phone: +49 (731) 502 4145

    Wolf               Informational - Expires December 1999              11