Distributed Internet Archive Protocol (DIAP)
draft-brasher-diap-00

Status of this Memo

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on October 4, 2008.

Abstract

A de-centralised, self-contained and managed storage protocol. A system to provide strong storage fail over by using existing resources over networks distributing vital data evenly. Rapid deployment and high redundancy for small to medium organisations as well as individuals. Designed to reduce dependency on tape backup systems. The protocol also has implications for long term archiving. By classifying data vitality values the limitations in physical space due to bandwidth constrictions can be overcome and the usefulness of DIAP maximised.

Table of Contents

1.  Introduction
2.  Architecture
3.  Prototype Design
4.  DVLT
5.  Hyper Virtual auto-changer
6.  Data Vitality
7.  DIAP Rule of Thumb
8.  Community Project and UK Trademark
9.  DPA
10.  Conclusion
11.  Security Considerations
12.  Acknowledgements
13.  Informative References
§  Author's Address
§  Intellectual Property and Copyright Statements

1.  Introduction

The system design uses FIFO Pipe queue theory. Three nodes either between sites say between offices, homes, on a campus or over WAN's, which could be dedicated to storage or used for existing services, have a round robin synchronisation of FULL - differential backup pools where the source of data ranges from a personal laptop to a file store over unused band-width where the data rate is dynamically controlled, including compression, according to load and availability. Three for simplicity and because the probability of failure beyond three is so small the extra coding to accommodate more nodes would be self-defeating. In real life use of three nodes for a DIAP pool is strong enough. Chaining together DIAP pools to extend data retention periods is a future aim of the project. Also designed, as project maturity is reached, to help reduce an organisations carbon footprint, the extent to which is unknown at this stage.

2.  Architecture

The system reduces single point of failure by creating a single FULL copy on each node at the beginning of the month the storing the differentials in a distributed manner. Use a program such as Bacula set to use a monthly FULL – differential. If a copy fails then the system will retry the next day but you loose the day of failure. Using rsync logs you can trace / track the successful copies. The copies are staggered so that each rsync copy list is made before new files are put into each directory by a few minutes. I.e. bd9-cd9 starts before ad9-bd9 and ad0-bd0 is last. Redundancy is split across three nodes, no duplicate days apart from the first FULL copy. DIAP pool is possibly equivalent to 30 tapes every month but stored at three locations. 10 days every three days, at each. You are advised to have some knowledge of the average differential size.

Slots.

Two entry points, aFull beginning of month and ad0 for the remaining days. Assuming entry points are filled during the day before the cycle begins at night 29 cron jobs split between 3 nodes. Max ad0 = LMB ((a-b or b-c or c -d) x 7 hrs) - (Ave. Diff x 9)**. Max ad0 must be greater than 2 x (Ave. Diff x 9). Max aFull = Max ad0. ** This copes with the situation when a FULL is copied at the same time the system is saturated with Diffs. LBM = Lowest Maximum Bandwidth. NB: actual max transfer will vary test so transfers are recommended for accuracy. Min DIAP space required on each node = ((max ad0 + (9 x Ave. Diff)) – max rsync logfile size. This varies according to the size of Differential – see example. Example system: (LMB x 7 hrs Ave. Diff) = 128KByte/S 1G. Max ad0 (aFull) = 24G – 9G = 15G. Max ad0 > 2 x (Ave. Diff x 9 = 18G) so ad0 so system will not fail.

Flow of data.

3.  Prototype Design

The prototype is built of several components and uses the Linux Operating system. Bash scripts are used to deploy DIAP on three POSIX user accounts using expect and ssh. Ssh certificates are setup between three POSIX accounts. A single configuration file is use to set environment variables.

The system requires a series of directories used to store the data fed into ad0 and aFull:

mkdir aFull ad0 ad1 ad2 ad3 ad4 ad5 ad6 ad7 ad8 && touch log_a

Cron jobs are used to implement table 2 architecture:

0 1 0 * * rsync -az -e ssh --timeout=1800 --numeric-ids \ --log-file=/home/diap/log_b --ignore-errors --bwlimit=128 \ ~/aFull/ diap@$IP_ADD_B:bFull

4.  DVLT

(VTL) The virtual tape Library is a device located in one location whereas DIAP enables a Distributed virtual Tape Library to exist.

5.  Hyper Virtual auto-changer

This term is derived from the term virtual auto-changer. A virtual auto-changer still requires hardware tape drives, 'Hyper' takes this one stage further by emulating the virtual auto-changer in software.

6.  Data Vitality

Data vitality is a measure of the organisation subjective view of the value of particular data types.

7.  DIAP Rule of Thumb

Observing an email archive, at 272MBytes, having never deleted an email permanently and the file, ../mail, has been in use for 4 years. During this time available xDSL line Bandwidth has increased, 2004 500MBits/sec to 1GBit/sec, 2008 1GBit/sec to 6GBits/Sec this is about a 150% yearly increase whereas the mailbox has increased yearly by about 50%. It is this difference which DIAP attempts to use classing email record as 'mission critical' - Other record types will increase at different rates, as will bandwidth depending on location, but probably less than the average yearly bandwidth increase. This idea needs expanding but forms the foundation for the usefulness of DIAP, describing a DIAP rule of thumb. DIAP can also be viewed as a technique.

8.  Community Project and UK Trademark

A community project resides at http://www.diap.org.uk to facilitate the development of working implementations. The current working prototype is released here under GPL licence rules. A UK Trade Mark has been applied for to protect the acronym DIAP for use by the wider Open Source community.

9.  DPA

DPA compliance and awareness.

10.  Conclusion

The incremental data retention tuned to the needs of an organisation so that some data is always available from any node in the backup pool quickly to within a certain time frame and perhaps tape storage stations strategically places in various secure locations for older data retention. This system would avoid using prohibitively expensive packages by reusing resources, building on Open Source technologies and have a coherent strategy across many sites increasing the level of redundancy to a high degree. A three tier strategy involving DIAP as the bottom layer, file collection uppermost and use of pre-existing mid-term infrastructure could make up a disaster recovery plan.

With layers of indexing, accounting and management facilities. An assumption is that individual file encryption the responsibility of the file owner, this does not rule out hard drive or partition encryption of individual nodes considered to reside at insecure locations. If used for these locations physical security automatic fail-safe measures to trigger archive deposits useless upon theft can be deployed. Similar fail-safe techniques deployed for attempted network security breaches. Virus scanners would be set to scan existing archives periodically and on entry to the archive pool.

11.  Security Considerations

Open root access is not recommended for SSH. Using ports other than the default 22 is advised.

12.  Acknowledgements

Thanks are due to members of Hampshire Lug, Colleagues at ECS and OMII-UK at Southampton University, inputs on this topic, including Myles McClelland, Marisa McClelland, Stephen Pelc.

13. Informative References

Author's Address

Full Copyright Statement

Copyright © The IETF Trust (2008).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an “AS IS” basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.