Recovery point objective ('RPO')

When computers used for normal "production" business services are affected by a "Major Incident" that cannot be fixed quickly, then the Information Technology Service Continuity ("ITSC") Plan is performed, by the ITSC recovery team. This plan will always assume that the production computing equipment and the wider geographic location they normally reside at might become completely out of bounds, at an unpredictable time, without warning. The location chosen to rebuild the service (the "recovery site") must be at least 20 miles from the normal Production site and suffer no common threats with the production site (e.g. they should not be near the same coastline). The ITSC Plan must also satisfy two measurements- the Recovery Time Objective ("RTO") and the Recovery Point Objective ("RPO") for the potentially affected services. These measures are set by people who quantify what losses might ensue if the services are not available- the Business Continuity team. It is sobering to think that "potential loss of life" appears in far more IT service risk assessments than one might unthinkingly assume. The BC team specify the RTO and the RPO to be the longest time the business can allow without significant risks or significant loss- which allows system designers to specify designs that are as cost effective as the RTO and RPO will allow.

RTO and RPO are time intervals- normally expressed as some number of hours.

The RTO is the time it takes to recover the service. The events that mark the start and end of the RTO duration must be pre-agreed between Business Continuity and ITSC staff. It is best to agree to start the RTO clock at the moment when it is decided to proceed with the recovery. Sometimes too much time is taken over the decision to invoke recovery, sometimes Major Incidents do not start at easily definable wall-clock times anyway. The RTO clock should be deemed to stop once the team responsible for testing the service (before it is successfully released to the wider user community) begin work. By defining the RTO in this way it can be set to a very specific time period, which allows better decision making at all levels- accepting that this compromises a little the principle of setting the RTO to be "the amount of time the business can be without the service".

The RPO is deceptively difficult to explain. The RPO is only a measure of the maximum time period in which data might be lost if there is a Major Incident affecting an IT Service- not a direct measure of how much data might be lost. BC staff can then more easily take steps to cover this maximum period and make plans to avoid or mitigate any impact of loosing data that is entered in a time period as defined in the RPO. Consider a very simple example- a data entry clerk transfers data to an IT Service, by copy typing from paper forms. If the only consideration is RPO, the clerk needs to keep back enough recent paper forms so that he is certain to be able to retype all of them going back the same amount of time as defined in the RPO. This article does not seek to address the complexities that arise if transactions are completed electronically between organisations, and the home side of such transactions are lost because of a Major Incident.

Data Synchronisation Points

A data synchronisation point is a point in time. It is used to assess the way in which data backups relate to each other. Data backups need to be related to each other correctly when considering the time of day they were made, or their relationship to computer system activity events. A data synchronisation point is a point in time when a set of backups exist which if restored from can be synchronised to the same point in time. Often this point in time is some hours before the last backup is completed, i.e., some hours before the data synchronisation point. Backups that have no synchronisation points are generally useless.
A frequent mistake when setting RPO for traditional daily tape offsited backups is to assume 24 hours for the RPO. This mistake is the result of not considering that the RPO time begins with the start of the first data backup used in the synchronisation point; and must also include time for boxing the tapes; the inevitable contingency time that must be allowed for "waiting for courier transport"; loading and final escape from site (not always at exactly the same time of day- the RPO must be increased by an amount of time equivalent to any such variability). It is also risky to assume that tapes will always be physically intact- the RPO should include enough time to use a previous synchronisation point too.

RTO and RPO- Effects on computer system design

The RTO and RPO form part of the first specification for any IT Service. The RTO and the RPO have a very significant affect on the design of computer services and for this reason must be considered in concert with all the other major system design criteria.

When assessing the abilities of system designs to meet RPO criteria, for practical reasons, RPO is tied to the times backups are sent offsite- if for instance offsiting is on tape and only daily (still quite common), then 49 or better, 73 hours is a good time to set for the RPO so as to cover for tape hardware problems (tape failure is still too frequent, one bad tape can write off a whole daily synchronisation point). Another example- if a service is properly set up to restart from any point (data is capable of synchronisation at all times) and offsiting is via synchronous copies to an offsite mirror data storage device, then RPO is to all intents and purposes 0 hours- although it is normal to allow an hour for RPO in this circumstance to cover off any unforeseen difficulty.

If the RTO and RPO can be set to be more than 73 hours then daily backups to tapes (or other transportable media), that are then couriered on a daily basis to an offsite location, comfortably covers backup needs at a relatively low cost. Recovery can be enacted at a predetermined site. Very often this site will be one belonging to a specialist recovery company who can more cheaply provide serviced floor space and hardware as required in recovery because it manages the risks to its clients and carefully shares (or "syndicates") hardware between them, according to these risks.

If the RTO is set to 4 hours and the RPO to 1 hour, then a mirror copy of production data must be continuously maintained at the recovery site and close to dedicated recovery hardware must be available at the recovery site- hardware that is always capable of being pressed into service within 30 minutes or so. These shorter RTO and RPO settings demand a fundamentally different hardware design- which is for instance, relatively much more expensive than tape backup designs.

If very high volumes of high value transactions are to be planned for, then the production hardware can be split across two sites; with a high bandwidth network connection between the two sites constant mirroring of data can be achieved. If the user community is dispersed or at least split across two geographic areas, then the configuration is resilient to single site Major Incidents- with zero RTO and RPO being achievable, and very often little loss of service being experienced at most times of day.

RPO and RTO- a worked example

The above figure is an example of how RPO and RTO might pan out in a practical situation.
Tape is used for backup in this example. The tapes are sent offsite once per day at around the same time, but this timing is not fully guaranteed. The offsiting operation does happen to occur at roughly the same time of day in the chart above.
The daily backup offsiting tasks in this example are as follows:

• A set of backups are made to tape, possibly via a disk staging area? The synchronisation point for each set of backups is late in the backup operation in this example as several large databases have to be backed up and all of them are required for a Synchronisation Point (this is typical of such systems).
• After that the tapes have to be ejected, collated, and catalogued as they are boxed. It is often the case that offsiting operations are batched across a wide spectrum of systems at a data centre; generally the backups for all services have to wait for the very last one to be created and boxed before they can be sent to the loading bay for transport.
• Pickups by offsite data repositories are expensive. Generally a daily pickup with a reasonably priced contract will have only an approximate time for pickup and will be predicated on the data centre being ready with the tapes when the van turns up- extra pickups will be generally too expensive to contemplate on a regular basis so a data centre must build contingency time into the preparation period before the pickup is due to occur.

All of which must be done before the pickup- and all of which must be included in the RPO calculation because the synchronisation point being sent offsite depends on backups that were started very near to the start of these activities. So: a recovered service, after a restore from one of these daily backups, will be very likely to start up as at the end of the online day perhaps 13 or so hours or more, before the restored tapes were driven away from the Production data centre.

Against this background, suppose that a Major Incident occurs just before an offsiting pick up (worst case) and as always the assumption is "total site loss, instantly"- so the prepared backups never leave the site.
In this case the RPO is set to 48 hours- only twice the normal offsiting cycle. As it happens, on this occasion pickups have been regular for a while and you might make the mistake of thinking that because two offsiting operations have occurred within the RPO period noted above, you have two sets of tapes you might be able to use and still be within the RPO. This is not the case- the earlier set of tapes will produce a recovered service as at a recovery point that is much older than it needs to be to meet the 48 hour RPO. In this example perhaps 12 or 13 hours over that time.
In this example, consider the effect of the latest set of offsited tapes being rendered useless by a critically defective tape in the set (perhaps a 5-10% chance?)- as you can see by the example above, you can now NOT meet the RPO at all. Tape capacity is increasing all the time- fewer tapes mean that individual tape defects damage more backed up data.

To complete the picture, the RTO is noted above too. In this case the service was recovered well before the RTO limit was hit. It is however interesting to contemplate the fact that in this example the RTO does NOT start just after the Major Incident. In this example, as often there is in reality, there is seemingly too much delay. A quick decision to go to invocation of the ITSC Plan is always the best decision; in principle... The rule in setting an RTO should be that the RTO is the longest period of time the business can do without the IT Service in question. On the back of this appropriately economic decisions must be taken at the design stage about how the IT Service is built and run. It must be allowed however that some time has to be spent in making the decision to invoke the ITSC Plan, this decision time is an unknown variable- remember too there are often quite large sums of money spent immediately the decision to invoke is taken- staff being called in for extended periods of 24 hour working cover and large fees charged by some recovery service providers. In the example, there is the almost inevitable fudge that the RTO is set to the maximum time the business can do without the service whilst knowing full well that there is very likely to be a period of decision making before it.