
The Challenge
When Cambridge Support started working with this client, we identified that the backup infrastructure that their previous IT partner implemented was not fit for purpose. The backup jobs consisted of portable hard drives in daily rotation which were then either taken offsite or stored in a safe depending on the day. The safe itself sat next to the server rack.
There are many elements as to why there was room for improvement. Firstly, a manual rotation of hard drives on a daily basis introduces a human element whereby there is the potential for this task to be forgotten, thus overwriting the previous day’s backup if the drive wasn’t rotated. The person responsible for the backups would then need to take the disk offsite; this opens the possibility of loss or damage, and even theft.
Storing the backups in a safe next to the servers also presents a further risk to the organization. In an environmental catastrophe such as fire or flooding, it is highly probably that the server and safe (containing rotational hard drives) would be destroyed, leaving the only copy of the entire organisation’s data on a single portable hard drive for which there is no guarantee of safety or integrity.
Cambridge Support identified this configuration as sub-optimal and set out to implement a solution which would eliminate the human element by automating the backup procedure, ensure that a suitable offsite backup is held in more than one location, and build upon the backup infrastructure features by introducing virtualization and rapid restores.
The Solution
Cambridge Support recognised all the deficiencies of the existing backup infrastructure and recommended that a Datto Business Continuity and Disaster Recovery solution would be best placed to provide a backup configuration that would be reliable, resilient, and flexible.
The Datto BCDR solution consists of a Datto appliance and Datto Cloud. The appliance is a custom-built server running Datto’s proprietary software to perform backups and store them locally on the internal storage. The appliance communicates with the organisation’s servers via an agent which then performs hourly backups between 08:00 and 18:00. This ensures that changes throughout the day are backed up, not just what is left in the evening.
The final backup of the day is then replicated to Datto Cloud, a subscription provided by the vendor to store offsite recovery points in both the UK and Iceland. This meets two objectives; the backup data is stored both locally on the appliance and in a cloud storage location, and the cloud copies are stored in two geographically distant datacenters. In short, the backup exists in three locations.
The Datto solution provides an array of recovery options. Individual files can be restored if lost or deleted, entire servers can be restored from the last backup if they become corrupt, or you can virtualize your critical servers on the Datto appliance itself if your server hardware becomes unavailable. If your Datto appliance is down, it is even possible to virtualise your servers in Datto Cloud, so there are many options available with this solution.
The Outcome
Cambridge Support implemented the Datto appliance with the Datto Cloud subscription for our Thetford client in September 2020 alongside a new server infrastructure which ran perfectly fine until a hardware failure was experienced in February 2021.
A storage controller failed on the hypervisor, a virtual machine host, which meant that the client’s virtualised server infrastructure was unable to boot; only two other servers could operate as these were standalone. Once the fault had been identified, Cambridge Support instigated a virtualisation of all five virtual machines that would run on the failed hypervisor, but on the Datto appliance.
Within 2 hours, the client was able to continue operating their business even though their main server was offline. Once the servers were running on the Datto appliance, after some configuration which Cambridge Support had to perform, the hourly backups resumed on these virtualised instances.
This afforded us the time to work with the server vendor to instigate a warranty claim on the storage controller and arrange for an engineer to replace the failed component. The engineer arrived the following day, and we had the failed hypervisor running approximately 30 hours after it had failed. The client continued running on the Datto appliance until the end of the second day at which point, Cambridge Support were able to export an image of each virtualised machine that the Datto appliance had been running, and then import those back onto the failed hypervisor.
The client was up and running on their production hardware the following morning as if nothing had happened.
The Datto BCDR solution afforded the client the ability to continue running all of their servers for the business to function even with their production hardware in a failed state. Cambridge Support were able to bring up the recovery environment of five virtual machines within a 2 hour window. Cambridge Support were then able to restore the virtual machines back to production hardware outside of business hours. This brought the total outage time down from an estimated 30 hours to only 2 hours.