Broadband News

Progress report on PlusNet recovering the 700GB of lost email

An update can now be read on our forums here, which was copied over from the PlusNet UserGroup forums. The original loss of the email was covered by The Register.

The data recovery firm in the last month has been able to build partial file lists, but apparently without the directory structure from the first disk volume the task of rebuilding the data is going to be very complex. The end result being that it will not be able to restore the email. The email that was lost is believed by PlusNet to be roughly 50% spam, 48% emails that had been read, and 1-2% that was unread.

Losing 700GB of email means there may be many PlusNet customers who have lost information that cannot be replaced, but it does highlight the need to ensure that any critical information, be this access keys to software you have bought, through to family pictures emailed to you, is backed up in more than one location. Relying on your service provider or other webmail system to store all your email can lead to problems. We would have expected PlusNet to be able to restore some of the data from back-ups - but it may be their email system is run without back-ups? PlusNet's explanation of why traditional back-ups are not possible is below:

"Due to the scale of the email platform we run it is not possible to take traditional “off system” back-ups of the data. As is typical with an ISP our size, we see around 1.5million writes to the email storage each day. This rate of change leaves limited options in terms of backups, we as standard deploy our email storage in using Network Attached Storage devices capable of writing the 1.5million changes, and then replicating them in real-time to replica-offsite NAS device in our secondary data centre. As it has been well documented in this case, we were in the process of restoring the secondary unit (following firmware upgrade) when the command was typed on the primary device in error (deleting the partition). Unfortunately in this case, although we had resilience in the platform, and were taking real-time backups of data (through replication), we weren’t able to protect against the most destructive and un-predictable type of failure, human error.

Statement from PlusNet

PlusNet is now planning a new email platform that is being touted as "vastly scalable, site resilient mail storage set-up, with 6 hourly checkpoints to ensure we can roll back the majority of the changes that happen on the system within a four hour period".


Post a comment

Login Register