In this 3-part series, we discuss how cloud computing is transforming the IT landscape, as well as whether or not online backup services will become obsolete as computing moves to the cloud, and what this means to managed service providers.
In part 1, we discussed the benefits and drawbacks of cloud IaaS, and how in the cloud it is critical to expect and plan for downtime of instances, availability zones, and even multi-zone outages.
So what could be worse than down time? Data loss (or data corruption).
Cloud IaaS providers offer (at least) three types of storage:
- Object storage: This provides infinitely scalable “buckets” that allow arbitrarily sized, uniquely named “objects” to be uploaded, accessed, listed, and deleted. Data is typically replicated across several places. Data is accessed through custom APIs, so traditional filesystems (with full POSIX semantics) cannot run directly on top of object storage.
- Ephemeral block storage: This provides virtual hard disks attached to running instances that are wiped clean each time an instance reboots (or has a failure). You can (and must) run some type of filesystem on top of the block storage.
- Persistent block storage: This is similar to ephemeral block storage, except that the data is stored independently from any instance so it is preserved until explicitly deleted. Data is also stored redundantly within the same availability zone so that any single hardware failure will not cause data loss. Examples of this kind of service include Amazon EBS, OpenStack Swift, and Ceph RBD.
Effectively utilizing object storage and ephemeral block storage requires apps that are designed from the ground up to run on the cloud. If you’re running traditional IT workloads in the cloud, you’ll be working with persistent block storage services, so that’s what I’ll focus on here.
What are the considerations and risks of using persistent block storage?
- Data loss is expected: Say what? Yes, that’s right: it’s not if, but when, these volumes will fail, just like physical hard drives. Cloud providers are trying to be forthright about this reality. For example, Amazon EBS publicly gives an expected annual failure rate (AFR) of around 0.5% for an EBS volume. The developer docs for Google Compute Engine warn, “To protect against data loss, always back up your data and have data recovery policies in place.” To put this into another perspective, if you have 100 customers running in the cloud, each with 2 EBS volumes (one OS and one data), then you can expect about 1 EBS volume to fail each year.
- Silent data corruption is possible: Unless systems are engineered from the ground up to address silent data corruption from end-to-end, previously stored data and new data is at risk of being corrupted. Public clouds are no different. Amazon engineers have stated in forums that silent data corruption with EBS volumes is possible. eFolder is only one of a small handful of storage cloud providers that openly discusses and guards against silent data corruption. Certainly none of the big players are addressing this. (If you know of examples where they are, please send me details!)
- Write caching issues: Another problem is caused by how a cloud provider caches writes and flushes writes to stable storage. In a serious outage event (e.g., as in the recent June 29th AWS outage), data writes that were temporarily in flight may not be properly flushed to disk. Cloud providers are not fully disclosing how they are handling commit, cache flush, or “fsync” operations. For example, Amazon’s comments here and here indicate that volume corruption was possible because of the recent outage, and were placing any EBS volumes that had in-flight writes at the time of the failure in an “impaired” state until users manually checked (e.g., chkdsk’d) their volumes and manually resumed I/O. Even if the filesystem was OK, this could have potentially catastrophic consequences for critical applications (e.g., databases) that depend on storage properly honoring cache flushes and write ordering guarantees. It’s the same problem as if you were running a database server on a RAID controller with write caching turned on but without a battery-backed (or capacitor-backed) write cache and the power goes out. You’d have to run some application-specific integrity checks to be sure your data was good still.Guaranteeing proper write ordering and flush semantics is difficult and must be dealt with at every layer in the cloud — inside the instance OS and paravirtualized device drivers, in the hypervisor, in the hypervisor “host” OS (if relevant), in the network, in the storage processing nodes, in the storage HBAs, in the enclosure storage controllers, and in the hard drives themselves. Rather than engineer cloud services to fully guarantee data integrity, public cloud services have been engineered for performance and scale, and instead give the expectation that data loss and corruption are possible and expected to happen.
As a side note, eFolder’s cloud is engineered to hold pristine copies of backup and archival data–thus, data durability and integrity are built into the heart of all of our systems. Our private storage cloud has some deep engineering at both the hardware and software layers to address all of the above risks. eFolder’s storage cloud properly honors cache flushes, fsync ops, and write ordering requirements. We also have extensive end-to-end mechanisms to protect against silent data corruption, and we use much higher levels of storage redundancies that can safely tolerant multiple hardware failures without data loss (unlike typical persistent block storage cloud services, which only guard against single hardware failures). We haven’t had any data loss or corruption incidents in the history of the company, and we’re committed to always keeping it that way. For a brief overview, see our infrastructure page.
Back to the topic at hand… if data loss and data corruption in public clouds are expected (and many incidents of data loss and corruption have been documented already), what do cloud providers expect customers do? The larger cloud providers offer volume snapshot capabilities that allow point-in-time snapshots of persistent block storage volumes to be copied over into cloud object storage, where data is replicated across geographic regions, and the presence of multiple snapshots reduces (but not eliminates) the risk of data corruption.
Problem solved? If only it were that simple. In the final part of this series, we’ll examine the underlying risks of relying solely on snapshots, discuss non-technology-related risk factors, and present how MSPs are a critical piece making cloud technologies better solve the business challenges of end-users.
/pic9772199.png)
