Podcast #127 - Backblaze Drive Report

Brian welcomes Andy Klein to the Podcast this week. Andy is the Principle Storage Cloud Storyteller at Backblaze. In addition to the storyteller role, Andy currently serves as the Technical Marketing Director. Andy has been associated with storage and security technology for most of his career and has been with Backblaze for over 12 years.

Brian and Andy discuss storage technology and direction and how Backblaze uses and qualifies drives for use in their data centers. This did prove to be a lively discussion around storage products, but not necessarily data center storage products. Backblaze is a service adored by home users.

Andy is one of the techs behind the Backblaze-generated quarterly storage report. The annual report details failure rates for hard drives and, in a limited sense, SSDs in use at Backblaze. Backblaze is one of the best in being open and communicative about drive usage and failures.

We decided to go live with this podcast and invited our Discord to join. Of course, since it was live, ran into a bit of an issue getting started, so there had to be a restart to get back on track.

00:00 Introduction

Details from Q1 storage report
Backblaze indexes over 300k hard drives
High-level overview
Drive-farming
Drive insights delivered that are unavailable elsewhere

05:00 Operational details

16K 4TB drives
20K 1TB drives
Drives stay relevant because they work
Different drives have different failure rates
45 drive chassis
drive chassis
Densities
Cost per TB
Reference to Podcast #124: The Path to 50TB HDDs with Frickin Lasers

10:00 Procurement Process

Testing them all
No smart drives
Heavy performance penalty
Delete and rewrite
- Encouraging users to delete files
- Smart drives don’t fit the Backblaze model
- Catching the drives destined to fail
- Weeding out the drives that won’t fit the model
  - Discord question
  - Aggregating drive families or even brands
  - Every drive has a different personality
  - Inconsistent model numbers

15:00 Making it easy to track drive changes

The report is geared to making it easy to follow trends, changes, failure rates
- Firmware versions
- Configuration changes
- Skew changes
Buying the least expensive drives
Drives that arrive DOA
Let’s go to Costco or Best Buy
Oops! Banned from Costco

20:00 Transitioning to Enterprise drives

Warranties
And then voided warranties
You never get a new drive for replacement
Learning the drive history!
Using Flash
- Still buying the cheapest
- Use cases

25:00 Operational considerations

Another trip to Costco
Burning out drives BEFORE they fail
- Write, delete, write, delete, use, delete
- Kinda like a restore server
- All data goes to hard drives
- Flash drives are for short stashing
- Operationally efficient

30:00 Form factors

Discord question
- How do form factors influence purchases
- Densities can be managed with HDDs
- Influenced by environment
Adding a new tier
- Dream it, price it
- Keep it simple
HAMR
Scheduled
Pulled back

35:00 Experimental Tech

Watching and Waiting
Planned Helium drives
- Filled a small vault
- Vault=1200 drives
- Striping across 20 systems
Legacy question from Discord
Storage servers are contracted out
Supermicro servers

40:00 Protocol for failure

Cloning
Read-only servicing
Recovering data
Liquid cooling
A data center without AC
Nautilus
- Pulling water from the river
Some light banter about pulling water from the Ohio for the Cincy Lab!

45:00 Finding the right chassis

Not all Chassis are equal
Savings from water-cooled data center
Running smart software
- Servers remember what they were doing in the event of failure
- Bringing them back online
Resiliency
Telemetry

50:00 Using Solid Technology