Hard drive technology is fundamentally broken.

The main problem with hard drives are that they haven’t been able to keep up with Moore’s Law. While computers have gotten faster and faster, hard drives seem to have topped out some time ago. At the same time, the total storage capacity of drives has continued to increase. What this means is that the performance/storage ratio has gotten significantly lower. It is so low, even, that it has started to put some significant strain on the average computer user.

Way back in 2000,  when drives were typically 20 or 30 Gigabytes, it wasn’t so bad to be able to access it at a typical 20-30 MB/s. This meant that you could copy an entire hard drive in about 17 minutes. Today, however, when drives are as large as 1 Terabyte (1,000 Gigabytes), and they can only be accessed not much faster, it can take significant parts of a day to copy a disk! Combined with the fact that most people use USB to attach their extra hard drives, and the whole operation takes even longer.

Corporations and users with more demanding needs try to get around this issue with RAID. Originally, RAID systems (Redundant Array of Inexpensive Disks) were designed to provide better reliability than individual hard drives (data is kept in multiple disks, so data isn’t lost if a disk breaks). Increasingly, however, RAID systems are being used to provide improved performance. The idea is that if you have two hard drives you can access both of them at the same time. Therefore, you can read or write twice as fast as if you only had one. Companies like EMC and NetApp take this principle to the extreme. What if you can read from 15 disks at once? Or from 1500?

Unfortunately disk performance doesn’t really scale in real life as well as it does in theory, and so  even high end RAID products don’t perform too much better. Two disks are not twice as fast as one. The more disks you add, the smaller the incremental benefit. Even if it did help more, adding disks ad infinitum is not a reasonable solution for home users.

So what’s the solution? It’s getting to the point where this is a real problem. It now takes me a few days to shuffle my data around when I get a new disk drive.

Newer technologies are on the horizon, but who knows when they’ll get to market. The ones you read about in the paper are still years from hitting the market as a finished product. Flash drives were promising, but in real life use, they seem to be the same or even worse performers than traditional disk drives.

If something doesn’t give soon, we are going to cross a critical threshold where some drastic design decisions need to be made on future PC’s, or the benefits of faster CPU’s and RAM will be hidden by the glacial speed of the system’s long term storage.