Hard Drives are Fundamentally Broken

Hard drive technology is fundamentally broken.

The main problem with hard drives are that they haven’t been able to keep up with Moore’s Law. While computers have gotten faster and faster, hard drives seem to have topped out some time ago. At the same time, the total storage capacity of drives has continued to increase. What this means is that the performance/storage ratio has gotten significantly lower. It is so low, even, that it has started to put some significant strain on the average computer user.

Way back in 2000,  when drives were typically 20 or 30 Gigabytes, it wasn’t so bad to be able to access it at a typical 20-30 MB/s. This meant that you could copy an entire hard drive in about 17 minutes. Today, however, when drives are as large as 1 Terabyte (1,000 Gigabytes), and they can only be accessed not much faster, it can take significant parts of a day to copy a disk! Combined with the fact that most people use USB to attach their extra hard drives, and the whole operation takes even longer.

Corporations and users with more demanding needs try to get around this issue with RAID. Originally, RAID systems (Redundant Array of Inexpensive Disks) were designed to provide better reliability than individual hard drives (data is kept in multiple disks, so data isn’t lost if a disk breaks). Increasingly, however, RAID systems are being used to provide improved performance. The idea is that if you have two hard drives you can access both of them at the same time. Therefore, you can read or write twice as fast as if you only had one. Companies like EMC and NetApp take this principle to the extreme. What if you can read from 15 disks at once? Or from 1500?

Unfortunately disk performance doesn’t really scale in real life as well as it does in theory, and so  even high end RAID products don’t perform too much better. Two disks are not twice as fast as one. The more disks you add, the smaller the incremental benefit. Even if it did help more, adding disks ad infinitum is not a reasonable solution for home users.

So what’s the solution? It’s getting to the point where this is a real problem. It now takes me a few days to shuffle my data around when I get a new disk drive.

Newer technologies are on the horizon, but who knows when they’ll get to market. The ones you read about in the paper are still years from hitting the market as a finished product. Flash drives were promising, but in real life use, they seem to be the same or even worse performers than traditional disk drives.

If something doesn’t give soon, we are going to cross a critical threshold where some drastic design decisions need to be made on future PC’s, or the benefits of faster CPU’s and RAM will be hidden by the glacial speed of the system’s long term storage.

iPhone SDK: Correcting BREW and J2me

Apple seems to be getting a lot of negative press on its recent SDK announcement. Much of the criticism seems to focus around two issues: That not all functionality of the phone is accessible via the SDK, and that Apple controls the distribution method to the phone. I’ll address the second point first. Some perspective on the history of apps on cell phones will do a lot to put this in perspective, and see why these decisions were made as tradeoffs, and actually strengthen the position of the iPhone as a leader in custom applications.

For a minute, put yourself in the mindset of a developer of phone software. Arguably, having developed applications which were sold on Verizon Wireless and other cell carriers, this is a bit easier for this author. As a developer then, and attempting to receive some sort of compensation for your work, it would seem that there are too many platforms to develop for, all of them bad for different reasons.

Qualcomm’s Binary Runtime Environment for Wireless (aka BREW) is used by several major wireless carriers. From a technical perspective, it is a C-based API, which means that the learning curve is slight for C programmers. The best thing it has going for it, though, is that there is a centralized game store and market place. In theory, developers post their applications and games up to Qualcomm’s website. Carriers look through those apps and choose which ones they want to sell to their customers. Customers have one place to go to buy apps (Verizon calls it “Get It Now“). For a developer, BREW sounds like a great model. You don’t need to worry about selling to end users, or billing, or packaging up your product and selling it in a store. Customers can go on their phone and see a list of every piece of software available to purchase. If they do buy yours, they get the price of your app added to their phone bills. Nice and simple. What could be wrong with this?

Well, as it turns out, carriers aren’t interested in providing, what is to them, low margin software to their customers. They want to be able to sell games and applications (and ringtones) mostly as a way to get customers to switch to their network, and buy their cellphones. Consequently, there is little or no incentive for a carrier to decide to actually carry the game you (the developer) posted to the Qualcomm web site. This is why if you have Verizon Wireless, almost all the games available from “Get It Now” are from either from a huge game studio like EA or Sony, or based on a popular TV Show or Movie. The carriers simply don’t want to be bothered by a plethora of developers for what they consider to be chump change. As a developer, if the carrier doesn’t choose to carry you, you are out of luck. No way to get around them, no way to appeal, do not pass go, do not collect $200.

The other major platform for phone development is Java 2 Mobile Edition (AKA, J2me). This is the complete oposite end of the spectrum. Anyone can create a J2me app. When it first came out, J2me looked very promising. Like Java, which it is a subset of, software written in J2me could be run on any phone with Java support. Customers would be free to acquire software from any developer, anywhere on the planet — the carrier wouldn’t have complete control of the application pipe like in the BREW model. This would mean that developers have a much larger market of customers to sell to. Sounds like a good solution, right?

Unfortunately, while J2me was promised to be a great equalizer, this has turned out to be far from the reality. While BREW apps do require some amount of customization for each different handset it is released on, J2ME can vary even more greatly between them. Even different phone models released by the same manufacturer may not support the same J2me program! Because of the sheer number of phones and carriers which support J2me software, it is nearly impossible for a developer to write and test software on all of them. This means that any J2me application will only run on some subset of J2me phones.

Additionally, while the phone carriers cannot blockade access to their devices, developers must figure out how to get their product in front of customers. They must conduct marketing, figure out a billing model, and make sales individually to each customer. Applications are not digitally signed (as they are in BREW), so it becomes difficult for developers to prevent piracy of their product. Combined with the fact that most phones have a horrific user interface in general, and especially for installing J2me software, there are a series of significant barriers for selling J2me software which make it unpredictable to determine beforehand whether a product will succeed. This is a scenario that deters business-minded developers.

Of course, there is also the set of “Smart Phone” platforms, Palm-OS(now defunct), Windows Mobile, and Simbian. These each have their own sets of pros and cons. Certainly they have been successful targets for some developers, but for the purposes of this article we will say that the average user of those phones are typically very different from the average user of a regular phone, and specifically of an iPhone.

This brings us to the iPhone SDK. Apple seems to have derived the strengths of the business models of both BREW and J2ME. All software will be digitally signed, and distributed centrally by Apple. The digital signatures work two ways: They protect the developer from customer piracy, and they protect the customer from mischievous developers. There will be a centralized list of applications, so users can easily browse through apps they might want to download or purchase, and billing will be handled by Apple, which allows developers to concentrate on what they should be: developing. Unlike BREW, Apple has taken a stance that encourages independent developers to target the iPhone. They will place lesser-known, less expensive, or even free applications up on their store right alongside the bigger market players. Like J2ME, developers don’t need to strike a special deal with each carrier in order to get their software into people’s hands.

So, with this perspective, what are people complaining about? That they can’t write software which unlocks the iPhone. That they can’t publish software which curtails Apple’s own SDK or Safari web browser. Make no mistake about this: those complaints are pure ridiculousness. While it is to their advantage to do so, Apple didn’t have to release an SDK at all. Looking at the leading established models of software development, BREW and J2me, we can see that the Apple model takes their strengths and leaves their weaknesses — for the benefit of all 3rd party developers, and especially the independent and open source developers! This should be self-evident by looking at who the people are who are making the complaints — unfortunately, as with all things Apple, the enormous hype machine of the Interwebs has distorted the picture. Complaints are driven by …. Sun (founder of J2me, which Apple has no use for, and which will consequently suffer), Firefox (which, while a great desktop browser, wants to get into the mobile space dominated by mobile Safari) and Opera (struggling to be relevant in any market, desktop or mobile). Obviously, these complainers have motivations that are not entirely altruistic. (Note to avoid flamewar: this author is a huge desktop Firefox fan).

The second topic of complaint is that Apple won’t allow applications to run in the background, and they won’t allow voice-over-ip applications (like Skype) to run over the cell carrier (although running over Wi-Fi is fine). These really shouldn’t be the sore points they seem. From a developer’s perspective, there are certainly neat things one could do if allowed to run applications in the background (like an IM client, for example), which aren’t really practical otherwise. However, looking from a holistic perspective, some testers found that the battery would run dry in as little as four hours while running only basic background tasks. The radio and the CPU, when used actively, use a lot of power. This isn’t Apple’s fault — it’s a law of phsyics. And while I’m sure there are many people who would like to use Skype instead of their AT&T phone minutes, I’m sure the average kindergarden student can figure out why Apple won’t allow voice-over-ip apps to run over the unlimitted data connection instead of using your talk minutes.

So, what can we conclude about Apple’s SDK decisions? Certainly, they studied the existing market and the development models. The solution they came up with, from a business sense, not only takes the best of what is out there, but also meshes extremely well with Apple’s existing iTunes one-stop-shop model for how they already handle music, TV shows, and movies. While some developers may have gripes about some of the policies of the SDK (background tasks, Sun, Opera), the limitations are in actuality completely reasonable.

While the ultimate success of custom apps on the iPhone will only be determined with time, it is certainly off to a good start. As a past independant software developer, I see all of Apple’s decisions on the SDK as smart moves (even the ones that aren’t the most convenient to me), and ultimately very good to the customer, while also being reasonable, fair, and enabling opportunity for the developer. The only ones who don’t like it are the big-name established businesses which this new model will disrupt.