ShopTalk Blog

After writing about The Real Web OS, I was happy to see that Google is getting a lot right with Chrome OS. Naturally, it is easy to focus on what they're getting wrong, but that's another post for another day.

I am absolutely thrilled to see one long-standing feature of desktop operating systems dropped from Chrome OS: the filesystem. From the user's perspective, I see no trace of a hierarchical file system. This is great news for those of us who organize everything in our lives horizontally. We've witnessed an increasing prevalence of fast and effective desktop search in recent years, yet users are still expected to manage a tree of folders and files. That's a vestigial wart that needs to be removed.

Apple's apps generally provide a layer of abstraction on top of the filesystem, so that users don't interact directly with it. You interact with your music via iTunes; iPhoto manages your photos. They can't trust third party apps to do the same, however, so they have the catch-all Documents directory. In fact, they can't even trust themselves. Apparently the iWork team doesn't talk to the iLife team much; iWork still uses the old school model: browse the filesystem to open a file and pick a folder and a name to save a file. If they're clutching to old patterns in the name of easing the switch from Office, they are making a mistake. The best way to encourage switching is to build a new and better product, not emulate old ones. It's always better to usher in the future than cleave to the past. The iPhone is the first popular computing platform to eschew the filesystem, and, if the rumors are true, Apple's tablet will be the next; that's the future.

John Siracusa's excellent review of Mac OS X Snow Leopard contains a section about filesystem APIs. He points out some of the pitfalls and gotchas:

Mac OS X has historically supported many different ways of referring to files on disk from within an application. Plain-old paths (e.g., /Users/john/Documents/myfile) are supported at the lowest levels of the operating system. They're simple, predictable, but perhaps not such a great idea to use as the only way an application tracks files. Consider what happens if an application opens a file based on a path string, then the user moves that file somewhere else while it's still being edited. When the application is instructed to save the file, if it only has the file path to work with, it will end up creating a new file in the old location, which is almost certainly not what the user wanted.

He proceeds to discuss the various file access methods and file metadata options available to application developers. Siracusa also wrote an entire article about metadata madness. It seems simply attaching a few pieces of metadata to a file is a complicated task these days. As a programmer, I couldn't help but think that users aren't the only ones massively underserved by filesystems; we too are hampered by our only interface to persistent data storage.

John Gruber wrote a wonderful article not long ago about some of the pitfalls of filesystems. His primary example is the requirement to name a file before you can save any data to disk. He encourages application developers to solve this problem for users, but I think the issue cuts deeper than the application, right to the OS itself. After all, the application would just have to choose a name for the file if the user isn't expected to do so. Some data just can't be contained by a nice little box with an appropriate label. If we're going to provide users with a better interface to their data, why not give developers one as well?

As my college roommate once said, developers need to persist data objects that have an API other than just stat, open, seek, read, write, and close. I realize that the Unix philosophy is "everything is a file" -- which really means "everything is a stream" -- but in my applications I very rarely choose to use streams except to satisfy Unix I/O APIs. Many years immersed in Python have taught me that a few primitive types can represent most data: int, string, list, dict. I wholeheartedly agree with Scott Jackson that everything should have an API, but I don't agree that we should embrace the part of the Unix philosophy which states "Store data in flat text files." I suppose we should embrace it so long as our operating system gives us no choice, but isn't it time for modern operating systems to move on? Wouldn't it be much easier to share data between applications (which is, after all, the point of giving everything an API) if the data were just stored in sensible data structures? Instead of "my application generates a file of tab-separated columns of floating point numbers encoded as ASCII which can be parsed using regular expressions" you could say "my application builds a list of tuples; no parsing necessary". Instead of using YAML for configuration, which your application parses into dicts and lists, you could just use dicts and lists directly.

There is a ton of innovation in the database space these days. These projects stuff all of their data into a small handful of on-disk streams. They have to use the lowest common denominator API, provided by the operating system, to seek around within the stream and read and write data. It's about time for operating systems to make higher-level APIs like these first class citizens. Filesystems are no good for users and they're no good for developers. I'm starting to see the beginning of their demise, and I say good riddance.

In addition to working on ShopTalk over the summer, I joined forces with my brother-in-law to build a tree house for my kids (ages 5 and 3).

We were going to build the coolest tree house around. It was going to be 10 feet off the ground at floor level and have 120 square feet of space under roof. It was going to be big enough for even adults to stand up inside. There were going to be skylights, a trap door and a covered porch.

You've probably already guessed that we did not, in fact, build the Minimum Viable Tree House.

Like projects tend to do, the scope crept, the expenses soared and the time line stretched. As the project dragged on for weeks, the stakeholders (my kids) began to grow understandably impatient.

"Papa built our last tree house in a day!", my oldest said.

"Yeah, but that tree house was a couple pallets and a ladder", I replied.

You see, we had been working on it all that time and yet there were no tangible results for the kids. It was too high and unsafe before it had walls. Even after we put up the walls, the railing for the porch had to go up. Then there was the issue of the big hole in the floor where the trap door was going to go.

We did actually finish the tree house a couple weeks later. It's been towering over the backyard for the last few months.

Spiders think it's pretty great.

My kids though? They get more into playing in the brush pile from the tree that we cut down to make way for the tree house.

I'm pretty sure I've noticed the squirrels laughing at me.

One of the many mistakes I think I made was making the tree house too big. It comfortably accommodates adults. It's really high off the ground. Meanwhile, my kids are cramming themselves into cardboard boxes whenever they find one, squeezing themselves under the furniture and tunneling under the brush pile beneath the tree house. Kids like small spaces, and the final product doesn't fit the current clientèle. Further proof of this is their actual favorite part of the tree house - the "cramped" little loft area above the porch.

Another mistake was working for weeks and weeks before having something the kids could use. All of us were pumped up about the project at the start. Kids and adults alike sat around drawing up sketches of what it should look like and talked excitedly about what we'd add to it down the road [0]. After construction started though, it just took too long to get it to a point where they could play in it. The excitement wore off as the weeks wore on.

I still have hope that the kids will enjoy it when they are older. Maybe they'll grow into it.

For now though, when working on ShopTalk and other projects, the tree house is a reminder. A reminder to listen to and build for my audience. A reminder to deliver useful features in short iterations. A reminder that next time I need to build the Minimum Viable Tree House first [1].

[0] Be on the lookout for my next post, "The Minimum Viable Zip Line" ... just kidding!
[1] Hat tip to Eric Ries and my late father-in-law and his "two pallets and a ladder" tree house.

Our users frequently request a desktop client for ShopTalk. As with all feature requests, my response is to ask myself this question: what real need is prompting this request? They need ShopTalk to sit in their Dock. They want ShopTalk to visually and audibly notify them when new messages arrive. They want to be able to drag and drop files onto ShopTalk to share them with their coworkers. I think they want The Real Web OS. Let me tell you why.

Right now there is a huge divide in applications programming.

In one corner, we have native apps, the reigning champion. They are built with toolkits provided by the OS. They use standard interface controls and conform to a standard look and feel. They can harness the full computing power of their host machine using things like threads and systems-level languages like C. They get full access to the hard disk. They sit in the Dock or the Task Bar or the System Tray or the Menu. They can notify the user if anything interesting happens within the app. They can use sockets. You can press Alt-Tab to switch between them.

Glaring at native apps from the opposite corner is a formidable challenger called web apps. Web apps are the hippies of computer applications. Everything is free and open. They don't require any installation process or periodic updates. They give their creators nearly unlimited freedom with interface elements and look and feel. They are easily connected to the endless piles of data and vast computing power that live in The Cloud. They are easy to share with friends. They are easy to check out casually and move on if you're not interested. They can be accessed from your computer at home or your computer at work or a friend's computer or an Internet Cafe. They are built using open standards. They are taking over the world.

What does the current trend toward web apps mean? It means people want all that stuff listed in the previous paragraph. They need it. What's the current way that operating systems provide for those needs? We give people one native app, the browser, in which all of these web apps run. That browser is like the Great Wall of China. It protects the computer from these nefarious web apps, and it ensures that most of the resources within the wall are completely inaccessible.

The Real Web OS will tear down the wall.

The Real Web OS is not a suite of native-like applications that runs in the browser. It's an OS that runs web applications natively. All applications running on The Real Web OS will be developed like web apps. They will look like web apps. They will be shared and discovered like web apps. They will combine local and remote data and computing like web apps. However, they will have all of the resources of the host machine at their disposal. They will be able to use the disk and the graphics card. They will spawn threads and processes. They will receive notifications when their host goes to sleep. They will notify the user if they require attention. They will efficiently sync all of your data Dropbox-style so that you can access it from anywhere*. All of the capabilities now ascribed to native apps, which are protected by the Great Wall of The Browser, will be exposed to web apps via JavaScript APIs (for extra points, we could replace JavaScript with Python ;) ).

I read an insightful comment on Hacker News recently:

Desktop applications have shifted to the web. It's only a matter of time before mobile apps are also obsolete.

Actually, it's only a matter of time before the distinction is obsolete.

* If the Dropbox team, or anyone else, is ever game to build The Real Web OS, count me in.