ShopTalk Blog

There are two types of people in the world: those that put other people into categories and those that don't. While returning to Wheeler Auditorium after lunch on Saturday, I made a comment that put me firmly in the former category. I had noticed a pattern among the speakers at Startup School. Some speakers had apparently figured out a formula for success, and emphatically encouraged us to repeat that formula. Other speakers were quick to point out that they were merely reviewing their experience, which may or may not apply to audience members, because every startup faces unique challenges.

Jamie's response to my observation was that the speakers' values were evident in their talks. Hackers value precision and accuracy. Great hackers have to be precise and accurate, because computers do what you write, not what you intend. Salesmen value confidence. To close a sale and get paid (which is, as we learned, the absolute greatest expression of love), customers have to be pretty convinced that you're right. That's not going to happen if you're not confident. Why should I believe you if you don't seem to believe yourself?

This classification reminded me of one of my favorite math jokes (from Wikipedia):

An astronomer, a physicist and a mathematician are on a train in Scotland. The astronomer looks out of the window, sees a black sheep standing in a field, and remarks, "How odd. Scottish sheep are black." "No, no, no!" says the physicist. "Only some Scottish sheep are black." The mathematician rolls his eyes at his companions' muddled thinking and says, "In Scotland, there is at least one sheep, at least one side of which looks black."

You could probably substitute a salesman, a VC, and a hacker into this joke and it would still work.

Jason Fried was the most conspicuous salesman of the bunch. He spoke almost entirely in absolutes. He raised his voice. He was defiant and confident. He paced around the stage to bolster his stage presence. And if that wasn't enough to convince you that he was right, he threw in a slew of curse words. It's hard to argue with that.

Paul Buchheit stood on the other end of the spectrum. One of his early slides literally said this:

Limited Life Experiences + Over-generalization = Advice

He was relatively timid, yet funny and insightful. He looked at problems from multiple angles. He discounted his own opinion frequently. He offered options, but didn't make any demands.

Technology entrepreneurs need to be half hacker and half salesman. When you're writing code you have to be intimately familiar with the limitations and weaknesses of your product. When you're selling, however, your product is suddenly limitless and indestructible. When you're working on strategies for your startup you need to consider multiple scenarios and outcomes. When you're talking to the press, you only discuss one possible outcome: unmitigated success.

Since Startup School is attended mostly by hackers, I felt a little insulted by Jason Fried's lack of intellectual honesty. I had to wonder if he had deluded himself or if he was trying to delude the audience. I appreciated many of his one-liners and the entertainment he provided, but I felt much more comfortable listening to Paul Buchheit's humble and honest approach.

Mark Zuckerberg is an interesting specimen. At Startup School 2007, he took the stage after Mitch Kapor. Mitch had just given a fantastic talk based on his stellar track record in the industry over the last three decades. Mark got on stage and arrogantly contradicted much of what Mitch had just said. Of course, Mark wouldn't know that, because he couldn't be bothered to sit through Mitch's talk and even showed up late for his own. Brad Fitzpatrick refers to LiveJournal as his "accidental success". In 2007, that attitude was completely absent from Mark's talk. The guy was obviously drinking his own Kool Aid. I lost a lot of respect for him that day. This year, however, he managed to gain a lot back. He attended the entire conference (and even participated in other talks from the crowd ;) ). He was much more humble and honest. He recognized the achievements of others and the viability of a variety of approaches. His first line was, "These are my people," putting himself on an equal playing field with all the hackers in attendance. I only wish he had responded to rms's question about taking PR lessons, because it seems like he's making a concerted effort in this area, and it's paying off. Good on you, Mark.

Salesmanship holds an important place at Startup School. It's likely the facet of entrepreneurship that attendees need to work on most. I appreciated Jason Fried for giving us a great example of how it's done well. I think most attendees are grounded enough to know that there is no silver bullet and that each talk should be consumed as the opinion of the speaker and not as absolute truth. From the variety of styles, experiences, and opinions, we can formulate an approach that works best for us. Thank you, Y Combinator, for putting on a terrific conference again this year, and for teaching this hacker how to become an entrepreneur.

We've been busily working on the latest iteration of ShopTalk. Our asynchronous networking library Diesel plays a key role in the application. Recently I got the chance to use Diesel outside of its normal problem domain of network applications.

I have been paging through Beautiful Code for a while now and read the chapter on Beautiful Concurrency the other day. In the chapter, Simon Peyton Jones lays out his solution for "The Santa Problem" using Haskell and Software Transactional Memory (STM). He writes that it is a difficult problem to solve with traditional concurrency primitives (threads, locks) that are provided with procedural programming languages.

The problem goes like this:

Santa repeatedly sleeps until wakened by either all of his nine reindeer, back from their holidays, or by a group of three of his ten elves. If awakened by the reindeer, he harnesses each of them to his sleigh, delivers toys with them and finally unharnesses them (allowing them to go off on holiday). If awakened by a group of elves, he shows each of the group into his study, consults with them on toy R&D and finally shows them each out (allowing them to go back to work). Santa should give priority to the reindeer in the case that there is both a group of elves and a group of reindeer waiting.

Diesel is primarily a library for writing non-blocking network applications but you can also use it to write general concurrent applications using Python generators as coroutines. I thought it would be interesting to use it to write a solution to the Santa Problem.

The solution I came up with uses a combination of shared lists (one each for reindeer and elves) and cooperative events to manage state.

import random
from diesel import Application, Loop, fire, wait, sleep

deer_group = []
elf_group = []

The lists contain the waiting reindeer and elves that eventually cause Santa to wake up when they are full (3 elves, 9 reindeer). The events let Santa communicate with the actors contained in each group. Speaking of the Jolly Old Elf, let's take a look at how he ticks:

def santa():
    while True:
        deer_ready, elves_ready = len(deer_group) == 9, len(elf_group) == 3
        if deer_ready:
            yield work_with_group('deer', deer_group, 'deliver toys')
        if elves_ready:
            yield work_with_group('elf', elf_group, 'meet in my study')
        yield sleep(random.random())

The logic for Santa is quite simple. If a group of deer or elves are ready, he works with them, otherwise he sleeps. The order of operations guarantees that if a group of deer and elves are ready at the same time the deer get priority. That is a requirement of the problem.

The reindeer and elves are created via a generic actor function that returns a generator.

def actor(name, type, group, task, max_group, max_sleep):
    def actor_event_loop():
        while True:
            yield sleep(random.random() * max_sleep)
            if len(group) < max_group:
                group.append(name)
                yield wait('%s-group-started' % type)
                print "%s %s" % (name, task)
                yield wait('%s-group-done' % type)
    return actor_event_loop

The elves and deer each "sleep" for a period of time, which actually represents time spent working (elves) or vacationing (deer). They then wake up and attempt to get into a group. They are only successful if the group isn't at capacity (max_group) already. Once in a group, each actor then waits for an event that signifies the group has been ushered in to see the big man. Elves that aren't lucky enough to gain an audience have to wait - they will continue in their loop and try again later.

Once they are signaled with the *-group-started event they do their task, which amounts to printing it to the screen in this case. They then wait until dismissed by Santa when he fires *-group-done.

While working with a group, Santa does the following:

def work_with_group(name, group, message):
    print "Ho! Ho! Ho! Let's", message
    yield fire('%s-group-started' % name)
    yield sleep(random.random() * 3)
    yield excuse_group(name, group)

def excuse_group(name, group):
    group[:] = []
    yield fire('%s-group-done' % name, True)

Santa prints a greeting to the group, and then signals them to start the task at hand. The sleep call represents the time spent cooperating on the task. Finally, the group is cleared and the actors are signaled that the group activity is done. The reindeer return to their vacations and the elves resume work.

This is all easily coordinated since Python's generators, combined with Diesel, perform cooperative multitasking. Each generator gets processing time to do its own thing and then yields control back to the Diesel core so it can wake another another generator. Any actions performed by a given generator are atomic with regard to the others. This makes working with shared structures easy.

It cuts both ways though. The actions performed by a given generator while it is active block the others. Generators should do what they have to do quickly and then yield control back to the core.

Hopefully this little diversion from the norm here at the ShopTalk blog was educational and entertaining. I know it was for me. The complete code is below. Also see here for Haskell and Erlang versions.

import random
from diesel import Application, Loop, fire, wait, sleep

deer_group = []
elf_group = []

def santa():
    while True:
        deer_ready, elves_ready = len(deer_group) == 9, len(elf_group) == 3
        if deer_ready:
            yield work_with_group('deer', deer_group, 'deliver toys')
        if elves_ready:
            yield work_with_group('elf', elf_group, 'meet in my study')
        yield sleep(random.random())

def actor(name, type, group, task, max_group, max_sleep):
    def actor_event_loop():
        while True:
            yield sleep(random.random() * max_sleep)
            if len(group) < max_group:
                group.append(name)
                yield wait('%s-group-started' % type)
                print "%s %s" % (name, task)
                yield wait('%s-group-done' % type)
    return actor_event_loop

def work_with_group(name, group, message):
    print "Ho! Ho! Ho! Let's", message
    yield fire('%s-group-started' % name)
    yield sleep(random.random() * 3)
    yield excuse_group(name, group)

def excuse_group(name, group):
    group[:] = []
    yield fire('%s-group-done' % name, True)

def main():
    app = Application()
    app.add_loop(Loop(santa))

    elf_do = "meets in study"
    for i in xrange(10):
        app.add_loop(Loop(actor("Elf %d" % i, 'elf', elf_group, elf_do, 3, 3)))

    deer_do = "delivers toys"
    for name in [
            'Dasher', 'Dancer', 'Prancer', 
            'Vixen', 'Comet', 'Cupid', 
            'Donner', 'Blitzen', 'Rudolph',
            ]:
        app.add_loop(Loop(actor(name, 'deer', deer_group, deer_do, 9, 9)))

    app.run()

if __name__ == '__main__':
    main()

Our users frequently request a desktop client for ShopTalk. As with all feature requests, my response is to ask myself this question: what real need is prompting this request? They need ShopTalk to sit in their Dock. They want ShopTalk to visually and audibly notify them when new messages arrive. They want to be able to drag and drop files onto ShopTalk to share them with their coworkers. I think they want The Real Web OS. Let me tell you why.

Right now there is a huge divide in applications programming.

In one corner, we have native apps, the reigning champion. They are built with toolkits provided by the OS. They use standard interface controls and conform to a standard look and feel. They can harness the full computing power of their host machine using things like threads and systems-level languages like C. They get full access to the hard disk. They sit in the Dock or the Task Bar or the System Tray or the Menu. They can notify the user if anything interesting happens within the app. They can use sockets. You can press Alt-Tab to switch between them.

Glaring at native apps from the opposite corner is a formidable challenger called web apps. Web apps are the hippies of computer applications. Everything is free and open. They don't require any installation process or periodic updates. They give their creators nearly unlimited freedom with interface elements and look and feel. They are easily connected to the endless piles of data and vast computing power that live in The Cloud. They are easy to share with friends. They are easy to check out casually and move on if you're not interested. They can be accessed from your computer at home or your computer at work or a friend's computer or an Internet Cafe. They are built using open standards. They are taking over the world.

What does the current trend toward web apps mean? It means people want all that stuff listed in the previous paragraph. They need it. What's the current way that operating systems provide for those needs? We give people one native app, the browser, in which all of these web apps run. That browser is like the Great Wall of China. It protects the computer from these nefarious web apps, and it ensures that most of the resources within the wall are completely inaccessible.

The Real Web OS will tear down the wall.

The Real Web OS is not a suite of native-like applications that runs in the browser. It's an OS that runs web applications natively. All applications running on The Real Web OS will be developed like web apps. They will look like web apps. They will be shared and discovered like web apps. They will combine local and remote data and computing like web apps. However, they will have all of the resources of the host machine at their disposal. They will be able to use the disk and the graphics card. They will spawn threads and processes. They will receive notifications when their host goes to sleep. They will notify the user if they require attention. They will efficiently sync all of your data Dropbox-style so that you can access it from anywhere*. All of the capabilities now ascribed to native apps, which are protected by the Great Wall of The Browser, will be exposed to web apps via JavaScript APIs (for extra points, we could replace JavaScript with Python ;) ).

I read an insightful comment on Hacker News recently:

Desktop applications have shifted to the web. It's only a matter of time before mobile apps are also obsolete.

Actually, it's only a matter of time before the distinction is obsolete.

* If the Dropbox team, or anyone else, is ever game to build The Real Web OS, count me in.

I knew that being an entrepreneur would involve becoming product-focused instead of engineering-focused. But I've spent so much of my life shaping my brain around the discipline of programming and system architecture that it's requiring a concerted effort to keep the product and its users' interests foremost in my mind. It's just so easy to fall back on old priorities and practices--especially when they're as elegant and easily validated as these particular ones are.

See, problem-solving in the realm of code usually wraps itself up into neat little packages:

1. A requirement exists
2. A system design is established/extended to support it
3. That design is implemented
4. Validation is done to ensure that the requirement has been satisfied

Make sense? Great. Now, for the sake of clarity, allow me one extended hamburger metaphor. Let's call that nice little endorphin-releasing four-step process the hamburger patty.

If only the world were all patties... no carbs! But it's not quite that simple:

1. Gather user feedback, do user studies, maybe talk to company stakeholders, weight everyone's priorities, maybe make enemies
2. Elect and spec some requirements as The Ones We're Going With Right Now
3. patty()¹
4. Deploy said requirements and desperately try to measure their effectiveness (assuming you actually figured out what their goal was in the first place)

There's a quite a bit of messy, but critical stuff in that bun. When you're all patty, you're missing a lot of the burger.²

"But I'm bought into all that!" Joe Hacker says from his cube at Big-Widget-Co. "I'm sympathetic to the product design part of the process!"

Yeah, in your regular coding gig, you might be in a few bun meetings... but if you're anything like me, you're probably sitting there thinking "arg, this crap is really gonna screw up my patty!"

Well, I'm a bun man now, Joe.³ There are no longer proxies between the user's satisfaction and my paycheck. They have the credit cards, and they don't know or care if my functions fit under one hand, if I used a recursive or imperative approach to that algorithm, or if an armadillo is frantically doing modular arithmetic in the machine room. They just want the software to satisfy some subset of their needs vastly more effectively than their existing methods. And "effective" usually has more to do with UI than OO.

The sad realization is that in most cases the code only has to be good enough to be extensible (we can add features!) and not break publicly (we won't embarrass ourselves).⁴ But the quality of the product experience has a direct impact on the company's success.

In the well-known essay The Rise of Worse is Better, Richard Gabriel argues that in commercial programming, pragmatism has for all practical purposes won the battle against purity. "The right thing" philosophy is dead or dying--take your pick.

But I'm not sure that's the whole story--it seems to me we now recognize that "the right thing" is better applied in that space where the user actually notices.