And all at once, my adventure in Tokyo is coming to an end.

Today, I officially joined the Gowalla team. I’ll be leaving the Land of the Rising Sun for the Lone Star State; trading ramen for ribs, karaoke for concerts, and utter linguistic befuddlement for…well, maybe I’ll hold onto that one. The big move happens after I wrap up a few things in Tokyo in the next couple weeks.

さよなら, Cerego

My time at Cerego working on Smart.fm, although brief, was an amazing experience that I will not soon forget. From the long hours we pulled in the office to the late nights out in Shin-Ōkubo, these were some of the best times I’ve had. Though Cerego is in a state of flux at the moment, there are a lot of smart, passionate people who are poised to really change the world, and I’m confident that they will do just that.

G’Day, Gowalla

It feels absolutely surreal to join the ranks of Gowalla. For years, I’d been an avid Alamofire fanboy, devouring IconBuffet, losing sleep over PackRat, and most recently, flouting acceptable social cues by obsessively checking in with Gowalla when going out with friends.

As fate would have it, I have the fortune to join at a pivotal moment for Gowalla. 2010 will be the year for geo and the social web. These are the days of Blue Sky Solutioneering™, a time of endless possibilities. All bets are off, and all eyes are on us.

That said, I am sure of one thing—
foursquare: With all due respect, you guys are toast.

Peace & Love,

Mattt

The Ebbinghaus Forgetting Curve describes the way humans retain knowledge.

When we learn something for the first time, it stays fresh in our minds for a little while. For instance, if I tell you “the Japanese word for Dog is ‘Inu’ (犬)”, you should have no problem answering when I ask you about it in the next paragraph. Ready?

What’s the Japanese word for Dog?

“Inu”. Right. Easy.

Well, if I ask you again in a few paragraphs, you may well have forgotten it. This is what the Forgetting Curve describes: As time passes, your ability to recall a particular fact diminishes until, after a while, it’s forgotten.

This inescapable decline seems pretty dismal. Don’t worry, though! The fact that you’re able to read and understand this very sentence proves that learning is, in fact, possible. (In case you forgot)

Learning is, in a way, just a process of continually not forgetting things.

We learn and remember facts by refreshing our memory from time to time, before it completely fades away. Doing this enough times makes it stick in our minds, for good.

Dog? “Inu”. Great.

There’s an art to figuring out when to review something—when to “reset” the forgetting curve. Turns out, the optimal time to do this is just as you’re about to forget it. This is the premise of a technique called spaced rehearsal.

For nearly a century, psychologists and teachers alike have demonstrated the efficacy of spaced rehearsal in the lab and in the classroom. It’s the invisible thread that connects so much of how we learn today, from flash cards to Pimsleur.

But for all of its advantages, spaced rehearsal is difficult to do by yourself. Keeping track of what you need to learn next carries a cognitive load, which only increases as the number of things you are studying grows.

Luckily, this is a problem that can be solved by software. Let computers figure out what you need to study instead! That way, you can focus on actually learning.

That’s the foundation of Smart.fm: a tool to manage your “brain bank”, allowing you to learn faster and retain knowledge longer. We all want to be more intelligent, more sophisticated, more articulate. With Smart.fm, you can get there by studying less, and have fun while you’re at it.

Over the last 6 months of working at Smart.fm, I’ve become fascinated in the way technology can hack our brains, and how it has the potential to completely transform and democratize education. A cognitive revolution is underway, and being on the front lines has been an unbelievable experience.

Some really amazing things are on the way.

Out of curiosity, do you remember how to say “dog” in Japanese?

Based on a resurgence of interest in Chroma-Hash (hi reddit!), I thought it’d be useful to revisit this oft-misunderstood project.

If you haven’t already seen it already, you should check it out. Even if you have, you might want to play around with it again to see if any new insights come to mind (go ahead, it’ll only take a minute).

Chroma-Hash has been a particularly interesting project because of the controversy it creates in discussions. Some people won’t understand how it could ever be useful, while it couldn’t be any clearer to others. There’s often a back-and-forth about the potential security risks of the system and the pragmatics of how those risks are insignificant.

I’ve learned a lot from reading through these various threads, and I wanted to share some of my thoughts to help bring light to the discussion. Or maybe just fuel the fire, perhaps.

Alright, let’s get right to it then:

So what’s The Point?

Good question! It started out as a simple UI experiment, but it soon developed into something that I think could be really useful. Here are some of the use cases that emerged from various feedback and iterations:

Use Case #1: Password Confirmation

Perhaps the most obvious use case from the demo, Chroma-Hash allows you to quickly compare the contents of two secure text fields. It’s common for a signup flow to ask you to type your password twice (to make sure you didn’t mistype it). With this visualization, a user can instantly check to see if what she typed was the same each time, without having to submit the form.

In a similar vein to password confirmation at signup, Chroma-Hash can be helpful when logging in from day-to-day. Whenever you log into your webmail or your favorite social network, you could expect to see your signature color combination. If not, you’d know that you somehow fudged it along the way. Especially for sites with 3 strike lockouts, Chroma-Hash could save a lot of needless frustration.

Use Case #2: Anti-Phishing Mechanism

As you might have caught on from the last use-case, Chroma-Hash could be effective in mitigating the risk of a phishing attack. Similar to the account-specific images that online banking systems recently added, your password becomes a visual signature that you can look for. Websites can securely serve unique color signatures by issuing a hash salt through a browser cookie, for instance.

Let’s say you go to a site that you think is PayPal. If you start to type your password and you’re getting unfamiliar colors (or no colors show up at all, for that matter), you’ll know something’s fishy.

Use Case #3: Password Strength Feedback

Let’s think back to signup flow: When creating a Google account, for instance, you’ll get visual feedback of the strength of your password as you type it in. Stop after 5 characters, and a partially-filled red bar will accompany a message telling you to pick something stronger.

Similarly, Chroma-Hash has a parameter to specify the minimum number of characters before colors start to display. Until that threshold is reached, all the user will see are boring, gray bars. It’s more implicit than using strong colors and words, but there’s something to be said about ambient feedback, no?

Use Case #4: Clipped or Constrained Input Feedback

Let’s say you’re like me, and you go a little over-the-top with passwords. Although it’s a bit of an edge case, Chroma-Hash can be useful for providing visual feedback when a user types beyond the boundaries of a field.

Conversely, if there is a cap on the maximum amount of input, the lack of visual feedback, in that colors stop changing even though you keep typing, can just as well provide a cue to stop.

Objections, Concerns, and Questions

Reading through various threads on reddit and Hacker News gave me a much deeper insight into everything from aesthetics and usability to security and information theory. A lot of feedback was immensely useful, and helped take Chroma-Hash to that next level.

There may still be some legitimate usability or security concerns, but after several iterations, I’m confident that Chroma-Hash is a robust UI component. That said, I’d love to be proven wrong so I can continue to improve it even more.

For your consideration, here are some common concerns that are raised, along with my response to them.

“MD5 Is Weak”

From my understanding, a weak hash function is exactly what makes something like MD5 well-suited to this application. Usually, a hashing function is rated on its aversion to collision. For instance, if you are taking a checksum of a file, you’d want to be confident two files with the same checksum have the same content—otherwise, no collisions.

In the case of Chroma-Hash, collisions add security. If passwords have the same checksum, it makes it harder to isolate which one a checksum represents. Collisions are good for the purposes of the visualization. Collisions are what make security possible with Chroma-Hash.

“Knowing the Colors is Knowing the Password”

Although Chroma-Hash seems to cover the entire spectrum, it’s actually constrained to a limited palette. Small palette means more collisions, which means that it’s more secure. Here are two techniques that are used to minimize the use of colors:

• Grayscale Threshold - As mentioned in Use Case #3, colors don’t show up below a specified number of characters (by default, 6). Instead, the bars display in 4-bit monochrome. Not only are the colors difficult to differentiate based on plain eyesight, but within that range, you are nearly guaranteed to have collisions. Bump up the minimum, and it becomes exponentially more difficult to trace through to the final password.

• Least-Place Crushing - Above the grayscale threshold, the range of colors are still constrained. Based on a great insight by Ian Young, the least place of a hexadecimal color can be rounded down without compromising aesthetics. For instance, the color #CE2029 is nearly the same as #C02020, but the latter could be any one of 3375 colors. Since humans can’t perceive these slight differences, the extra (leaky) detail is removed without cost. Your password still looks like “Red Green Purple” with or without that extra place.

This naturally brings us to…

“What About Colorblind People?”

Consider the last points about grayscale and constrained palettes. Although I’m not colorblind myself, I would suppose that having some form of colorblindness is comparable to these two examples. In an extreme case, in which someone could not differentiate any color at all—everything is grayscale—you could still use Chroma-Hash.

The only snag is that without perceiving the additional color information, they have a greater chance of colliding, as you see it. Because one-off input is unlikely to be consistently similar one could imagine this system as still somewhat useful (expected dark gray, black, white; got light gray, white, black: no match).

“Chroma-Evangelism” OR “The Many Colors of Chroma-Hash”

Last but not least, I wanted to point out some of the awesome contributions other developers have made to this whole experiment. They took the ideas behind Chroma-Hash and ported it to their favorite libraries and languages, and added so much more along the way. I’m truly humbled by your contributions.

• pyChroma - leegao (Lee Gao)
• YUI3 - foxxtrot (Jeff Craig)
• Prototype - wki (Wolfgang Kinkeldei)

Also, an Objective-C port and sound-based version by me:

• CHSecureTextField
• Sonic-Hash

Interested? Feel free to fork Chroma-Hash and make something even cooler!

Finally, a Shout-Out

I would be remiss without mentioning the main inspiration behind Chroma-Hash, HashMask by Chris Dary of Arc90. Cheers, Chris!

Hey, go check out the more recent blog post about Chroma-Hash for a better and more up-to-date explanation of everything.

Yesterday, I posted Chroma-Hash, an experiment in how to visualize the live-input of secure fields, such as a password on a login screen. So far, I’ve received a lot of great feedback, as well as a number of questions that I thought deserved a proper response.

Before I go into any details, I invite you to check out the live demo, (if you haven’t seen it already), so you can get a clear idea of what Chroma-Hash does.

The Concept

When you type something into a secure field, each character is displayed as a •. Good news for people who don’t want others to see their password; Bad news for anyone who has a long or difficult password (or is bad at typing). How could we improve the experience of secure text input so that the user entering information could have an idea of what they entered, without anyone else knowing it?

Chroma-Hash approaches this problem using an ambient color representation of the input as it is being typed.

Use Case 1: Login Check

If your password normally is represented as “red, purple, orange”, and after you’ve finished typing you see “pink, green, grey”, you’ll know you mistyped it somewhere along the way. This avoids a potentially long wait for the server to respond with a “failed login” notice.

Use Case 2: Password Confirmation

When you sign up for a web service, you often have to type your password twice to make sure that you entered what you wanted correctly. As in the demo, a user will be able to confirm that two fields are the same visually. There are, of course, many alternatives for live-input validation of password confirmation, but this is another viable use case for Chroma-Hash.

Security Concerns

Under the scrutiny of the sharp minds on proggit and Hacker News, it’s only natural that some really good concerns were raised. As an experiment, this is the best kind of input I could ask for, because it challenges the viability of this as a visual metaphor, and works to improve the usability of the project as a whole.

“It’s Not Non-Reversible”

By all accounts, I would not, in fact, bet on Chroma-Hash being unbreakable. At least in its current iteration.

One of the common arguments is that by showing the colors as you type, one could step through and guess along each way. chaosmachine on Hacker News explained it this way:

Because you can see the color code at each step, it’s easy to compare results very quickly, even by hand. Did I get letter 1 right? Ok, move on to letter two, try each key until the colors match the recording. Do this for each step. At most, you have to try about 64 key presses to crack each letter.

Theoretically, this is definitely a concern. One of the ways I tried to prevent this was to animate transitions between the color sequences as they were being typed, so intermediary colors were never shown. Given a sufficiently slow typist, though, all bets are off in Chroma-Hash’s current state.

Another consideration, however, is how exactly someone would be able to tell what the colors are, at least in common use-cases. As a color expressed in Hex, there are 16,777,215 possible colors for each bar. Eye-balling it wouldn’t be enough to get an exact color value—the difference between #952A08 and #952A09 is nearly imperceptible, but represent an completely different hash input. Unless you get a really good look, it would be pretty hard to tell. And at that point, you might as well be looking at the person typing it instead

MD5 is Weak

Another concern was the use of MD5 rather than a stronger hashing algorithm, like SHA-1. For this first release, MD5 was a choice of convenience. MD5 is marginally faster, and with live-validation, I wanted to make sure that the animation was smooth and didn’t interfere with user input.

One of the ideas behind the project was that by using one hashing algorithm, a user could expect the same color on any website that implemented Chroma-Hash. This is not a central concept to this visualization, and may prove to be a bad idea. For the next iteration, I’m considering adding support for SHA-1 as an alternative hashing algorithm that can be passed in as a parameter.

Iterative Improvement

Weighing in these concerns, I believe that this kind of visualization can become a viable UI metaphor—all it would take are a few minor improvements. Here are a few suggestions that I am currently considering for the next version:

Adding A Time Delay To Live Input

To avoid the possibility of exploiting information as the user types their input, a short time delay could be added before the colors are displayed.[1] [2] [3] [4] This way, all a potential attacker could know is a portion of the password’s hash, which is not nearly as useful.

Using A Stronger Hashing Algorithm

As stated before, I may add an option to use SHA-1 instead of MD5 in the next version. This is pending research into the potential gains and the concerns of maintaining a fluid user experience.[1] [2]

Using a Salt

A salt “Salt on Wikipedia”) based on other field inputs or a server-defined constant, for instance, would further increase the security of the hashing algorithm. The only downside is that this approach would force the user to remember different color combinations for each site, which as I mentioned above, may not be such a bad thing.

Once again, I’d like to thank everyone who’s given me so much useful feedback and encouragement. I’m really excited to take Chroma-Hash to the next level, and as such, I extend you to continue this conversation on possible uses and security considerations.

Photo Credit: binarycoco

We tend to think of videogames in terms of success. It’s all about rescuing the princess, getting the high score, beating the final boss, or pwning your friends. The last thing you want to think about is seeing that Game Over screen.
… That is, unless you’re a game designer, in which case the whole “Game Over” thing may be the most important detail to get right.

Consider what Will Wright, creator of such iconic games as Sim City and The Sims, has to say about the matter:

When we make games now, we very much think in terms of what are the interaction loops, and what are the success and failure sides of those interaction loops. One of the things that’s kind of non-intuitive here is that it’s actually more important to really think through the failure side than the success side. Because, when you think about it, the success side is pretty boring: you want to get to the next level. You could just spend most of your time failing, and it’s important that the failures are interesting, varied and primarily that you understand why the failure occurred.
Will Wright (2003)

Failure is an inevitable part of the gaming experience, but accounting for it is a detail too often overlooked by developers. And who could blame them? It’s much more interesting to think about cool new game mechanics or engrossing narratives. But as Will Wright points out, failure is every bit as important. Failure closes the interaction loop. No matter how interesting a game mechanic or story you have, it doesn’t matter if the player never experiences because they get frustrated and give up half-way through.

Thinking through the contingencies of failure for an interaction is an exercise of empathy with the user. Whether in videogames or more traditional UIs, framing development in a mindset of failure allows you to get in the head of the typical user and design accordingly.

Below are 2 examples of videogames that each approach failure in a poignant way. Together, they illustrate what I call 2 core mantras of designing for failure. By exploring the ways that these videogames deal with failure, we can learn a great deal about we can improve our own user experiences.

Team Fortress 2

In the overcrowded space of multiplayer first-person shooters, Team Fortress 2 really stands out. Between its innovative class system, meticulously-groomed collection of maps, and an unmatched focus on teamwork, TF2 gets a lot of things right. One of those things that I’d like to focus on is the screen you get when you die.

Being blown apart by a rocket, lit up by sub-machine gun fire, bludgeoned by a baseball bat, or being burnt to a crisp isn’t anyone’s idea of a good time. It’s a delicate matter to communicate to a user, but TF2 nails it:

There’s actually a lot going on with this screen.

First, the gameplay stops with a distinctive “whoosh” noise. With the frantic pace of a 24-person match, a break in the action affords the player a chance to step back and collect themselves. Beyond a visual cue that action has been suspended, this pause is a mental cue to reflect on what happened, so the player can understand how to not get pøwn'd next time.

Next, the camera zooms in to where your killer was when you bit the dust. This particular detail is something Valve talked about in their developer commentary: Testers sometimes had trouble understanding how or why they were killed. Especially in the case of a Sniper’s headshot from the other side of the level, for instance, it would be very frustrating to drop dead out of without adequate explanation. Thus, Valve settled on the aforementioned solution, to the chagrin of campers and griefers everywhere.

One of the most unique parts of this death screen is the “On the Bright Side” message that pops up from time to time. Just as zooming in to your killer added context of how you were killed, “On the Bright Side” adds context to your death itself. For instance, “You tied your previous record for kills this round of 4”, or “You stayed alive as Scout longer that round than your previous best”. As it were, this message recontextualizes death from failure to a form of success. Rather than be upset that you died again, you get recognized for things you did particularly well this time around.

Our insights from Team Fortress 2 leads us to the first mantra of Failure:

When we fail, much of the pain we feel derives from a lingering sense of confusion. “What the heck? Why did that happen?” Without properly communicating the situation, the player is unable to causally link their actions to the response. This can quickly lead to frustration, and ultimately cause the player to give up. When we fail, it’s not that we’re necessarily upset with failing itself—after all, in videogames, the stakes are pretty low. What’s really upsetting is when, because of a broken feedback cycle, we don’t seem in control of our own destiny.

World of Goo

Fueled by its critical reception and widespread distribution, World of Goo has become an iconic success story for the emerging generation of indie games. But more importantly, it shows how a strong concept combined with creativity and a great attention to detail can produce an amazing experience.

Each level of World of Goo challenges the player to build structures in order to reach a goal. How the player does it is up to them.

Game mechanics are introduced with new varieties of Goo Balls, like Ivy Goo, Balloon Goo, and Bomb Goo. These game mechanics are reinforced through different puzzles that explore the intricate ways that Goo Balls can combine and interact. Levels incorporate obstacles, like spike-lined chasms, mechanical automata, and hurricane-strength winds. All of these elements come together to keep gameplay varied and interesting, and in turn, make the player come up with new and creative strategies.

Sometimes, the solution is obvious, and it’s just a matter of getting there the fastest or with the fewest Goo Balls. Other times, things aren’t as clear, and it may take several iterations just to find something that works at all. Perhaps the worst, though, are the times when it’s clear exactly what needs to be done, but you just don’t have the finesse to get your goo tower to balance the right way.

For how challenging the game is, though, it never seems overly frustrating. Why this is has a lot to do with the way World of Goo mitigates failure.

In World of Goo, there is no Game Over screen. Rather, the notion of failure is recognized as being very distinct from Game Over. This touches on a fundamental truth of problem solving—that it is an iterative process; a process of constant failure, but also of constant learning. To expect the player to get it right every time undermines the spirit of the whole puzzle genre. If the player isn’t failing, it’s not really a puzzle, now is it?

Even though failure is inevitable, and indeed important, it’s important to recognize that not all failures are equal. For instance, certain levels in World of Goo have white “undo bugs” flying around on the screen’s periphery. Activating one reverts the last thing you did, making a particularly disastrous move not force a full restart. To that point of all failures not being equal, “undo bugs” allow for an acceptable level of inevitable human error to occur without taking away from the core gameplay experience.

All of this brings us to the second mantra of failure:

It’s like that archetypical High School French teacher—the one who corrects every last word you say, and then makes you repeat the whole sentence again, once you’ve painfully managed to eek it out. Don’t be that High School French teacher. If not just for the fact that such an interaction isn’t so much fun, they’re also not that effective. Going back to the first mantra of failure, provide the player with enough context to understand what they’re doing wrong, and they’ll often correct themselves.

Of course, there are many more things to be said about failure. This is only a start to a conversation that I extend into the ethos of everyone interested in making user interactions better.

For a deeper appreciation of the role of failure in game design, be sure to check out the work of Jesper Juul, particularly his essay, “Fear of Failing? The Many Meanings of Difficulty in Video Games”. Other suggested reading includes a Gaming World article on difficulty and failure (which also cites Juul) and A Wired article by Clive Thompson on Peggle.

Earlier in the year, as is the tradition at Carnegie Mellon, there was an open contest to be the class speaker at commencement. As someone who never identified strongly as a student qua student, I knew my submission would be a long shot. It was. But how much better to have tried and failed.

So for what it’s worth, here’s my parting message to all of you (us) graduating seniors.

Fellow members of the Class of 2009, I stand here before you to talk about a very important subject: hot dogs.

Well, not hot dogs, per se, but a story about hot dogs, which, as you might expect from a graduation speech, has very little to do with hot dogs.

This story takes place at the Vienna Sausage Company’s factory on the North Side of Chicago. In 1970, the company completed construction on this building, to replace the original plant on the South Side: a sprawling mass of inter-connected properties covering an entire city block. It had been built up incrementally over the company’s 70-year history. This new plant was designed from the ground-up to streamline the manufacturing process into a single, state-of-the-art facility.

When the plant finally opened, the first batches of the company’s signature product—their “natural-casing, old-world, hickory-smoked sausages” weren’t coming out right. They tasted fine, but they didn’t have the right snap when you bit into them. Even worse, the color was wrong. Rather than a distinctive bright red that had defined the brand for so long, these new batches were pink.

So, over the next 2 years, the Vienna Sausage Company did everything they could to figure out what was wrong. The ingredients were all the same. The process was all the same. Maybe the ovens were cooking differently? Maybe the water on the North Side of Chicago wasn’t the same as the South Side? After all this time, no one had any idea what could possibly be missing.

Then, one night, a few workers were out reminiscing on their days in the old plant, when someone mentioned Irving. Irving was the kind of guy that had been there forever. He knew everyone there; had nicknames for everyone. Listen to what he did: his job was to take racks of sausages from refrigeration to the ovens. Though this could take as long as 30 minutes, as he had to wind through the maze of hallways and buildings of the old plant to get there. He would go through the warm hanging benches for the pastrami, through the boiler-room, next to the tanks where they cooked the corned beef, sometimes even up an elevator, until he finally got to the smokehouse.

In this new plant, there was no Irving—he didn’t want to commute from the South Side to the new plant. And his long journey to the smokehouse was missing too—in this new facility, there just wasn’t any need for it. As it turns out, Irving’s trip, which gradually warmed the hot dogs before being cooked in the smokehouse: that was the secret ingredient. So secret, that not even the company itself knew it.

I originally heard this story from Ira Glass, on an old episode of a radio program called This American Life. And the reason I found it so compelling was his take on it:

“What I like about the story is the fact that these guys at the factory had done everything right. Finally built their dream factory, with the best equipment and expertise that money could buy. But you can’t think of everything”

“Sometimes, you have no idea why you were a success in the first place.”

That’s something you don’t really hear enough. All too often, we try to simplify things into easy-to-remember formulas, that either equate success as a function of how hard you work, or conversely, as a matter of blind luck.

Graduation is a big milestone in life, certainly. It’s one among many that you use to fold up your personal story into neat little episodes.

It’s that interface where one chapter wraps-up and another begins. Where our only choice is whether to reflect back or plan ahead.

I ask you today, to consider your own messy factory that you’ve unwittingly built-up over the last 20-some years. At milestones such as this, there is a tendency to want to reinvent yourself; to move across town, into a brand-new facility with everything meticulously designed to absolute perfection. And there’s no way to stop this—it’s human nature. Really, there’s no reason to either: sometimes it’s worth thinking about starting fresh with a clean slate. In fact, probably most of us will do just this to varying degrees, as we move across this country, and around the world, where our new identities await us. But as you stand there, with your clean slate in hand, ready to build “the perfect you”, remember this:

“Sometimes, you have no idea why you were a success in the first place.”

Look deep inside yourself, and you might begin to see where Irving is. He might be in that random elective you took on a whim that turned you onto something completely new and different. He might be in your renewed sense of hygiene after you transferred out of Computer Science. He might be in a very special person you met when you least expected it, who completely changed your life.

I ask you today, to look deep inside yourself, on every level, with the resolve to know yourself as well as you possibly can.

With luck, you may meet a fellow named Irving, who just so happens to be the very reason why you’re here today, graduating from Carnegie Mellon University.

URL shorteners are quite the topic of conversation following the announcement of the DiggBar last week—marking Digg’s impressive entry into this crowded and contentious field.

Back in the day, there was TinyURL, which turned that interesting web page you found into something you could easily slip into conversation, or confidently paste into an e-mail. But with the advent of Twitter, a whole ecosystem has sprung up to make sure you’re able to milk every last character, whether by clever uses of exotic TLDs” or recently, unicode characters.

Yet, for the convenience these services provide, there is a hidden cost to this transaction, as Joshua Schachter points out in his recent article:

The worst problem is that shortening services add another layer of indirection to an already creaky system. A regular hyperlink implicates a browser, its DNS resolver, the publisher’s DNS server, and the publisher’s website. With a shortening service, you’re adding something that acts like a third DNS resolver, except one that is assembled out of unvetted PHP and MySQL, without the benevolent oversight of luminaries like Dan Kaminsky and St. Postel.

If these shortener services were to suddenly vanish, their links would go with them. Sure, a majority of those links are of no greater significance than your average lolcat, but 5, 10 years down the line, we face losing a lot of meaningful content. It poses enough of a problem that there is now a project to systematically crawl all of these services to archive and preserve their links.

As users, we are left without much recourse: most SEO junkies and REST purists are pretty set in their ways of descriptive URLs, and Twitter’s core philosophy revolves around 140 characters—so that’s not changing anytime soon (short of Kottke’s proposed solutions). Fortunately, for those of us who make websites, we have the opportunity to roll our own solution, and reclaim our URLs.

DNS Diet

So you’re all gung-ho about preventing the link-rot apocolypse of the internet. Sweet! Now what?

For now, home-grown, site-specific short URLs seem to be the most sustainable solution. Not only do they reinforce your domain in the minds of the end-user, but they keep a closed loop on valuable analytics. And as long as you make sure to redirect intelligently, you can build it right on top of your existing site without sacrificing SEO.

Every website has its fair share of quirks, but if Amazon, a website rife with long, impenetrable URLs can make amazon.com/wii work as expected, surely you can implement something for your own blog or web application. For a starting point, you might want to check out something I made recently.

Midgt: Well, Here’s One Way of Doing It.

Midgt is a Ruby implementation of a simple, reversible algorithm that encodes numeric IDs into mixed-case alpha-numeric sequences that can be used as a drop-in component in web applications.

With some creative routing, you could change this:

http://ecommer.ce/catalog/products/94573

Into something like this:

http://ecommer.ce/nBo

The ID 94573 can alternatively be encoded to the shorter string, nBo. Custom routing can be used to trim the verbose hierarchy built around the ID down to a shorter alpha-numeric ID, which redirects back to it to it’s proper listing.

This example is admittedly contrived, but the utility of such an approach increases dramatically as the range of legal characters used to encode is expanded, and the number of unique IDs to represent continues to grow. For applications operating in terms of millions of items, this could mean 3 or 4 less characters. Since every character matters in this 140-count economy, why not? Besides, transparent numeric IDs are so 2006.

Check out Midgt for yourself on Github:

http://github.com/mattt/midgt/

Nothing too revolutionary, but I hope it’s a bit of inspiration for those of you now wanting to roll your own solution.

A wise man once told me that if you train an n-gram model for generation with too much data, it will hurt. Bad.

We’re talking Kurzweilian singularity ➡ grey goo ➡ ??? ➡ profit! kind of hurt.

That’s the way I’ve felt over the last month or so, thinking about my thesis; there were so many directions I could go in, so many theoretically intriguing and clever avenues to venture.

In order to head off my progressive channeling of Don Giovanni, I got back to fundamentals, and rediscovered what it really meant to be a haiku.

俳句 – haiku

“Ignorance of other cultures is the currency of ours.”

As much a law of the universe as entropy, any genuine artifact of a culture will inevitably become completely bastardized. Much the same fate has befallen the noble haiku, now distilled to a metrical exercise of 5-7-5. (Then again, who am I to judge, like I of all people could bemoan the loss of 17th-century Japanese culture). What appealed to me from traditions lost were two missing pieces— kigo and caesura —that come together to further constrain the project, making the problem at hand more well-defined.

季語 – kigo (season word)

梅がかに　／　ノット比の出る　／　山路仮名

scent of plum blossoms
on the misty mountain path
a big rising sun

Traditional Japanese haiku have this remarkable ability to evoke a sense of place, as in this classic example from Bashō. In a post-modern context, though, such naturalistic references might best be supplanted by a pop-culture reference. Here’s my take:

the earth is crying
our only hope is Al Gore
it’s gettin hott in hrrrr

休止 – caesura

spring rain —
small shells on a small beach
glittering

ボックスの話 – Natural Language Processing

sunny cheerful day -
gloomy nimbus on Wall Street
today as stocks

Stranger still, after generating this, my program decided to move to New York City to make it big, where it received mixed reviews on his interludes to the oppression of the working-man’s clock cycles.

Introducing the Cast

Keats is composed of several different NLP modules, each optimized for different tasks, which are combined using some pretty slick Ruby glue code. Here are the components used at the moment:

CMU Pronouncing Dictionary

As I mentioned before, the CMU Pronouncing Dictionary was the technical inspiration for my thesis project in the first place. It offers both phonological and prosodic information on tens of thousands of words. I indexed the flat-file into a MySQL database, which cached calculated information like number of syllables and vowel geometry, along with a delightful IPA string representation.

WordNet

Perhaps one of the best-known NLP resources out there, Princeton’s WordNet” is an semantic index of staggering breadth and depth. Not only does it have hundreds-of-thousands of entries, but for each entry, there is an extensive relational graph for everything from synonymy to verb frame. In order to integrate it into my project, I wrote a script to transform the Prolog database into a MySQL relational database, and cross reference it with the existing phonological information from the CMU Dictionary.

PCFG

After entertaining some ideas about language model training on a corpus of haiku, I decided the most reasonable way to approach it, at least for now, was with my trusted friend, the Probabilistic Context-Free Grammar. To get a feel for what it could do, I hand-wrote a series of transformation rules like S ➡ Adj N V or S ➡ Adj Adj N V N. From there, I would partition syllables for the line and find a candidate word from the WordNet SynSets.

n-Gram Modeling

Building off a small sample of a New York Times text corpus, I started to work with trigram and larger n-gram language models to produce meaningful (or at least reasonable) sequences between target keywords. This is what generated “on Wall Street / today as stocks” in the example poem. There was a rule in the PCFG that replaced M with this Markov model output.

All Together Now

Given the kigo, or theme word, I look up all of the semantic pairs associated with it in WordNet, whose correspondent has an entry in the CMU Pronunciation dictionary. After ranking the candidates (now randomly, but later by phonological features), it will partition syllables and generate a PCFG for a sequence before and a sequence after the split. Using the candidate rankings, it will fill in the slots as necessary, until the grammar is satisfied and the poem is complete.

At this point, I’m feeling pretty good about my progress so far. After some fine-tuning and figuring out better ways to rank candidate words, I should have a pretty robust engine that will produce not only passable haiku, but with slight modification, Fibonacci poems as well. As for the Limerick part of the project, I’m hoping that something will click in the next month, so I’ll have some idea of where to start with that.

There’s a sort of listlessness when you exist between lives. I guess you can only live life in one of two states: content with eternity, or bracing for a change that sweeps over you like a lazy, salty tide. You can talk to permanent fixtures of the universe—champions of their own unique set of mannerisms & affectations, or if you’re unlucky (as I am) you’re talking to ghosts, just waiting for the last anything (everything).

Perhaps it’s the presence of disorientation that allows me to explore. I wonder how different that really makes me, anyway. The weeks build upon each other, and the sins of monotony, of convenience, of routine, become unbearable temptations. The air ceases to carry that unfamiliar electricity.
The roads cease to branch
It’s a game of how long you can pretend to embrace the frightening beauty of chance,

Riding on the Caltrain enumerates the untraveled possibilities of my former voyeurism.

Questions of “what if” replaced by “why not”. Yet their vapor trails cast a shadow of phony, constructed illusions. Of what it was actually like to toss caution to the wind and venture to the Hillsdale stop and visit the racetrack. These un-lived carcasses of imagination hurt the most. As if their vacuous lies actualize their emptiness deep in your gut. It teases you with the prospect of making up for missed time. What a sorry trap.

June 1st marks the beginning of the next big chapter in my life. Just 15 days after graduation, I’ll officially begin work with Cerego, in the heart of Shibuya, Tokyo. Cerego just yesterday launched a major brand and site overhaul with smart.fm, the premiere learning resource on the internets. (As it were, I held off on an official post like this until the redesign hit, if nothing else than to avoid confusion).

Although I expected to land somewhere in the bay area, my new digs in Tokyo felt like it was destined from the start. I mean, the job posting advertised for a Rails developer with hacker aspirations to work on a product that was rooted in linguistics. And as if that wasn’t enough, the offices were in Shibuya, which not a few hours earlier, I had looked up on Wikipedia to satisfy my curiosity after playing The World Ends With You, a videogame set in a virtual rendition of Shibuya. On my first visit, this would prove to be surprisingly useful: I can’t describe how weird it was to think “Okay, here’s that Moyai head, so that means Scramble Crossing and Hachikōō are coming up” and actually be right. Thanks Square-Enix!

One of the first things people ask when I tell them about my new job is, “Do you speak Japanese?”

To this, I usually respond with a combination of furtive glances and an awkward smile, speaking to the fact that I, in fact, cannot speak Japanese, as well as my understanding that this is not exactly ideal for someone about to live in Japan. I could be worse off, though. I do, after all, study Linguistics and I’ll be working for a company that teaches Japanese. Nonetheless, there’s still a part of me that regrets my decision against taking Japanese in high school, because it was too オタク for my tastes.

Moving to Tokyo involves packing up everything I own into a couple suitcases and leaving everything else behind. To be honest, the whole process has this eerie religious undertone to it: selling all of my earthly possessions, making amends with friends and family, and whatnot.

I have an entire bucket list of things to do before I leave, including an epic midnight bike tour of all of the places I’ve ever lived—retracing fragments of South Fayette and exploring uncharted areas around Oakland. A sad consequence of this preparation are the empty checkboxes next to the names of all of my friends and family, placed there with the full understanding that with each check, I prepare myself for the possibility of that being the last time I ever see them again. And worse, leaving here means leaving behind someone very special, at least for a while. At moments, I can’t help but feel frozen in time, in the eternal now of self-reflection.

At heart, I’m a sort of romantic fatalist: things will happen as they will—no matter what—and it usually turns out alright. How wonderful it is to be a part of something so amazing. </waxing-philosophic>

June 1st marks the first day of my new life, and I couldn’t be more excited! This is the opportunity of a lifetime, and I’m ready to start kicking some ass. So to all my future colleagues in Team smart.fm: I’ll see you soon.
And to all of my friends: let’s go out for a drink.

- マタタ

Whether it’s Half Life, Team Fortress 2, Portal, or, most recently, Left 4 Dead, Valve has an attention to detail that is unsurpassed among game developers. If it wasn’t for their commitment to opening up their development process, such as through their brilliantly-conceived in-game developer commentary, I’d have no recourse but to conclude that these games were delivered on high from His Noodley Goodness.

Thankfully, their efforts have not gone unnoticed, given their commercial success as well as strong critical acclaim from game journalists and bloggers alike. Since the release of Left 4 Dead in November 2008, there’s been an steady stream of articles about the subtle touches that made for a such a unique gameplay experience. One particularly awesome piece by John Brownlee details how the 5 minute intro video communicates all of the important gameplay dynamics without the player even noticing. Brilliant.

Reading this article got me thinking about what good ideas more conventional software developers could absorb from Valve. What I found particularly intriguing was L4D’s achievement system as a way to explore its potential in the context of the social web.

Mind you, achievement systems are not particularly novel. Perhaps the most ubiquitous example is XBox Live, wherein players unlock achievements by meeting conditions in a particular game, thereby earning trophy icons and Gamerscore points. These accomplishments are usually along the lines of “complete the game on hard difficulty” or “score 100,000 points”. Valve’s implementation (viz Steam) in particular, though, is the first to really use achievements effectively.

In particular, I’ve identified 4 primary uses of the achievement framework: to be Instructive, to be Prescriptive, to be Demarcative, and to act as Incentive. For each of these, I’ll explore a canonical example from the Left 4 Dead achievements, and tie it all together with microformats. Yes, microformats. Excited yet? Just so you know, there’s an achievement to earn by getting to the end of this article.

Left 4 Uses of Achievements

Instructive

All FPS’s are pretty much the same on the surface:
W-A-S-D, Mouse to look and aim, Click to shoot, R to reload, Numbers for weapons, Space to jump.
Play one and you’ve played them all.

This may not be a bad thing in itself, but it does present some design issues. For instance, how do you tell players about something new? No one reads instruction manuals, complex controller maps on loading screen are obnoxious, and in-game tutorials are really awkward in a story’s context (“Wow, an upgrade! Now I can press B to fire missiles!”)

One such hidden ability is that you can instantly kill an infected by sneaking up and melee-attacking them from behind. As you might expect, melee-ing an enemy from behind isn’t a very standard idiom. Not like a Head Shot, at least. So how did I learn to do that? Well, there just so happens to be an accomplishment called “Spinal Tap”, which describes this exact situation.

As obscure as this might seem, buried in an achievements list and whatnot, according to Valve’s stat tracking at the time of this writing, over 60% of users have discovered this on their own, perhaps many of them because of this being an accomplishment.

Achievements like “Spinal Tap” promote exploration by creating an invitation for users to try new things. By learning just this one new thing, a user will be more compelled to step out of their common habits to discover something new for themselves.

Prescriptive

What separates Left 4 Dead from pretty much any other multiplayer game is the intense focus on small-group cooperation. Valve took a huge risk with this too: if it hadn’t nailed that game dynamic, the whole game would have failed. As the central design philosophy for the game, everything comes back to cooperation in Left 4 Dead. Pull a Leeroy Jenkins and fight the horde on your own, and don’t be surprised when a hunter starts feasting on your innards.

So consider another achievement in Left 4 Dead:

When you’ven been rattled by zombies to the point of near-collapse, it’s difficult to even consider healing someone else before you. However, achievements such as “Dead Giveaway” exists to prescribe such selfless actions in order to reward cooperation and cohesion within the group. Subtle rewards like this stress the importance of teamwork and empathy—just as Valve had in mind.

Demarcative

A core facet of our human nature is the importance of understanding one’s place in the world. For platformer and adventure games, like Super Mario World, the world map provides a sense of the vastness of the game, and your progress through it. RPGs use highly-developed plot in conjunction with leveling systems to contrast how far you’ve come since the beginning of the game. Puzzle games taunt you with high-scores.

Left 4 Dead, if you didn’t have achievements, wouldn’t have a strong sense of place on its own. You could spend hundreds of hours blasting through each campaign, and be right back where you started.

Of course, this isn’t the case.

A strong cast of player achievements serve as a permanent record of where your time has gone, and how far your accomplishments reach. You get an achievement for playing through any of the campaigns once, like “Toll Collector” for completing the Death Toll campaign. You get one for beating them on expert too. Even though you spent hours needlessly earning each of them, when you certainly had better things to be doing, at least you have nothing to show for it.

Incentive

One thing is for sure, given fads and trends that pop in and out of existence: people love to collect shit.

The whole concept of an achievements system is based upon this core premise. Present a player with an empty trophy case, and they’ll spend hours upon hours scouring the game world to fill it up. A game might only last a few hours, but the meta-game is, or can be, eternal.

Want to earn all of the achievements in Left 4 Dead? You’ll have to kill 53,595 infected first. To put that in perspective, I average a couple hundred on a normal campaign, each of which takes about an hour. For the average player, this accomplishment will might take 100 hours to complete. What’s amazing is that about 3% players have done it. That’s some dedication.

Left 4 Relevance?

So to recap, here’s what Valve’s achievement system does:

• It encourages players more likely to explore and discover (“Spinal Tap”)
• It instructs players on the right way to do things (“Dead Giveaway”)
• It provides players a sense of place and accomplishment (“Toll Collector”)
• It challenges players to invest more time in the game (“Zombie Genocidest”)

This is exactly what you want users to be doing. Not only is are achievements effective, but there’s very little technical overhead at all. So long as some real time is put into thinking about how you want to encourage users, the only other thing you need are some delicious icons and witty titles. For a fantastic write-up about using these in a web context, check out the Yahoo! Reputation Design Pattern Library.

Web 4.0 Achievements

Putting things into context, imagine if popular sites implemented their own achievement system:

microformats 4 the win

Just Blue Sky Solutioneering™ here, but a portable achievement network like XBox Live or Steam on the web would be pretty sweet. Call it a spiritual successor to those vBulliten-era forum rankings and titles.

To help you get started, here’s a first conceptualization of a microformat spec that I’ll be pursuing in the near future. For now, let’s just call it hAchievement:

<div id="blogdor" class="achievement">
  <img src="images/trophy.png" class="icon" width="60" height="60"/>
  <h1 class="title">Blogdor, the Wordenator!</h1>
  <p class="description">You read this blog post. Gratz!</p>
</div>

That the creative processes of all writers—poets, novelists, academics, and the like—are completely hidden, is what makes NLP so frustrating. Such underdetermination is the reason why XKCD can (justifiably) dish beeves upon computational linguists with such pizazz. There’s just no way to know what the hell is going on underneath the hood with human language, and all we have to go on is what comes out on the other side.

On that note, my Plan B for this project is a magical horse named “Zellig Horsis”. Give him any subject matter, anything at all, and that thoroughbred synthesized lyrical genius will tap a Pulitzer-winning poem in morse code for you. Put peanut butter under his lips, and you can even imagine he’s actually saying it!

Horses aside, with all of the million ways I could approach computational poetry, my inner-linguist compels me to seriously consider my formal training. My roots. “Consult the Kazan School, my son” suggests my inner-voice, sounding suspiciously like Jeff Goldblum.

Two Things I Learned in Phonology

Feature Analysis of Consonants

One of the great linguistic traditions is the Prague Linguistics Circle from the 1930’s. Among their many contributions was Feature Analysis, an entirely different way to understand phonetic inventories. It did so by looking inside the phonemes themselves; breaking through the seeming atomicity of phonemes to understand them as bundles of descriptive features.

For instance, consider the phoneme [d], as in :

Although the familiar articulatory perspective is more concise, feature analysis has the ability to take any two sounds and describe how they differ in much finer granularity. By breaking down phonetic strings into these features, I could construct a much more complex machine learning strategy. For instance, it could computationally determine the efficacy of sounds coming together inside words to compute a sort of poetic “score”.

Much of the delight in poetry comes from the delicious way sounds crash and coalesce to form a unique identity. Poetry, after all, is to be enjoyed out loud; the layering of semantic and prosodic and phonetic imagery is the essence and ultimately the mystique of poetry.

Articulation of Vowels

Vowels, as you might expect, are quite a bit different from consonants. Whereas consonants have a more distinct point of articulation (whether bilabial or dental or velar or uvular, placement variation doesn’t matter too much within that location), vowels are quite a bit harder to pin down. With continuous airflow through your vocal tract, you become a musical instrument of sorts. That is, the shape of your mouth changes what frequencies are generated and how it sounds.

If you ever took an undergraduate class in Linguistics, you’ve probably seen the infamous “tongue video”. It’s rough, but it does a good job of showing just how the tongue affects the shape of the oral tract, and how that corresponds to each phoneme:

Thing is, between languages like Hungarian—which has 14 distinct vowels—and the combination of neighboring consonants, which color a vowel, there are a lot of possible sounds for each “vowel”. That’s when I got this completely random insight: what if I used the same geometric algorithm to calculate vowel proximity that I would use to calculate cartographic entities? Since I’m already using MySQL, why not add a spatial column to map the geometry of each word? It might be just crazy enough to work.

Grasping at Straws

Where either of these approaches gets me remains to be seen. Like most things with NLP, it’s a crap shoot until you actually try it.

In the meantime, I’ve had fun writing Ruby glue code to pretty-print IPA. My next step is to implement a formal subclass or maybe independent analog of the String class that stores IPA strings as an array of phonemes, which each with their own feature bundle definition. Yes, it’s nerdy, but damn is Ruby meta-programming fun.

In order to complete my BA Linguistics, I have to write a Thesis. But not on just anything. No no. I had to come up with a topic that met some level of marginal approval from my advisor, Professor Mandy Simons, a sweet but daunting woman of ~5’2” with a bespectacled glare and a curious, nearly-British accent that escapes all notions of placement like a magical knot that becomes tighter when you try to untie it.

Because OT field research on Cherokee wasn’t going to cut it, I had to soul-search for a topic whose premise would not ostensibly get me thrown out a window.

What I settled on was a combination of my favorite parts of linguistics: probabilistic models and phonology.

You see, what first got me intensely interested in linguistics was Markov Chains. It’s crazy that such a simple mechanism can produce something that has all of the great taste of lexical value, but with none of the calories! A few months after I first discovered them, Lorem Ipscream was born, and the world rejoiced at it’s creamy goodness. A furrier project of the same Markovian fervor was born a few months later. That, alas, was an insurmountably bad idea.

These days, I’ve been on quite the phonology binge. Ask any of my friends.
For instance, my girlfriend has refused to let me practice my voiced uvular trills, despite my well-argued point that it’s a fun noise to make.

Anyway, combine the two, and you get my thesis project, which I’ve come to call Keats.
Just as you might expect, I chose to honor the great poet purely to exude an image of sophistication.
Never read a lick of him in my life, to be honest.

Thesis Abstract, would I were stedfast as thou art

My goal for this project is to develop a programmatic module that, given a subject—be it love, Paris, or Heisenberg’s Uncertainty Principle—can produce a valid (and ideally tear-jerking or awe-inspiring) poem in the forms of Haiku, Limerick, and Fib.

These three chosen forms represent three distinct problem spaces, as well as three very different opportunities to investigate the essence of poetic form.

Haiku

Haikus are easy

But sometimes they don’t make sense

Refrigerator

Threadless T-Shirt

5/7/5. Anybody who’s anybody has written a Haiku before. What could be simpler?
Without any constraints on rhyme or syntax—just the right meter—Haiku are both the easiest to implement, but hardest to get right. What is it about a Haiku that makes it so great? Is it all in the content of simple, but powerful words in sequence, or is there a hidden prosody that escapes normal conscious detection?

Limerick

There once was a man named Bertold

Who drank beer when the weather grew cold

As he reached for his cup…

“NEEEEVER GONNA GIVE YOU UP!!!”

Oh, snap! You just got limerickrolled!

Limerick DB #383

Ah, the lyric form of the Everyman. Filled to the brim with innuendo and wit, it just wouldn’t be any fun to do a project without Limericks.

As luck would have it, Limericks have a pleasant balance of meter and rhyming constraints to make the problem of weaving sultry narratives interesting, while narrowing the problem into something bite-sized and manageable. Already, I have a sense that this may prove to be a harsh battleground between the forces of linguistic-based models and statistical, machine learning models.

Fib

One

Small,

Precise,

Poetic,

Spiraling mixture:

Math plus poetry yields the Fib.

Pincus, Gregory K.

And finally, the wildcard: Fibonacci-metered poems—known by the poets on the ins as, simply, “Fib”. Forged in the crucible of rebellion against the scourge of free-verse poetry in the 1990’s, this postmodern construct provides much-needed structure to the syntax-starved poetry slammers while maintaining an open-endedness and irony that resembles Germany’s inexplicable lust for all things cowboys and David Hasselhoff.

Like Haiku, meter—not rhyme—is enforced. However, the number of syllables per line is dictated by an expression of the number of rabbit pairs per generation over time. Aside from showing up everywhere in nature, the imagery of the two previous lines combining to form the next is quite, well, poetic.

As a theoretically open-ended form (although conventional Fib poets stop at line six), it also presents the challenge of producing the longest possible poem. Line 20, for example, would require 6765 syllables, which is the just about the length of a SAT writing sample. Clearly, things are going to get pretty freaky if I leave this running overnight.

So there you have it. The beginning to my semester-long quest to uncover the secret of this timeless form of expression. For posterity, and as a demonstrative means of not putting everything off until the last minute, I’ll be posting updates as the project develops.

Resolutions, come to think of it, are rather silly.

Every January 1st, as is the custom for millions of Americans, we enter into a collective delusion to commit ourselves to vague truisms that we already know to be good for us.

• Read more.
• Eat healthy.
• Work out.
• Don’t procrastinate.
• Be a better person.

Well, duh.

This time around, I’m tired of setting myself up for failure.

This time around, I’m ditching resolutions for simple accounting.

Life is like World of Warcraft—It all comes down to grinding.

Among the books I’ve had the chance to finally enjoy over the holiday was Malcolm Gladwell’s “Outliers”. One of the most striking ideas in the book is “The 10,000 Hour Rule”. Simply stated, it takes about 10,000 hours for the human mind to fully assimilate a skill to the point of mastery.

It’s an incredibly empowering sentiment: provided that you meet some minimal intelligence or talent requirements (chances are, you’re fine), it’s conceivable that the old adage, “You can do anything you put your mind to”, will come to pass. Even if you don’t completely buy into this magical number, 10,000 hours is a long ass time; if not mastery then, hell, 10,000 hours certainly couldn’t hurt.

Rather be working for a paycheck, then waiting to win the lottery.

Consider this excerpt from the book Art & Fear (via Coding Horror):

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: fifty pound of pots rated an “A”, forty pounds a “B”, and so on. Those being graded on “quality”, however, needed to produce only one pot – albeit a perfect one – to get an “A”.

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the “quantity” group was busily churning out piles of work – and learning from their mistakes – the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.

For all that we stress about in our creative processes, it all boils down to the simple tautology—that the only way to get better at something is to do it. Yes, a lot of it’s going to suck. Yes, you’re going to hate when things don’t turn out right. Just learn to get over it. And for god’s sake, don’t get embarrassed about it. You’ve got 10,000 hours to get it right. Don’t worry about getting it wrong a couple times.

Underneath every brilliant work, there was a lot of sweat, love, and determination to get there. Figure out what you love, and start chipping away.

Happy New Year.

Just a few weeks back, I was in the middle of a rather spirited discussion about MTV and its fall from grace over the years.

“In the early days, it was all about the music”, I had reminisced (never mind the fact that it would be another 6 years until I was born when MTV first launched in 1981).

Over the decades, though, the old girl’s lost her edge. Now if you tune into channel 29 (around here, anyway), just pray you don’t stumble across any of MTV’s recent reality programming, like “Paris Hilton’s My New BFF”, “Sex … with Mom and Dad”, or shudder “My Super Sweet 16”. Even MTV2, the channel they spun off to get back to their music video roots, has turned into a kind of X Games-themed version of the original, dedicating huge blocks to reruns of “Pimp My Ride” and “Jackass”.

Perhaps realizing this themselves, the good folks over at Viacom recently launched MTV Music, a website that offers untethered access to their entire music video collection. Overall, the site is a polished throwback to the early days of MTV, with all of the focus being put on the videos themselves.

They also released an API, which I’ve been playing with for a couple weeks.

The Rundown

Protocols	REST	REST
Formats	mRSS, Atom	XML, JSON, RSS
API Groups	Artist, Genre, Video	Artist, Category, Image, Rating, Release, Station, Track, Video
Dev Key Required?	No	Yes
Usage Limit	“Reasonable Usage” (See Terms of Service)	5,000 reqs / day / App ID
Number of Music Videos	20886	47561

Just like the MTV Music site itself, its API is all about the music videos. So, unlike other APIs like Last.fm and Yahoo! Music, there’s just enough information about artists and genres to get you to the main attraction: every music video MTV ever aired. Certainly, that’s a compelling reason to check it out, but how does it stack up against Yahoo! Music, which also boasts a huge collection of music videos?

It turns out that Yahoo! Music has more than double the number of videos as MTV Music, which might come as a surprise, considering MTV’s synonymy with music videos. As for the rest of the chart, it’s clear that MTV and Yahoo! are geared for different purposes. So, to get a better idea about what the MTV API is all about, read on for a more detailed overview.

Reading the Fine Print

For developers out there who are interested in building on top of the MTV API, you should carefully read through the Terms of Service. Here are two parts that are potentially deal-breakers:

3.(f) MTVN API Usage Quotas. MTVN may, in its sole discretion, restrict your use of the MTVN APIs at any time for any reason, including, without limitation, establishing quotas on the number of daily calls you may make from Your Website(s) to the MTVN APIs. Without limiting the generality of the foregoing, you may not use the MTVN APIs in any manner that exceeds reasonable request volume, constitutes excessive or abusive usage, or otherwise fails to comply or is inconsistent with the Specifications, all as determined by MTVN in its sole discretion. You shall not attempt to aggregate use of the MTVN APIs or representation of multiple MTVN API IDs, if applicable, for the purpose of circumventing any such usage quotas.

Usage quotas were an especially sore point for me in developing my Ruby library around the API (oh yes, there’s code at the end…funny how that seems to be the trend lately). While soft rate limits might sound good at first, a lack of agreed-upon rules up-front can lead to major awkwardness later. From my experience, MTV won’t get mad if you keep it below 1 request / second.

3. (c) Personal Use Only. Your access to and use of the MTVN APIs and MTVN API Content are limited for personal, non-commercial use only, which may include, without limitation, (i) use of the MTVN APIs to display MTVN API Content on Your Website(s) to the extent the same does not recreate or duplicate a material portion of any end user experience provided via any web sites or online programming service owned or controlled by MTVN or its affiliates and (ii) the sale or distribution of any device that provides access to or use of Your Website(s). You may not derive income from the use or provision of the MTVN APIs and/or MTVN API Content, whether for direct commercial or monetary gain or otherwise including, without limitation, income derived from the sale or access to the MTVN API Content or any advertising, sponsorships, promotions or any other content displayed, served or otherwise made available within or adjacent to the display of MTVN API Content via Your Website(s).

Limiting use to personal and non-commercial purposes means that you can’t do much beyond embedding videos on your blog—at least that’s my take. Even blogging might be going too far for the ToS if you have banner ads. Really, you might be better off just with YouTube. Throwing caution to the wind, I thought I’d add in a video that I had stuck in my head while writing this post:

Assorted API Beef

Beyond some of the bigger issues, there are some other minor quirks that you should know about the API. Once again, this is beta software, so this could all be fixed really soon. I’ve been in contact with the lead developer over there, and it seems that they’re working hard to push new releases to address these issues. In the meantime, though, here’s some random beef about the API:

Sparse Developer Resources

It might just be the page design, but something about their developer site just feels…empty. There’s about as much documentation about the API as needed, but it lacks an overall cohesiveness and vision of how it should be used. There are message boards, which are encouraged for requesting features and bug reports, but there’s not much incentive to use it. There’s a blog, which has the latest information about the API, but it only gets updated once a month. Despite all of this, I can’t help but thank developer Justin Tormey, for responding to my e-mails and answering questions on the message boards. Unfortunately, it looks like it’s all on him to take care of whiny developers like me.

No Flash Player Events

Currently, there’s no way to communicate with the flash player for the music videos. There’s no way to tell if it’s playing or not, if the video has loaded, played through, or even what the video is. Contrast this with the Y! Music Video Player, which gives you all the control you could ask for.

No XML or JSON-formatted Responses

One of the more interesting choices made by the MTV Music API is to provide responses only in formatted feeds. Normally, web services have a few formats to choose from, like JSON or XML, but not in this case. While I applaud the appropriate choice of mRSS for video results, it ends up feeling out of place when returning results for artists, or genres.

Incomplete Error Handling

Remember that rate limit mentioned in the Terms of Service? If you go over, here’s what you get back:
<h1>403 Developer Over Qps</h1>
What’s most confusing is that this isn’t in the documentation at all. A recent update added error responses to the API, but not for this.

Limited Meta Information

Like I mentioned above, there’s only as much information about artists as necessary to then access their videos. Either as a consequence or the mRSS format or a design constraint, simple information like the album title that corresponds to a given music video just isn’t there.

Selling Points

Now that I’ve vented my frustrations about the API, I’d be remiss in talking about some of the things that it really has going for it:

Huge Collection of Videos

Although the Y! Music video collection has twice the number of videos, MTV’s 20,000+ videos are definitely nothing to scoff at. Although it had less success with some of the fringe genres I listen to, it had a good selection for the mainstream stuff I tried out.

No Ads (yet…)

Perhaps the biggest competitive advantage the MTV Music API has over Yahoo! is that its music videos don’t have ads. Ads are one of my biggest gripes about Y! Music videos, so it’s a breath of fresh air to be able to rickroll someone without being foiled by an ad for Nissan.

mRSS and OpenSearch

Somebody at MTV has a good head on their shoulders for making use of these technologies, even if not the best use. Despite my mild protestation, the MTV API is exactly what mRSS was spec’d for. Adding OpenSearch into the mix was also in good taste.

The Verdict

The MTV Music API, as it currently stands, is a bit of a mixed bag. It’s still in beta, so it still has some room to grow before its release proper, but for now, it leaves a lot left to be desired. What I’m most concerned about, though, are the constrictive Terms of Service, which are enough to make me wary of choosing to develop with their technologies.

Either way, you can check out http://developer.mtvnservices.com/ for everything you need to get started. Or, if you’re a Rubyist, read on for an even easier way to get started…

require ‘mtv-music.rb’

Following the “You can only know an API once you’ve walked a mile in its shoes” tradition, I’ve fashioned a Ruby wrapper around the API.

If the code samples below entice, go ahead and try it out.

require 'mtv-music'
include MTV::Music

artist = Artist.new("Radiohead")

puts artist.name
puts artist.website

puts '*' * 40
puts

puts 'Videos'
artist.videos.each do |video|
  puts "t- %s" % video.title
end

video = Video.new("hznHivqrbHHZBZNXB") # Radiohead's "There There"

# The MTV Music API makes it easy to embed videos into any webpage
puts video.embed_code
# <embed src="http://media.mtvnservices.com/mgid:uma:video:api.mtvnservices.com:202930"
#        width="448" height="366" type="application/x-shockwave-flash"
#        allowFullScreen="true" allowScriptAccess="always" flashvars="autoPlay=false" />

]]> http://mattt.me/2008/11/i-want-my-mtv-api/feed/ http://mattt.me/2008/08/y-music-battle-of-the-apis-in-the-key-of-ruby/ http://mattt.me/2008/08/y-music-battle-of-the-apis-in-the-key-of-ruby/#comments Fri, 08 Aug 2008 00:53:38 +0000 Mattt Thompson http://matttthompson.com/?p=96 The Yahoo! Music API is impressive, but the question remains: How does it stack up to the reigning champion of music APIs? How does Yahoo! Music compare to Last.fm?]]>

Earlier this week, Yahoo! Music announced the release of their API. Jim Bumgardner, a front-end engineer from the team, gave a nice introduction, which included some examples of what you could do with it. In fact, his post inspired me to dive into the docs to see what else was in there.

Truth be told, the Yahoo! Music API is a substantial entry into the music-related APIs. It offers a powerful and elegant interface to one of the largest music catalogs in the world, including hands-down the largest volume of music videos out there. As a Yahoo! property, it serves a massive user base, whose listening habits power a comprehensive graph of for finding similar artists. Through its corporate partners, Yahoo! Music also enjoys the benefits of direct access to artists and labels from around the world.

The Yahoo! Music API is impressive, but the question remains:
How does it stack up to the reigning champion of music APIs?
How does Yahoo! Music compare to Last.fm?

As you might expect, there’s no quick answer. Both offer features that are about on parity, and each have their own advantages and shortcomings, which is to be expected. What’s interesting is what those differences are, to see what’s truly novel about what Yahoo! Music has to offer.

If you already hip to the whole Last.fm thing, you can skip down to the next section for the gritty details.

A Brief History of Last.fm

Last.fm was one of the first and most notable music APIs, launching right as the Web 2.0 thing was hitting its stride in late 2005. It’s greatest strength is its ability to track its user’s listening habits real-time, through a process called “Scrobbling”. Over a just a few weeks, a user’s personal listening habits are matched against everyone else’s to see what else you might be interested in. Back in June, Last.fm announced the second iteration of their APIs, which further beefed up its offerings with support for tagging and user sessions. As a long-time Last.fm user, I can vouch for their recommendation engine: no matter how obscure or indie my musical tastes get, it always has great recommendations.

Enough about that, on with the gritty details:

The Gritty Details

Protocols	REST, XML-RPC	REST
Formats	Plaintext, XML, XSPF, RSS	XML, JSON, RSS
API Groups	Album, Artist, Event, Group, Library, Tag, Track, User	Artist, Category, Image, Rating, Release, Station, Track, Video
Dev Key Required?	Yes (v2.0 only)	Yes
Usage Limit	5 reqs / sec / IP Address	5,000 reqs / day / App ID

As far as functionality goes, there’s a lot of overlap between these two APIs. Both offer search APIs for Artists, Albums, Tracks, and Users–the bread an butter of any respectable music catalog. Last.fm, being mostly user-generated, categorizes its music using a tag-based folksonomy, whereas Yahoo! takes a top-down approach by using categories, which include genres, eras, and themes.

In terms of unique features, the Last.fm API exposes its database of upcoming performances by artists and other events. It also has supports a wider range of user interactions, including the ability to submit listening information ala “Scrobbling”.

On the other hand, Yahoo! Music, is unrivaled in its offering of music videos, and the ability to embed videos directly into web pages is a killer feature. Although less flashy, Yahoo! Music supports spelling recommendations to disambiguate results, which in practice, make it a lot more usable.

Both APIs offer a means of getting and manipulating user data. Last.fm uses session-based authentication tokens, and Y! Music does in its own way, via BBAUTH.

Logistically, there are quite a few differences in how developers can use either of the APIs.

Although the previous version of the Last.fm allowed use without an API key, the new version requires it. Fortunately, their terms of use remain fairly liberal, providing API keys for both commercial and non-commercial use. As far as usage caps, applications can make up to 5 requests per IP address per second.

The terms of use for Y! Music are just like any other Yahoo! API, with the big exception being that use of the Music API is limited to non-commercial use. Requests require an App ID, with which up to 5,000 queries per day can be made.

Lastly, although for the most part a matter of taste, each has their own way of formatting REST requests.

The REST architecture for Last.fm resembles a familiar, object-oriented approach:
http://ws.audioscrobbler.com/2.0/?method=album.getinfo&artist=Jamiroquai&album=Dynamite

By contrast, the relative cryptic nature of the Y! Music API takes a little getting used to, with a typical request looking like:
http://us.music.yahooapis.com/release/v1/list/artist/252859

Rock out with Ruby

Lucky for you, over the course of researching the Y! Music API, I wrote a pretty flexible Ruby library, which will be available as a gem in the next couple of days. You can get the latest version at the project’s Github page.

To give you an idea of what it’s like, here’s a simple example:

require 'yahoo-music'
include Yahoo::Music
Yahoo::Music.app_id = "..." # Put Your App ID Here

artist = Artist.new("Beirut") # Searches by name and uses first result

album = artist.releases.detect{|r| r.title == "Flying Club Cup"}

puts album.title
puts album.artist
puts "Release Date:" + album.released_on.strftime("%m/%d/%Y")
puts
puts "Tracks"
artist.tracks.each_with_index do |track, i|
  puts "t%d %s t%2d:%2d" % [i, track.title, track.duration / 60, track.duration % 60]
end

Pretty awesome, right? Although the library’s still in early development, and doesn’t completely cover all of the functionality of the full API yet, feel free to check it out for yourself and tell me what you think.

D.S. al Coda

Actually, talking about libraries is less of a digression than a good stopping-off point for this article. Like I said before, both Last.fm and Y! Music are both excellent music APIs that are definitely worth a look. The best place to start is to just dive into some code and write something.¹ I hope that this deeper dive not only gives you a context for what they can do, but also inspire you to get out there and make something awesome.

---

¹ If you’re looking for a good Ruby library for Last.fm, be sure to check out John Nunemaker’s Scrobbler gem (http://scrobbler.rubyforge.org/). It has served me well in the past, and was a great source of inspiration when I made my Y! Music library.

]]> http://mattt.me/2008/08/y-music-battle-of-the-apis-in-the-key-of-ruby/feed/ http://mattt.me/2008/07/gears-browserplus-and-web-3-oh/ http://mattt.me/2008/07/gears-browserplus-and-web-3-oh/#comments Wed, 30 Jul 2008 03:00:51 +0000 Mattt Thompson http://matttthompson.com/?p=67 If you’ve heard of Gears or BrowserPlus, chances are you've heard them positioned as competitors in one way or another. Another Google vs. Yahoo! showdown. Truth is, that’s not the case at all, or at least doesn’t have to be. After taking a look at what these technologies actually offer, we can see that they’re actually two distinct approaches to an exciting new direction for the web.]]>

Originally posted on the Yahoo! Developer Network Blog

Speculative tech journalism is wrought with some pretty bone-headed predictions.

I’m fully aware of that.

Every time you toss your hat in the ring to offer your opinion, you’re opening yourself up to the possibility of being as wrong as the guy who said, “No one will need more than 637 kb of memory for a personal computer. 640K ought to be enough for anybody.”

Despite all of that, I’d like to tell you about what the future of the Web will be. What Web 3.0 might look like.

But first, some context:

Web 2.0, for all of the nebulously related concepts it represents, was ultimately made possible by Ajax. Asynchronous communication between the browser and servers (a la Ajax) provided the latency needed to recreate the look and feel of a regular desktop application. Not to argue the finer points in this minefield of buzzwords and strong opinions, but the emergence of rich user interaction changed everything. It got people to rethink what a website could be. It was a newer, shinier series of tubes.

Seriously, think back to the first time you saw Google Maps and your jaw dropped, or how Flickr’s Photo Organizer blew your mind. Can you imagine Digg being anything more than yet another Fark without that “digg it” button? And how about all of the little transparent things, like sorting a list by drag and drop, accordion lists, live previews, or page animations?

So much of the internet’s look and feel is powered by Ajax. Go back to a time before Ajax, and the internet was something completely different; it’s not implausible to think that another substantial technology could make Web 2.0 no more than a distant memory.

Right now, two technologies in particular, Gears (formerly Google Gears) and Yahoo! BrowserPlus, have amazing potential to completely reshape the future of internet.

If you’ve heard of Gears or BrowserPlus, chances are you’ve heard them positioned as competitors in one way or another. Another Google vs. Yahoo! showdown. Truth is, that’s not the case at all, or at least doesn’t have to be. After taking a look at what these technologies actually offer, we can see that they’re actually two distinct approaches to an exciting new direction for the web.

(Google) Gears

Gears recently celebrated its first birthday in style, by being released as open-source and dropping the Google brand. In Gears’ anniversary blog post, software engineer Chris Prince explained it this way:

“We want to make it clear that Gears isn’t just a Google thing. We see Gears as a way for everyone to get involved with upgrading the web platform.”

In fact, perhaps the best way to think about what Gears is, to borrow a phrase from a blog post from Dion Almaer, as “a bleeding edge implementation of HTML5”, as many of the core functionalities of Gears have been inspired by the HTML5 draft spec (and the reverse is true as well).

Gears provides several major components, including Database, which can read and write to a local SQLite3 database; WorkerPool, which parallelizes processing tasks; and LocalServer, which allows websites to cache resources like HTML, images, or video, and serve them from the local machine.

So what can you do with that? If you just went on what Google had to say about it from their examples, well…you might miss the point completely. That is to say, the demos aren’t exactly flashy. To be fair, it’s difficult to come up with trivial examples that encapsulate such a nontrivial range of functionality.

To show what’s possible with Gears, let’s imagine what your favorite sites would look like enhanced by Gears:

GMail / Y! Mail

Messages are downloaded and synced with a local database on your machine, meaning mail can be read and written offline, and searching is really really fast, as in search-as-you-type fast.

Vimeo / Y! Video / YouTube

Instead of uploading videos and waiting for them to be encoded later, videos could be encoded locally and uploaded along the way, making the whole process much faster.

Kingdom of Loathing

…or pretty much any web-based game currently. Download all of the game files you need in one shot and adventure to your heart’s content, syncing up your local database every now and then.

As I said before, if you think back to how Ajax changed the way people could interact with websites, much of it is completely transparent. For instance, deleting items in a list with real-time feedback is so pervasive that it became an expected behavior. Just as that doesn’t lend itself to an impressive demo, I can see a lot of Gears’ functionality becoming equally pervasive, yet remaining invisible to the normal user. But as you know, Ajax has a flashier side too, which is where Yahoo! comes in.

Yahoo! BrowserPlus

Back in May, Yahoo! announced BrowserPlus, a technology poised to blur the lines between the browser and the desktop. Unlike Gears, which provides a single underlying framework, BrowserPlus is made up of individual modules, called services, which can be installed and updated on the fly.

If you haven’t done so already, you should check out some of the BrowserPlus demos firsthand to get the full effect of what’s possible.

PhotoDrop is a good example of how BrowserPlus services interact. Using the DragAndDrop service, users are able to drag photos from their desktop directly into the browser window. The ImageAlter service hooks in with ImageMagick to resize, crop, and apply filters to your photos instantly. When you’re done editing your photos, the FlickrUploader tool will take care of adding them to your photostream. Putting all of these services together creates a new, rich user interface that just isn’t possible with Ajax alone.

BrowserPlus comes with an assortment of built-in services you can build on, including one for text-to-speech, one that provides secure cross-domain JSON requests, and one that ties into local notifications like Growl for the Mac, or its Windows counterpart, Snarl. But what’s really exciting is that BrowserPlus provides its own Ruby interpreter, which gives developers the power to build their own Ruby-based services.

For now, BrowserPlus is only available to Yahoo! properties, but developers are able to test out the APIs locally. The reason for this limited release is explained in the FAQ:

“Once we’re confident that the system is safe for users when hosted on any site (even potentially malicious sites), and that users have all the proper controls over use of BrowserPlus on their computer – we’ll open it up. The “sneak peek” is a way to balance the desire for openness and the risk of exposing users to a new dynamic web-enabled software model.”

BrowserPlus has been criticized by some for its lack of openness, but there are good reasons for Yahoo! to play it safe. Yahoo! is known and trusted for its commitment to providing a secure online experience to its users. With BrowserPlus, Yahoo! has the ability to immediately restrict services if they’re found to pose a security risk. Pragmatically speaking, if a technology is going to provide such powerful capabilities to developers, a trusted gatekeeper like Yahoo! is ultimately what it takes to ensure that your machine is safe.

A Philosophical Difference

With their respective design goals in mind, it doesn’t make sense to look at Gears and BrowserPlus as competitors. Sure, both have an intersecting feature set, including desktop integration, and automatic software updates, but that’s where the similarities end.

Gears is built for the back-end, providing a solid foundation of core technologies to the browser. Being open-source software suits it well too, just as the W3C itself was started as a way to open the process of web standard adoption.

BrowserPlus, on the other hand, has amazing potential for the front-end in a way that’s only possible because of its modular architecture. It’s based upon Yahoo!’s strong reputation for privacy and security, which will ensure that users can feel safe trying out new things online.

Put the two together (BrowserPlus + Gears), and there’s a chance you’ll rip a hole in the space-time continuum, or something– they’re just that awesome. Seriously though, check out what the fuss is about now, and get a leg up on this Web 3.0 thing.

]]> http://mattt.me/2008/07/gears-browserplus-and-web-3-oh/feed/ http://mattt.me/2008/07/high-performance-mysql/ http://mattt.me/2008/07/high-performance-mysql/#comments Tue, 15 Jul 2008 22:42:11 +0000 Mattt Thompson http://matttthompson.com/?p=3 A lot has changed since the first publication of High Performance MySQL in 2004. At some point, the web turned 2.0, startups became cool again, and SQL became a bad word (regardless of how you pronounce it).]]>

Originally posted on the Yahoo! Developer Network Blog

A lot has changed since the first publication of High Performance MySQL in 2004. At some point, the web turned 2.0, startups became cool again, and SQL became a bad word (regardless of how you pronounce it). For many in this new generation of web development, hand-writing SQL has become a sort of vestige–something to suffer through only as a last resort. Frameworks like Django and Ruby on Rails provide a clean abstraction to the database called an ORM, or Object-Relational Mapping, which makes it possible to develop an entire application without writing a single line of SQL. Developers end up learning the hard way that reliance on database-agnostic development can have tremendous consequences once the application has to scale to thousands or millions of users.

High Performance MySQL, by Baron Schwartz, Peter Zaitsev, Vadim Tkachenko, Jeremy Zawodny, Arjen Lentz, and Derek J. Balling, is a high-level introduction to the most powerful aspects of MySQL that’s still accessible to anyone who’s worked with a database before. Although this book focuses on MySQL, many of the concepts like transactions, locking, and query optimization are important to an understanding of any database system. You can get through the book on just a basic literacy of SQL, but it might be helpful to have a companion reference lying around in case something comes up.

The book starts out with a detailed overview of the MySQL architecture, with careful attention to MySQL’s selection of storage engines, which offers a lot of flexibility in how you can optimize performance. As a way to explain the differences between each of these storage engines and when it might make sense to use InnoDB rather than MyISAM, for instance, the book provides a thorough explanation of how they implement locking and transactions.

Chapters 3 & 4 are also fairly specific to MySQL, as they explain the finer details of how it processes queries. MySQL does a fair amount of heuristics-based optimization on incoming queries depending on the nature of your data, and understanding what’s going on under the hood can not only help you fix queries, but start writing better SQL. If you’ve ever added column indices because it seemed like the cool thing to do, or couldn’t quite figure out how that simple query could be taking a few seconds, these two chapters especially will set you straight.

The rest of the book covers general best practices for optimizing server performance. These chapters provide a good reference for how MySQL best implements practices like benchmarking, load balancing, backups, and hardware scaling. Since most of these optimizations are external to MySQL itself, much of the information is important for any production environment.

Without a doubt, High Performance MySQL belongs on any serious developer’s bookshelf. Like the original, it’s an enjoyable, engaging read that provides battled-tested solutions to real-world problems that engineers face in scaling their applications. The second edition covers the new features of MySQL 5, including stored procedures, cursors, triggers and views, as well as a deeper comparative look into the various storage engines. Perhaps more importantly, the second edition brings with it a reminder of how important database design is to web development.

]]> http://mattt.me/2008/07/high-performance-mysql/feed/ http://mattt.me/2008/06/javascript-the-good-parts/ http://mattt.me/2008/06/javascript-the-good-parts/#comments Sun, 15 Jun 2008 19:00:48 +0000 Mattt Thompson http://matttthompson.com/?p=72 As weird as it is, there are a lot of good reasons to actually learn JavaScript, and not just pretend its some other language with C syntax. It takes a great deal of insight into this language to understand its true potential, and Douglas Crockford offers just this in his new book, "Javascript: The Good Parts."]]>

Originally posted on the Yahoo! Developer Network Blog

First off, a confession–I’ve rewritten the first sentence of this post maybe a dozen times. (How’s that for meta?) For one reason or another, I figured that writing about JavaScript would warrant some clever insight into the language. Some poetic tidbit that people could rally behind. The kind of voice that could shape a generation of empathetic web developers.

Something like:

Javascript is a language of alchemy, turning inconsistent behavior and design flaws into Web 2.0 gold.

Or maybe:

JavaScript is nothing short of quantum physics with its spooky action at a distance.

Before I decided to go all self-referential with my introduction, I had tentatively settled on something I think we can all agree with:

JavaScript is really, really weird.

As weird as it is, there are a lot of good reasons to actually learn JavaScript, and not just pretend it’s some other language with C syntax. It takes a great deal of insight into this language to understand its true potential, and Douglas Crockford offers just this in his new book, “Javascript: The Good Parts.” This offering from O’Reilly and Yahoo! sets itself apart from other JavaScript books in a lot of ways.

The first thing you’ll notice is how thin this book is. Weighing in at around 100 pages and another 50 with appendices, it’s a concise and remarkably pleasant read that you’ll probably finish in a single sitting. Nonetheless, there is enough packed into this book to have you reading it through a few more times. Another rarity among programming books is its remarkable clarity and pleasant, patient tone. That said, this book is not for beginners (JavaScript probably isn’t a great language to learn programming with anyway). By tailoring the book to people already familiar with programming fundamentals, Crockford is able to tease out the quirks that make JavaScript so confusing, and expose some of its hidden elegance.

The best parts of “The Good Parts” are definitely Chapters 4 and 5, which take a look at some of JavaScript’s more unique aspect–functions as objects and prototypes. In Chapter 4, Crockford explains and implements an impressive set of language features like modules, function currying, and memoization, which make JavaScript start to feel closer to languages like Python, Haskell, or Lisp. Chapter 5 does a great job explaining the difference between classical and prototypical inheritance, and how to take advantage of prototypes in JavaScript.

If you’re a developer looking to finally make sense of JavaScript, you’ll find this book to be right up your alley. It’s an enjoyable read that presents core programming concepts and won’t waste your time on the marketing buzzwords that usually come with the territory. Do yourself a favor and check out Douglas Crockford’s Javascript: The Good Parts, and forget everything you ever thought about this poor, misunderstood language.

]]> http://mattt.me/2008/06/javascript-the-good-parts/feed/