Gary Bernhardt's OLD Blog

New blog URL

2007-12-22T13:03:00.000-08:00

If you're seeing this, you're looking at my old blog. My new blog is at blog.extracheese.org.

Blog woes

2007-12-19T16:02:00.000-08:00

Last night, I switched my blog to tumblr; this morning, I switched it back to Blogger. Over the last two days, I've spent a ridiculous amount of time and effort trying to make the switch without breaking any links. I'll spare the details, but it involved a lot of mod_rewrite, a PHP script written by Henrik Nyh that proxies all requests to tumblr, and a huge list of URLs mapping the old Blogger ones to the new tumblr ones.

I found the proxy-with-a-PHP-script thing distasteful, but the lack of decent tag support was the thing that ultimately made me give up. I have a Python-specific feed that gets aggregated by the unofficial planet python, so whatever I switch to needs to be able to generate tag-specific feeds. Tumblr does let you add tags to your posts, but it doesn't seem to actually do anything with them.

So, I'm in search of blogging software. I want something that:

Is written in Python or Ruby. Python because I know it very well; Ruby because I want to know it better.
Is simple. I'm not interested in anything built in Django or TurboGears or Rails. Preferably, it would be something that runs on my local machine, generating the static files that make up the live blog.
Allows custom URLs but has sane defaults. (I.e., no exposed serial integer keys.)
Supports tags and can generate subfeeds based on them, as well as human-readable post lists that are filtered by them. (For example, with my current blog you can look at only Python posts if you want to.)
Does not involve a database. (Not even SQLite.)
Reads the posts out of plain HTML files, which I will write by hand.

I've looked around for something like this, but everything seems to be either big (e.g., Pyblosxom and Typo) or someone's weekend project that never got touched again. I'm sure that the bigger ones are quite good at what they do, but I just want something that takes my plaintext files and generates an appropriate URL structure. It can do it offline or online - I don't care - but it's got to be simple and require no complicated installation or configuration.

So, any ideas, or am I starting a new project? :)

Tubes by proxy

2007-10-09T15:03:00.001-07:00

On Friday, I am bored and want video games. But no Internet service! How to download new games to Wii?!

Assets:

One VX8300 cellular telephone.
One Macintosh Book Professional.
One Nintendo Wii.

Solution:

Connect VX8300 to MacBook via Bluetooth.
Configure MacBook to use VX8300 as a modem for free. This is an undocumented Verizon feature!
Put MacBook in ad-hoc wireless mode.
Enable internet sharing on MacBook.
Inform Wii of wireless network.
Access Wii Store!

Full data path: Internet -> Verizon Wireless -> CDMA 2000 (1x) -> VX8300 -> Bluetooth -> MacBook -> 802.11g -> Wii.

Operation was a success. Breath of Fire II acquired.

Undeletable zombie files on OS X

2007-09-13T13:48:00.001-07:00

Somehow, one of Bitbacker's system tests occasionally creates an undeletable file. It happens within a test that generates random filenames, then tries to back up and restore them. Unfortunately, there are apparently some character sequences that make HFS+ choke. I've only had this happen twice ever - once on my 17" MacBook Pro, and just now on my brand new 15" MacBook Pro. Here's what the file looks like:

It can't be deleted from Finder:

It can't be deleted from the console:

And it can't even be "ls"ed:

If I manually move the containing folder to the trash and try to empty it, I get this, which is even more ridiculous than the rest:

I've googled for undeletable files on OS X, as well as for the specific error messages Finder throws, and tried every suggested fix I found. And, of course, I've tried to delete the file from within Python:

>>> os.unlink(os.listdir('.')[0])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  OSError: [Errno 2] No such file or directory:
  '\\xe4\\xad\\xa9\\xe1\\x84\\x84\\xe1\\x85\\xaf\\[...truncated]'

Dear Apple: please replace HFS+. Seriously, it sucks.

Globals and cargo culting

2007-09-07T14:10:00.000-07:00

Matt Wilson wants a module's functions to log to one logger, but he can't change their interface and he doesn't want to use a global variable. This is the kind of thing that decorators are very good at, for better or worse. Here's a decorator that will do the job:

def with_logger(fn):
    def new_fn(*args, **kwargs):
        logger = get_singleton_logger()
        return fn(logger=logger, *args, **kwargs)
    return new_fn

And here's how to use it to define a function that gets a logger instance without the caller passing it in:

>>> @with_logger
... def add(x, y, logger):
...     logger.warning('x + y = %i' % (x + y))
...
>>> add(1, 2)
WARNING:foo:x + y = 3

This seems to be exactly what Matt was looking for: there are no globals, a logger gets injected, and the function's interface hasn't changed. But is it a good idea?

No, it's a ridiculous idea! It's just a reimplementation of global variables! All I've done is come up with a complicated scheme for injecting a single logger instance into every function in the module. But that's exactly what a global does! This is something that programmers have done over and over again in the name of OO. Everyone wants globals, but they go to great lengths to hide it.

Here are three possible ways to solve the original logger problem:

Use a singleton, and have each function retrieve the instance that way.
Use a decorator that injects the instance into the function's argument list each time. In with_logger, I combined this with a singleton.
Just use a global.

If you choose (1) or (2), the joke's on you. You're still using a global, but now you have two problems: global state and a complicated method for managing it. There's no need for that, because we already have a simple method for injecting instances into a module's functions: globals!

Of course, sometimes singletons or decorators like with_logging make sense, but only when they actually do something. If all they do is allow multiple functions to access a single long-living instance, they're dangerous and needlessly complex. In almost all cases, singleton and related techniques are nothing more than cargo cult programming.

When JSON isn't JSON

2007-07-23T09:20:00.000-07:00

JSON is so simple that you can specify it on an index card, but we still can't get it right. For example, here's what happens when simplejson and python-cjson talk about slashes:

# simplejson correctly decodes cjson's data
>>> print simplejson.loads(cjson.encode('/'))
/

# cjson fails to decode simplejson's data
>>> print cjson.decode(simplejson.dumps('/'))
\/

In this case, the problem is that cjson doesn't handle backslashes correctly. There are two ways to say "/" in JSON: "/" and "\/". When encoding, simplejson always escapes slashes, but cjson never does:

>>> print simplejson.dumps('/')
"\/"
>>> print cjson.encode('/')
"/"

The reverse is also true: simplejson knows how to decode "\/", but cjson decodes it incorrectly:

>>> print simplejson.loads('"\/"')
/
>>> print cjson.decode('"\/"')
\/

So there you go: simplejson and cjson don't interoperate. This bit me when I tried to move BitBacker from simplejson to cjson for performance reasons. The live alpha server had a few thousand records encoded with simplejson, all of which included slashes. When I switched to cjson, everything broke because every "/foo/bar" entry in the database came back as "\/foo\/bar".

As far as I'm concerned, this problem with JSON is actually an argument for simple data formats like JSON. If we can't get full interoperability between something as stupidly simple as JSON, how did anyone ever expect WS-* to work?

Adobe, master of the painful install

2007-05-27T20:57:00.001-07:00

After using Macs for about two years, I've become accustomed to the painless installation process there: open the dmg, drag the app to the Applications folder, and you're done. Unfortunately, I recently had to install Acrobat Reader to electronically sign some insurance documents. They have managed to come up with a downright pathological installation process:

Download AdbeRdr80_DLM_en_US_i386.dmg.
Open the dmg.
DownloadReader.pkg is inside; open it.
Click through some stuff.
Wait for the "Adobe Reader Download Manager" to download another 23 MB dmg.
The "Download Manager" has silently quit (what?), but the dmg is now mounted. Open "Adobe Reader 8 Installer.app" and wait for it to finish.
The "Installer" has silently quit (what?), but it must be done, because Acrobat is now running.

This is amazing. When I saw the very first pkg file, I was annoyed: I hate installers and they're rare in the Mac world. As I kept installing and installing, it was almost surreal. In the above list, there are three times where something is being installed. Working back from Acrobat itself,

"Adobe Reader 8 Installer.app" installs "Adobe reader.app", so it's the installer.
The "Download Manager" installs that, so it's the installer installer.
DownloadReader.pkg installed that, so it's the installer installer installer.

So, while most Mac programs have shed the archaic notion of an installer altogether, Adobe Acrobat Reader has a program that installs a program that installs a program that installs the software you actually need.

Texturing and programming

2007-05-22T19:08:00.000-07:00

Flayra just posted a video of one of the Natural Selection 2 artists texturing a 3D model. I've done a bit of modeling and texturing (poorly, of course), so it's awesome to watch someone who's really good. It also reminded me of how I work on RESTdb, though. I always have my heavily modified nosy running in its own terminal window, and it gives me constant feedback on the effects of every little change I make.

In the video, it's obvious that constant feedback is critical to the process. If the artist tried to create the entire texture without seeing it applied to the mesh, the process would take much longer, yield poorer results, or probably both. As far as I'm concerned, the same applies to programming: if you use a bit of TDD, along with a continuous test runner like nosy, the programming process becomes more organic and approachable. It becomes easier to break the problem down into pieces that can be tackled easily, and you have a better sense of the scope of your changes.

If you haven't tried nosy yet, do it! It's very easy - just download nosy.py, stick it in your project's directory, and run it. It's only about 25 lines long, but it can have a huge impact on how you work. If you're still not convinced, check out Jeff Winkler's screencast.

Are your tests lying to you?

2007-04-07T13:55:00.001-07:00

If you've written a test for a module, and the module is changed in the future, there are three things that can happen:

The test keeps passing because nothing is broken. (Good.)
The test fails because something is wrong. (Great - this is the test's job!)
The test keeps passing, but it silently stops testing the thing it claims to (BAD, BAD, BAD!).

Scenario 3 above is very dangerous, and it's a major problem in testing. What you have in that situation is a lying test: it says "I'm testing feature x," but actually passes without doing so. In other words, you have a test that no longer warns you if you break something.

If you've not been bitten by this, it might not be an obvious problem. To make it a little more clear, let's look at a toy example (in Python, of course!) Here's a silly WebClient class and its test.

 1 class WebClient:
 2     """An HTTP client that supports both SSL and plain connections"""
 3     def __init__(self):
 4         self.use_ssl = False
 5 
 6     def get(self, url):
 7         # Hand any request off to external functions
 8         if self.use_ssl:
 9             return get_with_ssl(url)
10         else:
11             return get_without_ssl(url)
12 
13 def test_web_client():
14     # Make sure everything works with normal HTTP
15     client = WebClient()
16     assert client.get('/') == expected_data #defined elsewhere
17 
18     # Make sure everything works with SSL as well
19     client.use_ssl = True
20     assert client.get('/') == expected_data

This works fine - the test passes and it tests what it claims to. But what happens if someone renames the use_ssl attribute later?

 1 class WebClient:
 2     def __init__(self):
 3         self.using_ssl = False
 4 
 5     def get(self, url):
 6         # Hand any request off to external functions
 7         if self.using_ssl:
 8             return get_with_ssl(url)
 9         else:
10             return get_without_ssl(url)

Take a look back at the test. It's no longer testing what it claims to, because "use_ssl" no longer means anything to WebClient. The test still passes, though - it's just that neither of the two get() calls actually uses SSL.

This is a serious problem - you need to be able to trust your tests, but for all you know your tests are giving you false positives. The question, then, is how can we detect this type of mistake? Well, there is a simple method that will catch at least some of them. What you need is a meta-test: a test that ensures that the tests aren't lying to you. It's really not that bad; here's the pseudocode:

for each test in the suite:
    for each line of code that isn't an assertion:
        remove that line of code (but not the rest)
        run the test and make sure that it fails

Basically, this meta-test is ensuring that every line of code in the test is required: removing any line should cause the test to fail. This sounds complicated, but it only has to be implemented once. Once it exists as a nose plugin, for example, you can use it without writing any extra code.

Let's look at how this would affect the example. Here's the testing code again:

 1 def test_web_client():
 2     # Make sure everything works with normal HTTP
 3     client = WebClient()
 4     assert client.get('/') == expected_data #defined elsewhere
 5 
 6     # Make sure everything works with SSL as well
 7     client.use_ssl = True
 8     assert client.get('/') == expected_data

The meta-test will step through, removing each relevant line and making sure that the test fails. The only executable lines that aren't assertions are 3 and 7. When it removes line 3, the test will fail because "client" won't be defined. So that iteration of the meta-test passes. When it removes line 7, the test will still pass. Because the test passes with a line removed, the meta-test will fail. The meta-test has detected the fact that line 7 isn't necessary, which is a red flag that says "this test might lie to you later!"

It's important to note that the meta-test will fail even when the test is working. It really is a meta-test: it's only testing the test. This is a good thing. It tells you when you've written a crappy test - a test that isn't paying enough attention.

Let's return to the example and try to fix it. To make the meta-test pass again, the test could be changed to be more sensitive to WebClient's state:

 1 def test_web_client():
 2     # Make sure everything works with normal HTTP
 3     client = WebClient()
 4     assert client.get('/') == expected_data #defined elsewhere
 5     assert client.use_ssl == False
 6 
 7     # Make sure everything works with SSL as well
 8     client.use_ssl = True
 9     assert client.get('/') == expected_data
10     assert client.use_ssl == True

Now the meta-test passes, and the original test_web_client is more resilient to silent failures. If someone renames WebClient's use_ssl attribute, the test won't silently stop testing like it did before. Instead, line 5 will raise an exception and the test will fail.

Of course, this isn't foolproof. If you added line 10 but not line 5, you wouldn't be doing yourself any good (figuring out why is left as an exercise for the reader :). The meta-test would still pass, though, and you would still have a test that may lie to you in the future. So this meta-testing method isn't a magic bullet that will force you to write good tests. For a careful tester, though, it throws up a red flag for tests that might be susceptible to very subtle errors.

(Nitpicker's corner: Yes, the problem in this test was caused by questionable design in WebClient itself. Using an instance variable to control a class's behavior in this way is error-prone to begin with. This testing problem also arises in much more subtle situations, though; I have the scars to prove it.)

The 13-Year-Old Equivalent of WS-*

2007-03-14T13:17:00.000-07:00

One of the very first non-trivial programs I wrote was a game (of course), and it's hilarious to look back on it now. It was sort of like Dope Wars meets Choose Your Own Adventure, with a ton of 13-year-old humor thrown in. Even though I haven't seen it in ten years, there's one hilarious feature I remember very well.

Since it was a game (and not an abstract), it needed random numbers: you could buy and sell various illicit substances, and I needed to randomize the market prices. After stumbling around the VB docs for a while, I found the "Rnd" function, which returns a random float from 0 to 1. Obviously, I could generate random prices by doing this:

price = max_price * Rnd

Unfortunately, I didn't realize that. The example I saw for the "Rnd" function showed it being used like this:

value = Rnd
if value < 0.5 then
    'do the first thing
else
    'do the second thing
end if

Being the inexperienced pseudo-programmer that I was, I blindly copied the form of the example to suit my needs. My code went something like this:

value = Rnd
if value < 0.25 then
    price = 7
else if value < 0.5 then
    price = 12
else if value < 0.75 then
    price = 25
else
    price = 31
end if

I'm pretty sure there were only four cases, but there might've been eight. Either way, the limiting force for the number of cases was simple laziness. Maybe if I'd been less lazy, and added more cases, I would've noticed that I was moving toward n conditionals with a probability of 1/n each.

I like to think about stupid things like this for two reasons. First, it helps me keep at least a shred of humility about my abilities. In ten years, the software I'm writing now will probably also look ridiculous to me (but hopefully not this ridiculous). Second, this is a great example of missing the elegant solution and ending up with something big and gross. You might say it's the 13-year-old equivalent of WS-*.

Zero to Slashdot in Three Days

2007-03-06T11:42:00.000-08:00

The Genesis

A few days before PyCon, Brian suggested that we build a web app in one night. It took a little longer than that to polish it up, but we launched sucks-rocks.com on Tuesday. Since then, it's had over 40,000 page views and been slashdotted (OK... it was the Japanese Slashdot, but it's still a Slashdot.)

Sucks/rocks rates the terms you enter by doing web searches and counting results. For example, if you search for "Windows sucks" using Google, you'll get many more results than for "Windows rocks". The opposite is true for FreeBSD. From this, we can infer that people probably like FreeBSD more than Windows. The actual searches that are done by sucks/rocks are more complex than this, but they follow a similar pattern.

The Search Engine Arms Race

Once we started getting a lot of traffic, it was very hard for us to keep sucks/rocks going because we kept running out of searches. Here are the search APIs we used, in the order that we added them:

Search Engine	Queries/Day	Interface	Suckiness of Results
Google	1,000	SOAP	Low
Yahoo	5,000	REST	Pretty low
live.com	10,000	SOAP	IMMEASURABLY HIGH!

We started with Google, but ran out of queries before we even launched. We then used Yahoo, but ran out when 100shiki.com linked to us, forcing me to add support for live.com. Unfortunately, live.com's search results are terrible. Terrible! If you search sucks/rocks for "lord of the rings", you'll get a "?" back. This means that the engine whose results are cached (which is live.com, of course) reported that there were 0 "total results available". Great.

Now we have a cache of almost 60,000 searches, most of which are from live.com. Many of those are totally wrong, of course. My next task is to add a background thread that slowly replaces all of the cached live.com results with Yahoo results.

The Code

Sucks/rocks runs on top of web.py, but only uses it for URL dispatching. Paste does the HTTP serving, with WebFaction's Apache instance on the front end (disclaimer: the WebFaction link is an affiliate link). This simple setup handled about a million HTTP requests in four days, using less than 5% of the CPU almost all of the time (except when it was at the top of slashdot.jp).

Easy Come, Easy Go

With our slashdotting over, We've gone from 10,000+ pageviews per day to about 1,000. Slashdot giveth, and Slashdot taketh away. That's OK, because I need some time to push all of the crappy live.com results out of the cache anyway.

(Brian has also posted about sucks/rocks: 1, 2).

PyCon 2007: The Untold Stories

2007-02-27T19:43:00.000-08:00

Most of the PyCon posts are about the sessions, so here are some of the interesting things I did outside of the scheduled talks. I have pictures for many of them thanks to Mike Pirnat's diligent photography.

Pagoda CMS

Brian, Chris, and Ian demoed Pagoda, their upcoming open-source CMS. It's very user-centric, and they're spending a lot of effort on the user experience. Even though I don't use CMSes, I'm excited about this project because I'm so sick of crappy UIs. Peoples' responses seemed positive, but I think some people were disappointed that Pagoda takes the easy-to-use approach rather than the kitchen-sink approach. That's ok; that's why we have Zope - the kitchen sink is there for the taking!

Python Is Basically DOS, Right?

I headed up to my room to grab my hoodie, and on the way back I was in the elevator with a 40ish couple. They asked me what this conference was about; I told them it was about Python, which is a programming language. The guy asked me whether "that's anything like DOS". It was kind of funny, but mostly just jarring. After being in close quarters with lots of smart programmers for 2 days, it was weird to suddenly talk to someone whose computer experience apparently began and ended around 1990.

The Mysterious Ellipsis

Dave, Mike, and I were at the hotel's bar, and the topic of Python's ellipsis operator ("...") came up some how. From the grammar in the slicing docs, we could tell that the ellipsis could appear in slices, but we couldn't trick Python into taking it without throwing an exception. I figured it out later - in a slice, the "..." token is just translated into an "Ellipsis" object:

>>> class Foo:
...     def __getitem__(self, x):    
...         return x
... 
>>> f = Foo()
>>> f[1:2:3]
slice(1, 2, 3)
>>> f[...]
Ellipsis
>>> f[1, 2, ..., 100]
(1, 2, Ellipsis, 100)

Apparently, it's mostly used for numeric stuff like Numpy. I definitely understand Python's slicing much better after that confusing night.

Mischief on The Open Space Board

Brian and Chris posted a "Python in The Adult Entertainment Industry" card on the open spaces board with my name on it. It was up for about 20 minutes before Brian pointed it out to me and I took it down. Hopefully, I escaped without too many prominent Python hackers associating me with pornography.

I have to wonder whether anyone saw that card and was actually interested in going to the session. Maybe Chris and Brian's silliness prompted an interesting discussion of Python and porn somewhere...

RESTDB

The open space I actually did lead was on "REST, Databases, and RESTful Databases" rather than pornography. Unfortunately, I dove into explaining RESTDB right at the start. It turned out that not everyone was familiar with REST, or convinced of its usefulness, or both. So, we ended up talking about REST for the second half. I think the session would've been more useful to everyone involved if we'd discussed REST first, then moved on to RESTful databases. I'm not sure how much everyone else got out of it, but I learned a lot about how to explain what RESTDB is and why we might want it.

Django vs. The World

I didn't fly back until Monday, so I was still there on Sunday night. Most of the people who were still there were staying for the sprints, so the conference area was pretty quiet as everyone quietly hacked away (with the exception of the Wii room).

I was on my way back to the "quiet room", which was full of Django guys. On my way there, a big group of people appeared and asked where the Django guys were. I pointed them towards the quiet room and joined them on their way there. The group was made up of TurboGears guys, Pylons guys, Paste guys, and some that I didn't recognize. They busted into the Django room and caused some friendly commotion, with one notable result being this post on Ian Bicking's blog. I'm pretty sure that EWT's bathtub full of alcohol (pictured to the right) was a factor in this incident.

Magic URL Mapping

After the ruckus in the quiet room ended, I hacked up some crazy URL mapping code based on an experiment Brian did a while back. Here's a controller defined using it:

class UserController(_/'users'/User):
    def get(self, user):
        return dict(
            email=user.email,
            name=user.name)

The _/'users'/User part defines the controller's URL, and User is actually a RESTDB resource type. So, for example, if someone requests /users/Bob, this controller will be invoked and the Bob RESTDB resource will automatically be retrieved and passed in. This works for multiple records, so you could also have more complex controllers like:

class BlogController(_/'users'/User/'blogs'/Blog):
    def get(self, user, blog):
        assert blog.user == user # yep!

BlogController would be called for URLs like /users/Bob/blogs/TheBobBlog and, once again, both Bob and his blog would automatically be pulled out of the database. Of course, it's fully RESTful (hence the get method).

Keep in mind that this is just a silly experiment; please don't freak out because I'm overloading division to produce a URL mapping object that I then subclass. (Although, to be honest, the code isn't that bad; it's only about 60 lines long.)

Overall, PyCon was awesome, and I'm really glad I went. It's going to be in Chicago next year, so I won't have to lose two full days to travel (awesome!)

Introducing Another Wildly Ambitious Database Project

2007-02-17T21:53:00.001-08:00

One week ago today, I started hacking on a new project: a database implemented as a RESTful HTTP service. Brian has been pestering me to post about it since before any code was even written, so here we are.

I've been calling the project RESTDB, but that's only because I haven't come up with a better name yet. It's sort of relational, but not quite. Depending on how you squint, you might think it is. Likewise, it's not completely RESTful: it lacks arbitrary POST semantics. Despite these caveats, it's quite similar to both RDBMSes and RESTful systems. Let's have a look.

Defining the Schema

We're going to build a multi-user TODO list (basically stolen from Brian's TurboGears tutorial). As with a traditional RDBMS, the first step is to define our schema:

class User(Resource):
    email = String(key=True)
    lists = List(Link('TodoList'))

class TodoList(Resource):
    id = Integer(key=True)
    title = String()
    items = List(Link('Item'))

class Item(Resource):
    id = Integer(key=True)
    value = String()

Obviously, this looks a lot like a SQLObject table definition. One thing is very different, though: the way resources are related to each other. In SQLObject, each TodoList would have a foreign key that points to its User. In RESTDB, this is inverted: the User contains a list of links that point to TodoLists. In SQL terms, you can think of this as a list of foreign keys. This has many serious implications, both for the database's implementation and for how clients interact with it. In the interest of brevity, I will valiantly gloss over every single one of them for now.

The Data

Now that we've defined our schema, let's see what's going on from an HTTP perspective. The definition above will lead to a URL structure like this (with arbitrary example records inserted):

/User
/User/me@example.com
/TodoList
/TodoList/1
/Item
/Item/1
/Item/2
/Item/3

These resources' structures are dictated by the resource classes we defined above. Resources are stored as simple JSON data, so they're human readable even in their raw form. For example, here's what "/TodoList/1" might look like:

{
    'id': 1,
    'title': 'Groceries',
    'items': [
        '/Item/1',
        '/Item/2',
        '/Item/3',
    ]
}

It's just plain old JSON data, but it follows our schema: there's an ID, a title, and a list of item links. You don't even need a database client program to look at it; just point your web browser at the resource and you'll get back the JSON representation. You can download the whole database with wget if you want to.

The Client

Of course, this database isn't designed to be used by humans directly; human readability is just a nice bonus. wgetting your database is a neat gimmick, but what we really care about manipulating it with code.

To illustrate how simple the client is, here's the entire client-side definition for our todo list database:

c = Client('127.0.0.1:17321')
User = Resource(c, 'User')
TodoList = Resource(c, 'TodoList')
Item = Resource(c, 'Item')

That's it: all you have to do is tell it the names of the resources. Note that this doesn't mean that there aren't constraints on the data - there are! Lots of constraints - all the constraints you care to define! It's just that they're only on the server side. If you step out of line, the server will slap you with an "HTTP 400: No Shenanigans Allowed".

We'll get to the shenanigans in a minute. First, let's try the client out by creating a user:

>>> me = User.post(email="me@example.com", lists=[])
>>> me.email
u'me@example.com'
>>> me.lists
[]

All we do is POST a new user resource with an email address and no todo lists. This is literally just an HTTP POST to /User. The database responds with an HTTP "Location" header to tell the client that the new resource is at "/User/me@example.com".

Now that our user is securely fastened to the database, let's create a todo list with some items:

>>> # Create a todo list and assign it to the user
>>> my_list = TodoList.post(title="Groceries", items=[])
>>> me.lists.append(my_list)
>>> me.put()

>>> # Create some items and add them to the todo list
>>> i1 = Item.post(id=0, value="Milk")
>>> i2 = Item.post(id=1, value="Eggs")
>>> i3 = Item.post(id=2, value="Bread")
>>> my_list.items += [i1, i2, i3]
>>> my_list.put()

We POST a new TodoList, just like we POSTed a new user before. Then we have to update the user resource to point at the new list. me.lists is just a plain old Python list, so we append the new todo list, then PUT me to update the server's copy. We then repeat the same process to add items to the todo list.

Light's Green; Trap's Clean

Now that we've trapped a bunch of data in our database, let's start from scratch and pull the todo list's items back:

>>> me = User.get("me@example.com")
>>> print [item.value for item in me.lists[0].items]
[u'Milk', u'Eggs', u'Bread']

Awesome.

But wait, I've conveniently left a loose end untied! I claimed that shenanigans were strictly forbidden. So far, we've been acting nice and giving the server exactly what it wants. Now let's try to feed the server some crap:

>>> me = User.post(email="you@example.com", lists=123)
Traceback (most recent call last):
  ...
client.BadRequestError: "123" is not a list
>>> me = User.post(screw_you_server="EXPLODE!")
Traceback (most recent call last):
  ...
client.BadRequestError: Didn't expect field "screw_you_server"

It's having none of it! It will snub your stupidly-formed data all day long. And it's not just simple things like types that are enforced. You can define regex constraints for your strings, ranges for your numbers, and whatever else you can dream up. You can suffocate your precious data with constraints. Your links are guaranteed to reference valid resources; your URLs are guaranteed to match your data; your data is guaranteed to match your schema.

Everything I've shown here is real, working code. A few of the things I've mentioned, like link validity constraints, aren't done yet, but they're coming. Unfortunately, I can't point you at subversion just yet, because I don't have anywhere to host it. That will hopefully change soon, and you'll be able to prod it for yourself. For now, you'll have to make do with imagining how awesome it would be to speed up your database by sticking a plain old HTTP caching proxy in front of it.

Unicode Weirdness

2007-02-14T10:57:00.000-08:00

I'm writing some tests to verify that BitBacker doesn't explode if it sees unicode filenames. For a while, I thought that OS X's terminal wasn't unicode-aware, because non-ASCII unicode characters just showed up as "?":

grbmbp:~ grb$ ls z*
z???       z??????    z????????? z???       z???

Then I happened to pipe the ls through a grep, and the unicode characters printed correctly:

grbmbp:~ grb$ ls z* | grep '.*'
z໐
z두
z툃
z䌨
z冕

What? Well, I guess I'll take it...

While posting this, it got even more fun. All five of the characters above print normally in the terminal and Finder, but only two print normally in Opera's text edit control. I wonder how many will show up once this is published. Can you see them in your RSS reader and/or browser?

Update: After publishing, I viewed the page in Safari and the characters displayed exactly like they did in the terminal and Finder. So at least Opera didn't mangle the bytes. However, Firefox draws the first three incorrectly (the same three that Opera couldn't draw at all). Unsurprisingly, IE7 can't draw any of them.

Mutable State Is Manual Memory Management

2007-02-07T13:25:00.000-08:00

The Haskell Sequence linked to my post on C# (although they obviously don't share my opinions on it). I'd never read that site before, and I came across this "quote of the week" while glancing over it:

"Mutable state is actually another form of manual memory management: every time you over-write a value you are making a decision that the old value is now garbage, regardless of what other part of the program might have been using it."

(Paul Johnson via The Haskell Sequence)

My first thought upon reading this was "why the hell didn't I think of that?" It's obvious in retrospect, and is a wonderfully concise explanation of why side effects are a bad idea. I always try to limit my use of side effects, but I never had a simple way to explain to myself (or others) why that's a good thing. Thanks, Paul.

Measure the Goodness of Your Code

2007-02-06T11:31:00.000-08:00

Because I want to keep my code short, I like to see statistics on my sandbox before committing to subversion. At first, I just did things like "svn diff | grep '^+' | wc -l", but that got old fast. So, I wrote a little script called gn (for "goodness") that computes some simple statistics. Here's what it says about my sandbox right now:

grbmbp:~/trunk grb$ gn
591 lines of diff
129 lines added
186 lines removed
-57 lines net change

I've added 129 lines and removed 186, which is a net change of -57. Any line that's simply replaced will result in one line removed and one added, for a net change of 0. The implication here is that the more negative your "net change" is, the better. (DISCLAIMER: Please don't take this literally and post angry comments.)

Sometimes I want to know the goodness of a bunch of related changes, so the script can also take any arguments that "svn diff" can take. It just passes them on, so you can compute statistics across revisions, etc.:

grbmbp:~/trunk grb$ gn -r420:451 
4246 lines of diff
1099 lines added
1627 lines removed
-528 lines net change

As you can see, I've been killing a lot of code recently. The script is below, in case you want to try it for yourself. I've only tested it on OS X; YMMV.

#!/usr/bin/python
import sys, os, re

svn_args = ' '.join(sys.argv[1:])
pipe = os.popen('svn diff %s' % svn_args)
diff_lines = pipe.readlines()

# Added lines start with '+' (but not '+++', because that marks a new file).
# The same goes for removed lines, except '-' instead of '+'.
added_lines = [line for line in diff_lines
    if line.startswith('+') and not line.startswith('+++')]
removed_lines = [line for line in diff_lines
    if line.startswith('-') and not line.startswith('---')]

print '%i lines of diff' % len(diff_lines)
print '%i lines added' % len(added_lines)
print '%i lines removed' % len(removed_lines)
print '%+i lines net change' % (len(added_lines) - len(removed_lines))

C# 3.0 Looks Promising

2007-01-24T11:38:00.000-08:00

At CodeMash last week, I learned quite a bit about C#'s new features from Scott Guthrie. I'm no fan of explicitly typed languages , but C# 3.0 has me very excited. It's getting a host of new features inspired by other languages, including anonymous types, type inference, and lambda expressions. There's plenty of discussion of these around the web, so I'm not going to rehash them all. I want to highlight the two features that I'm most excited about: type inference and LINQ.

Type Inference

C# 3.0 adds basic type inference, but it's quite weak and can only infer types on assignment. Basically, you can do these types of things:

    var i = 5;
    var s = some_function_returning_a_string();
    foreach (var s in list_of_strings) { ... }

Note that the "var" keyword does not make the variables dynamically typed - it just tells the compiler to infer their types based on what's being assigned to them. This is nice, but it's very limited. You can't, for example, declare a function with a return type of "var" and expect the compiler to figure it out. Haskell this definitely is not, but it's a huge step forward for C's verbosity-laden children.

LINQ

Let me get this out of the way: LINQ is awesome and I want it in my language.

In a nutshell, LINQ allows you to use declarative, SQLesque syntax in your C# code. I've been wishing for this feature for a long time, but Python's list comprehensions have had to hold me over so far. Here's a very simple example of what you can do with LINQ:

    int[] numbers = {5, 4, 1, 9, 8};
    var low_nums = from n in numbers
                   where n < 5
                   select n;

Here's an equivalent example in Python:

    numbers = [5, 4, 1, 9, 8]
    low_nums = [n for n in numbers if n < 5]

LINQ is far more powerful than Python's list comprehensions, though. It has many of the features of SQL: joins, grouping, count() and sum(), etc. And a single LINQ expression can query a collection of objects, a database, or XML data. This makes me drool.

Most of the discussion of LINQ at CodeMash boiled down to excitement over the idea of writing statically-typed, compiler-checked database queries. You can probably guess that this is not the reason I'm excited. For most interesting problems, the limiting factor is not whether you get your queries correct or not; it's whether you can coax your brain into (1) solving the problem and (2) translating the solution into code. Any feature that simplifies your code makes step (2) easier, and LINQ is definitely such a feature. LINQ is the only feature of any explicitly-typed language that I am covetous of. It really is that awesome, and Microsoft has managed to beat every other language to the punch with this.

Back to Reality

OK, enough gushing. I still dislike explicitly typed languages. These new features certainly aren't going to get me to leave dynamic languages behind - C# is still incredibly verbose when compared to Python, Ruby, etc. And, even though many of these features were borrowed from functional and dynamic languages, C# 3.0 is still as statically typed as ever. However, I have a newfound respect for Anders Hejlsberg and friends. If Microsoft manages to drag the hoards of C# programmers toward a more concise coding style, the world will be a much better place.

Why dynamic typing is useful

2006-12-23T17:06:00.000-08:00

Steve Yegge's Parabola has thrown a little more fuel on the static vs. dynamic fire. The comments so far have been mostly constructive, which is a nice surprise. However, there has been some back-and-forth about dynamic typing, mostly concerning why it's useful and what it actually entails. In a recent comment, "Matt" said

"I don't see how software could possibly attempt to handle anything it did not anticipate, except to gracefully fail, which T.S. did."

I've seen this sentiment before, almost always from people whose programming experience is limited to statically typed languages. If that's your background, then you've not seen the wonderful flexibility that dynamic typing adds. Here's a very simple example written in Python. Don't worry - Python is called "executable pseudo code" for a reason!

def process_many(things):
    for thing in things:
        thing.process()

Looking at this, you might expect things to be a list. That's possible, but it could also be a set, or an iterator, or even a dictionary (a hash table). As the programmer, you don't have to worry about it. As long as Python can figure out how to iterate over your things argument, everything will just work.

So, to return to Matt's statement above: this is one way code can handle situations that the programmer never considered. Maybe the guy who wrote process_many never even thought that someone might pass in a dictionary. That doesn't matter; it will work anyway.

Of course, you can get this effect in Java using interfaces. The problem with interfaces is that you have to make decisions about them ahead of time. If you were writing process_many in Java, you might not consider the case of passing in anything but a list. Then, if someone using your code wanted to pass in a set, iterator, dictionary, etc., they'd be out of luck.

This is a recurring theme with static vs. dynamic languages. In explicitly typed static languages, you have to spend time figuring out what type every little thing should have. Then, when you're finally done, there will always be some case you didn't consider, like a hand-written note telling you to call the ticket counter.

Python's default arguments are tricky

2006-12-19T20:09:00.000-08:00

Default arguments are evaluated at function definition time, so they're persistent across calls. This has some interesting (and confusing) side effects. An example:

>>> def foo(d=[]):
...     d.append('a')
...     return d

If you've not tried this before, you probably expect foo to always return ['a']: it should start with an empty list, append 'a' to it, and return. Here's what it actually does:

>>> foo()
['a']
>>> foo()
['a', 'a']
>>> foo()
['a', 'a', 'a']

This is because the default value for d is allocated when the function is created, not when it's called. Each time the function is called, the value is still hanging around from the last call. This gets even weirder if you throw threads into the mix. If two different threads are executing the function at the same time, and one of them changes a default argument, they both will see the change.

Of course, all of this is only true if the default argument's value is a mutable type. If we change foo to be defined as

>>> def foo2(d=0):
...     d += 1
...     return d

then it will always return 1. (The difference here is that in foo2, the variable d is being reassigned, while in foo its value was being changed.)