pocketMMaps (with apologies to Andrew)

I feel kind of bad calling them pocketMMaps since the name is only a single, easily overlooked, letter away from the pocketmaps that Andrew and Mikel built for Mapufacture. By the time it gets beyond being a 0.1 release I may try to find a better moniker but since its written in Python and all the Papernet-related software has a tradition of stupid naming conventions second only to Python itself I figured: Why not?

pocketMMaps is software for creating PocketMod-style books for a map image too large to fit on a single page. Maps are generated dynamically and then sliced in to (PocketMod) page-sized pieces and re-arranged on multiple sheets of paper so that when printed and folded you get a little book with the map laid out in handy 2-page spreads. Just like the maps you see in the back of travel guides. It may seem as though I have an obsession with destroying the travel book industry but I don't, really. It's just that their maps suck and their books are too big to fit in my back pocket.

It works like this:

height = 11
width = 8.5
margin = .25
dpi = 144

bbox = (45.482882,-73.619899,45.532687,-73.547801)
zoom = 16

out = "montreal_pocketmmap.pdf"

pm = pocketMMap(height, width, margin, dpi)
pm.load_provider('OPEN_STREET_MAP')
pm.draw(bbox, zoom)
pm.save(out)

When you're done, you end up with a PDF file.

pocketMMaps are built on the shoulders of ModestMaps and ws-compose, hence the MM-iness of the naMMe. Sooner rather than eventually I plan to import the code in to the ModestMaps Python trunk but it is still very early days feature-wise. The current short list looks something like this:

• Add support for markers, dots, pinwins, etc. and reference indexes just like the original pocketnet and ws-modestmaps libraries.
• Wrapper code, or libraries, to generate a map and plot points from a source KML, GeoRSS, whatever file just like the orginal mmPDF.php library. Hello, Dopplr tips...
• Better font handling and sizing with an eye towards optionally using PIL so that pycairo is not a dependency.
• Try and reconcile some of the more eggregious uses of lib_copy_and_paste back in to either the wscompose or ModestMaps branches.
• Tangentially related, writing bloody setup.py files for all the stuff in the ModestMaps Python trunk so you don't have to run everything out of the same folder.
• Support for the Metric system because, you know, the whole world uses it.
• Work out the gotchas and best practices for especially large maps that span many sheets of paper. Experience shows that a PocketMod book with more than five nested sheets of paper kind of sucks.
• ws-pocketmmap.

In the meantime, if you are planning to spend the day wandering around a new and unfamiliar neighbourhood and just want a simple map to print out and take with you, well, now you can.

Ladies and gentlemen, pocketMMap.py 0.1

http://icanhaz.com/flosm

No, really.

Dispatches From A Town Called Patience

Depending on how charitable you are feeling corrections in Flickr are either an admission of failure or an embrace of everything good about community-driven data and neo-geography.

A little bit of history:

The next two slides go on to say You won't get more precise than neighbourhood followed by the even more depressing Turning coordinates into something useful is hard. I don't remember how long it had been since we'd finally stopped displaying neighbourhoods for geotagged photos on Flickr when Paul gave that presentation but it highlighted why we did.

The reality is that we have always been surprisingly successful at determining a photo's neighbourhood but there have also always been mistakes and no one is very tolerant of mistakes about place. Between plane-living and the Internets we may be midway through the process of distilling every place on Earth down to one of a half-dozen archetypal city-states but until that happens trying to affect a person's relationship to the history and geography of whatever piece of dirt they call home will continue to be a source of tension.

Or, if you went to ETech in 2007, being told your photos were taken in the San Diego County Jail. In a note to the W3C's mailing list on geolocation APIs on the subject of reverse-geocoding, I wrote:

There is always going to be a bias of interpretation in the hierarchy of relationships you choose. This is okay, really.

The simplest example is to contrast the way that Flickr and FireEagle handle localities since the two sites share an almost exact hierarchy of places. Flickr treats anything with neighbourhoods as a locality so in our model Duncans Mills, CA (pop. 84) and Mexico City (pop. 19M) are assumed to be the same "type" of place.

FireEagle does not. If you authorize an application to know your whereabouts at a city level there is an expectation that your actual location will be suitably fuzzed (assuming that you share an expectation that cities are big) and in a town of 84 people there's not a lot of room to get fuzzy in.

Never mind so-called disputed places (Kashmir, the West Bank, Cyprus, etc.) all neighbourhoods are disputed around the edges. (This is often true of localities, as well.)

For example, the rough consensus in San Francisco is that Delores street is the dividing line between the Mission and Noe Valley. That said there are those people who may live on the one side of the line and very much believe themselves to be living on the "other". Our experience has been that there are few better ways to pick a fight than to tell someone what neighbourhood they are in (and being wrong).

There is also the problem where the data simply doesn't exist yet or it is just old and dusty, sometimes wrong, and often plain weird : Manhattan Valley, anyone?

Multiply that by 80 million geotagged photos.

This is mostly what Dan and I were talking about when we two-man-luged our way through a talk about reverse-geocoding at Where 2.0 this spring : Even if we did have a giant database mapping every point on the planet (multipled by decimal degrees, multiplied again by zoom levels) to a place people still wouldn't agree on its contents.

As useful as the frustration of having photos geotagged in my aparment show up in Noe Valley was to understanding and improving some of the issues involved in making this stuff work at all there is only so much human subtlety you can, literally, codify in to a computer program.

What if instead of simply trying to keep pace with all of human history and predujice as a series of cascading if/else statements we laid our cards — the places that we think a pair of latitude and longitude coordinates might be — on the table and if we get it wrong give people the chance to tell us what they meant and to learn from that? What if the next time you geotagged a photo we compared where we think that place is against the places that you've told us are nearby? If not you, then your contacts? What if every single person on Flickr points out that a neighbourhood, or town, is just plain wrong?

Dan has written a pair of good blog posts describing the nuts and bolts of how corrections works and in the nerdier of the two he sums it up nicely by saying:

On a slightly more philosophical level, it’s a never ending process. We’ll never reach a point where we can say “Right that’s in, all borders between places have been decided”. But what we should end up with are boundaries as defined by Flickr users.

...

For us, it’s a first small step into an experiment, and actually a pretty big experiment as we’re potentially accepting “corrections” from our millions and millions of users. We’re not quite sure how it’ll all turn out, but we’re armed with Maths, Algorithms and kitten photos.

For a lot of reasons (most good, some bad) corrections took a long time to make it out the door. Then again, so did geotagging itself and the important part is that it's out. We haven't deployed any of the code to test for nearby corrections since it's a bit of a chicken and egg problem without, well, corrected data but that should follow soon enough. It's a small thing, more like polish and detailing than a full-on feature, and like machine tags I don't expect people to get excited about it, right away, but once they need it I hope it will shine for them.

And we've started displaying neighbourhoods again.

If corrections were difficult to explain to people, describing the changes to way we handle airports — and why we should even bother — was often like talking to people in whale-song. Very quietly, last May, we updated the reverse-geocoding logic to stop looking for the nearest town when a photo is geotagged at an airport and instead treat the airport itself as the town. The Places pages were all updated accordingly and now you can type in /places/3-LETTER-AIRPORT-CODE and it will pretty much do what you expect (barring those places that need to be corrected, of which there are inevitably a few).

We did this not (only) because I have long read and admired the work of J.G. Ballard but because it is the right thing to do. Though, administratively, airports are generally associated with a single municipal or regional entity they often serve a range of towns and cities and the realities of contemporary travel mean that they have evolved from being simple gateways to captial-P places with their own culture, norms and gravity.

Whatever an airport may be for accounting purposes it rarely matches contemporary expectations and experience; no one wants to see pictures of Hounslow when they go to London Heathrow Airport and vice versa. The latter still happens unfortunately but the hope is that by adding airport and corrections knobs (and a few others still too soon to talk about) that users can adjust things to their understanding of the world and we can begin to map facts on the ground rather than from on high.

History Boxes

Two years ago, we went to Europe and when we came back I made a book. Last year, we went to Europe and when we came back I made a book with maps. That was the germ which started Net:Flickr::Geo and later my involvement with Modest Maps.

We went to Europe again this year and I left knowing that when we got back I would make another book, with fancy dithered maps, but that's as much as I'd thought about it. During the last two trips I had done a lot of work to position and render multiple images which ultimately ended with poster maps, a single large scale representation of a group of photos. As pleased as I am with poster maps they are only good when played loud and not necessarily the best way to share photos from a trip.

Lack of wall space aside I also don't want to look at this stuff every day let alone be forced to stand up while I'm doing it. Having a book idling around the living room means I can pick it up and poke through while I'm on the phone or carry them in to the kitchen when I am feeling all weepy-eyed and nostaglic.

I also now have three years and just over as many books worth of photos from Europe at this point. Since I am a creature of habit many of the pictures are taken in proximity to one another so I thought it would be nice to create a poster map for a single photo, instead of a photo set, and show it in the nested among all the other photos that I had taken nearby, over time.

Cluster maps for lack of any better name.

I am especially intrigued by the idea of generating cluster maps year over year and watching the spaces that I revisit gradually fill in with the echoes of the past. A sort of slowed down timeline done in book form where jumping through the layers means putting one down one book and picking up another.

This eventually led to historical cluster maps, like the one of the nave at St. Andrew's cathedral in downtown Sydney shown to your right. Historical because it blends the results of two queries for nearby photos. The first, (n) days on either side of when the source photo was taken. The second, photos taken an equal numbers of days since right now.

Cluster maps are not really suited to the firehose of photos available on Flickr since they mostly just end up being as large or larger than poster maps. As they are also constrained to a small area geographically (typically a 1km radius) you also run in to the problem where they are raining pinwins. What you're seeing in the cluster map of St. Andrew's are only photos that properly licensed for this sort of thing so you can imagine how many photos you'd get if you decided to not play by the rules (don't do that).

At least not for print, anyway. This sort of thing might sing in the hands of a good Flash developer. Either way it was a pleasant exercise in visualizing the history box that Flickr has become. It always a bit frustrating talking to people — especially people who make hardware because that part is still honest-to-god hard — and pointing out that if they sent us a bounding box (via the API) we'd send them pictures! And getting blank stares.

The thing that's kind of awesome about having 80 million geotagged photos (give or take) most of which are public at a time when location aware devices are becoming useful enough to bother carrying around is that every (every) place on Earth becomes a kind of Auggie Wren portal.

If you think relentless chatter about the Papernet is bad I've been obsessed with Paul Auster's story about the shop owner who photographs one corner in Brooklyn every day for 20 years since I read it, well, just about 20 years ago. I love that there is record of a shared experience of place with a complete stranger and I love to see how the contours have expanded and contracted. I love that the network explodes the notion of a photo album in to an omnipresence and allow me to go for a walk with the past and — to simultaneously borrow and abuse Neb's observation — wrap the present in an onion-skin of subjectivities.

Well, that's the idea anyway.

The first pass will probably suck because the current vogue for so-called Hertzian spaces doesn't leave much room for anything except devices (or worse, optical-cum-neuro implants) nevermind the imagination. I had a conversation with Steve Lambert a while back about wanting to something along the lines of the Anti-Advertising Agency, to string up fast and cheap displays aroung the city to show nearby geotagged photos; real-live history boxes. He suggested using white-box ATM machines.

Rather than devices which live in and are always announcing a perpetual present, maybe finding a norm and a practice to give all those dumb terminals that surround us the ability to display an awareness of their environment — nearby geotagged photos or something like a non-creepy version of Leonard's BlueBall hack as quasi screensavers on ATM displays — would be more fun, more compelling and more meaningful than a crush of people bumping in to each other as they awkwardly transpose their personal iPhone space on to the physical world.

Which is a very long way of saying: The first part was to get radial queries working in the Flickr API.

It only took 18 months following the release of geotagging so it is extra wonderful to see it become part of applications like Frasier Spier's Exposure so quickly and Matt and Alex's subsequent comments are the icing on the cake. The iPhone may not save us but we have to start somewhere (the peanut gallery should refer to my comments about hardware vendors above) and this sort of stuff is what makes it all worth it on both the good days and the bad days.

Technically you've always been able to do something like radial queries using plain old bounding boxes but this ignores two truths. First, that bounding boxes are boring when they're not hard. Second, if you wanted to sort by proximity to the center of the box you needed to so manually which meant fetching all the results in advance of doing anything with them (like sorting them by distance). Pagination is hard when it's not boring.

There's a lot to be said for just being able to do this:

my $flickr = Net::Flickr::API->new(...);

my $photos = $flickr->api_call({
	'method' => 'flickr.photos.search',
	'args' => {
        	'lat' => 37.785377, 'lon' => -122.414852,
                'radius' => 2, 'tags' => 'kittens',
                'page' => 1, 'per_page' => 15,
	},
});  
                

With that problem out of the way I started the work of updating Net::Flickr::Geo to fetch and layout all the photos.

The choice to use the square (75 x 75 pixel) thumbnail for nearby photos was largely dictated by proximty of all the images to one another and to limit, as much as possible, the goal of preventing overlaps from sending everything shooting off the map. For that same reason thumbnails that share the same latitude and longitude and clumped together in single pinwin whose size is determined by the square root (and some fussing) of the total number of images.

Like poster maps, cluster maps enforce the ws-modestmaps bbox method which says: Just get me all the tiles you need to display this bounding box at this zoom level. At some point, when there are tiles so granular you could read the morning's paper on them it should be possible use larger sized thumbnails and still stay within the map's canvas but not yet. There is still a maximum density (or clustering) after which point the map is dwarfed by the space needed to plot all the pinwins and there is no room left for anything interesting.

The bounding box used to fetch the tiles is not the circle-squared of the radius but rather the furthest points of the entire result set. This means that the principal photo in a cluster map may not always be centered. I suppose it would easy enough to make this optional but at least in a book context with many cluster maps I like the variety of composition this adds to the collection of images.

All things being equal, that's pretty much all there was to it. Dream up a new layout and ordering for the photos, update the Flickr API (would that it were always so easy) and tweak the code that calls it. Profit!

It's just that when I was generating test maps of photos from the trip to SXSW this year I couldn't help but notice (again) how ugly and jaggy the taller anchors for the pinwins were. Having long threatened to update ws-modestmaps to (optionally) use the Python bindings for the Cairo vector graphics library I figured that if I was going to spend the money to have a nice book printed I might as well also spend the time to create properly aliased pinwins.

  22 |# DANGER WILL ROBINSON : YOU ARE ABOUT TO
  23 |# ENCOUNTER CODE THAT WILL MAKE YOU WEEP.
  24 |# I KNOW. I CRY EVERY TIME I LOOK AT IT.

That is from the top of the library that was generating the pinwins when I began. The only kind thing I can say about it is that it somehow managed to work for as long as it has without completely failing. It's two biggest problems were that it was originally developed in tandem with the rest of the ws-modestmaps code and carried all the scars of the many, various attempts to get everything working and was an absolute clusterfuck of best guesses and approximations for generating perspective shadows in the absence of being able to work out the matrix math to do it the right way. Math is hard and I chose to go shopping.

Of course, the first thing I did was try a straight, brute force, port of all the old cruft to Cairo. This worked well until I got to the part with the shadows, specifically rendering the rounded corners in perspective. The problem I've always had with matrix math (used to calculate all the various parameters needed to do stuff like a persective transformation) is the absence of any documentation that doesn't involve Math Shapes. I can find lots of example code but it seems to exist only in a theoritical world where demonstrating the concepts using actual numbers as inputs and outputs seems to be tantamount to admitting defeat.

That's the bad news. The worse news was that reproducing the original pinwin shadow kludge, done using the Python Imaging Library in pycairo would be even uglier and I would have to live with the shame of having knowingly littered the world with such shitty and brittle code not once but twice. This isn't a I was so dumb back then scenario but rather hear no evil, see no evil material.

After about two weeks of chipping away at a whole series of mostly incomprehensible matrix-related tutorials and one invaluable mailing list message written in plain English (which I've managed to misplace in the interim) I was able to glean just enough information to start doing tests where I could finally see the relationship between all these goofy-ass variables and the actual thing I was creating.

Because there's a lot to be said for just being able to do this:

m = PinwinMarker(240, 180, 50)
m.cartoon_shadows = False

m.draw()
m.save("pinwin.png")

If you look carefully you may notice that the shadows still seem a little odd near the base of the anchor. That's because one of the side effects of blurring the shadows is that the shadow itself ends up being obliterated as it approaches the point. I decided that it was better to add a little bit of quirkiness (the length of the anchor is extended by some factor of itself before it is blurred) at the base of the anchor rather than having the shadow taper off midway from the canvas which is what happens when you do it the right way.

(You can also choose to disable blurry shadows at which point everything syncs up nices and properly again.)

Having finally acheived a true and proper Googlesthetic (sic) I ultimately opted to leave the cartoon version with their funny butler's platter on a giraffe's neck shadows as the default. Both versions now use the same perspective transformation for the shadow's canvas (the bit where the image you want to display is placed) but I still think the goofy shadows are warmer, less distracting and better suited for cluster maps where the pinwins are often floating in whitespace.

Then, just when I though I was done Dan pointed out that the shadow from one pinwin should never fall the anchor of another. Having implemented his suggestion I'm not sure I still agree but it does mean that when you create a pinwin you can ask for any one of three layers: the canvas, the shadow or the two combined. ws-modestmaps does this by default first adding all the shadows and doing a second pass with the pinwin canvases.

You can see the effect above, in the cluster map of Sydney, near the shadow of the cathedral's nave. If you're like me you may find the zebra-striping and the contrast a bit too harsh and be thinking about making it optional in the next code release.

All for a silly photo album. Oh yeah, remember the book?

Finally, something like a month and a half after getting back from Europe I had several hundred test images, a book with cluster maps (and funny bluish shadows because of the way blacks and greys are printed these days) and all the code to generate them committed to the CPAN and the ModestMaps SVN tree. That is:

• Net::Flickr::Geo.pm 0.72 — this contains new methods to generate cluster maps for a single photo, an entire photoset or an historical cluster map for a photo.
• ModestMaps, rev. 637 — this contains no user-facing changes, beside the way the shadows are drawn. Under the hood ws-modestmaps will silently check to see if pycairo is available and use it if it is. The new pinwin code comes with a bunch of knobs and toggles which I would like to expose in ws-modestmaps but that is going to wait until I get around to making the core service WSGI compliant.

Two other packages have been released separately, though they are both installed automatically as a dependency or included by default with the software listed above. They are:

• Net::ModestMaps.pm 1.1 — a simple wrapper object for calling a ws-modestmaps server in Perl (and preventing LWP::UserAgent from freaking out when it's passed several hundred HTTP response headers).
• pwmarker.py 1.0 — all the pinwin code discussed above but packaged independently of Modest Maps. This will work with both the Python Imaging Library or the Python Cairo bindings but in either case requires the former.

Having recently had to rebuild my laptop from scratch I can attest to the part where none of this is a simple one-click install yet. My only defence is that there are only so many hours in the day and these are the stars I am looking at, right now. If it's any help, at least on a Mac it's a reasonably straightforward albeit lengthy process using MacPorts and the CPAN (assuming of course you've also installed the OS X developer tools). These are some of my notes:

# py25-cairo (macports; this will install Python)

# py25-pil (macports)

# libgd2 (macports; this will install Perl for some crazy reason)

# p5-GD (macports)

# p5-XML-LibXML (macports)

# p5-XML-XPath (macports; installs p5-XML-Parser)

# symlink all your Perl and Python related binaries from /opt/local/bin
  to /usr/bin and /usr/local/bin

# Bundle::CPAN (cpan)

# force install Geo::Distance (cpan; there's a bad test that I need to work
  around in N:F:G)

# Net::Flickr::Geo (cpan; installs the remaining Perl modules)

Software is for humans, right?

Meanwhile, while all that was going on Andrew announced that Mapufacture has launched their PocketMaps feature, which will generate mmPDF-style PocketMod booklets for the maps you create on their site. If I ever manage to finish writing this blog post the next bits to bubble to the top of list should start to see a lot of the work described above rolled back in to some of the earlier Papernet begets PocketMod begets *-PDF related projects where all this nonsense started in the first place.

See also: The Tate's Your Collection pamphlets. Because they are fucking awesome, that's why!