Archive for the 'Geomensch' Category

Topology rules with GeoDjango

I’m still very new to GeoDjango and the Django world in general, but I have to say that, based on what I’ve seen so far, I’m very impressed by how easy, fast and clean you can build powerful geographic web applications with it.

Django 1.2 is the current version and one thing that caught my eye in the release notes was Model validation. Among other benefits, model validation allows you to “perform custom validation on your Model” everytime a user wants to save data through a ModelForm. Translated to GeoDjango, that basically means that one can access the unleashed power of PostGIS (or the functionality of your supported geo-database of choice) to validate geometries and provide feedback for users through a nice web interface. Best part of it: it’s all done only by a few lines of additional code in your data model.

I’m not going into much detail how Django’s model validation works, you should check out the documentation if you’re interested, but here’s a quick example of creating something like a simple topology rule with GeoDjango:

Let’s say I have users entering and modifying point data and need to make sure that each point falls inside a polygon from another layer.

# models.py
from django.contrib.gis.db import models
from django.core.exceptions import ValidationError

class Poly(models.Model):
	geometry = models.PolygonField()
	objects = models.GeoManager()

class Point(models.Model):
	geometry = models.PointField()
	objects = models.GeoManager()

	# simple topology rule
	def clean(self):
		# find Polys that contain the Point
		pq = Poly.objects.filter(geometry__contains=self.geometry)
		if not pq: # no Poly has been found
			raise ValidationError('Point is not within a Poly.')

That’s it. You only need to add a custom clean() method and put your validation rules there. Those few lines of additional code check if the new point falls inside a polygon and tell the user if it’s not through the Django admin interface.

GeoDjango Model Validation

I’ve posted the project on bitbucket if you want to give it a shot. Add a little more PostGIS extravaganza and you can have a pretty sophisticated set of topology rules in your web application, bundled with the great capabilities Django apps offer.

GeoDjango on a Windows server

Deploying GeoDjango projects on a Windows server is obviously possible, but not necessarily the most seamless process one can think of. To get started, there are some great tutorials – at the django project site and here, with some more information about the correct PATH settings – available, that guide you step-by-step through a GeoDjango installation.

Below are some things that I did in addition, did differently, or simply couldn’t do because I never got them to work.

The basics: Django and IIS

IIS is was the web server of choice on the Windows machine in question. Serving Django through IIS is of course doable, there is even a dedicated page on the Django project about it. However, I totally failed in getting Django to work well with IIS. It seemed painfull and eventually made me switch the entire web server to Apache. Django integration with WSGI under Apache is a piece of cake compared to PyISAPIe and IIS if you ask me, up and running in less than 10 minutes.

Virtual Environments

Virtual Environments are great when developing with Python. They give you separated Python installations for every project you work on and avoid interferences with dependencies and installed packages across your projects. Virtual environments is something I’ll really miss on Windows, because I never got it to work correctly. Virtualenv basically creates the correct folder structure and activates a Virtual Environment to work in, but for some reason dependencies in my projects were attempting to access the main Python installation. Which obviously defeats the purpose of Virtual Environments.
What I ended up doing was, using one Django installation and being extremely careful about what goes there. Basically I have to manually validate, check and test that new packages for new projects don’t mess with existing once. Painful, I tell ya.

win-psycopg headache

I ran into serious troubles with psycopg 2.2.1 (the PostgreSQL adapter for Python). My project wouldn’t connect to the PostgreSQL database server at all. I found this entry at stackoverflow, which describes exactly the same issue. The posted solution worked for me too: use psycopg 2.0.14 instead of 2.2.1, no problems so far.

Multiple Django projects in one Apache/WSGI instance

This was the most annoying issue: turned out that only one project at a time was working on the server. That might be an option for the iPhone, but it clearly was unacceptable in my case. The problem caused by the “Python for Windows extensions” is documented in this thread, and my solution, similar to the one posted at the end of the thread, was to install pywin32-212 instead of pywin32-214.

Lessons learned

If you have a choice, get a Linux box to deploy GeoDjango. Anything but .NET seems painful to serve on Windows.

Boston mapping folks!

… join the

Boston OSGeo Enthusiasts Meetup

next Tuesday, May 18th, 6:30-8pm

at the Bocoup Loft (319 A St, Boston, MA 02210)


View Larger Map

… and meet other Open Source mapping software users and developers, share experiences, learn about tools, discuss projects and developments.

To RSVP, please just add your name to the Doodle.

For more information and to participate in planning the meetup, please join the mailing list and follow the May meetup thread.

See you there!

Playing games

Last week has been pretty intense. My department finalized a, for a regional planning agency, rather unusual project and ran a couple of community meetings during the week. What happened in those meetings wasn’t the typical PowerPoint presentation followed by a Q&A. People, mostly of younger age, played a 3D video game and participated that way to Boston’s Chinatown Masterplan process.

Emerson students created the excellent video below, that captures the very core of the project in about 3 minutes:

I only got during the final phase of the project involved, helped testing the game, setting up the website and organizing our community meetings, but it was an absolutely exciting experience to take part in. Especially the first community meeting, hosted for Chinatown residents, stood out with an highly energized atmosphere.

The game helped in the meeting as ice breaker, facilitated engaging discussions about the neighborhood and provided a fun environment for the audience to work with. We all were somehow surprised when we actually saw game mechanics kicking in. At the point where scores and winners are announced, people started cheering, got all excited, compared scores and discussed strategies why one couldn’t complete all tasks or what would have been a better way to play.

The good part: it all had serious, real world backgrounds – well, real world compressed into 30 minute tasks. However, the game deals with existing challenges of Chinatown residents. The game allows players to walk in somebody else’s shoes through Chinatown and complete tasks as another person, equipped with different skills and opportunities. It’s an educational game, used to gather feedback on and engage in an urban planning processes.

But it’s still fun to play, and fun is the one aspect I wouldn’t underestimate here. It helps to broaden the audience of the community meeting, attracts especially younger people and provides an experience you wouldn’t expect.

Although the game can be played online, it is designed for meetings, to be played by people located in the same room. The game serves as facilitator for participation and gets people talking to each other, very actively as we found out during the meeting. The game is not designed as standalone solution.

The online game version should extend the dialog started during the meeting and provide opportunities for others to chime in at any point (we’re working on it, almost there).

Further readings:

Bike with Google

It’s all over the (map/bike) news: Google added the long anticipated Bicycling layer to Google Maps, which shows bicycle facilities and enables bicycle directions. Excellent!

It’s a great move for bicycle advocacy and helps increasing awareness about cycling as serious urban transportation mode. According to that blog post on the Google Lat Long Blog, the Google Maps engineers developed some solid bicycle routing algorithms. The data is apparently coming from Rails-to-Trails, public sources and Google’s Street View and Imagery. In the areas I viewed, all data was (c) Google, which is, after we saw the parcel layer (c) by Google, not surprising at all.

Nice side effect: bicycling is available in Google Maps API V3 (which recently made it into OpenLayers).

Even though the bicycle layer looks pretty impressive at a first glance – lot’s of green lines in the Boston area – it’s clearly in beta stage. For my daily commute it suggested three basically realistic alternatives, only with 1 to 3 blocks detours from the route I usually take. My usual route includes going up a quiet one-way street which Google doesn’t suggest, instead it would send me to the car packed 4-lane highway.

Playing around with the marker, the routes get a little funkier:

  1. It’s not the shortest/quickest route. Bike trails are clearly weighted very high in the algorithm.
  2. You’re allowed to ride through the Boston Common, but not through the Public Garden as the algorithm suggests; missing restriction.
  3. The shown bike facilities in Cambridge are pretty messy and Boston looks way better than it actually is – Mass Ave preferred for bicycling? That’s one of the craziest streets you could possibly ride on.
  4. At Kendall the algorithm clearly gets confused with some turn restrictions and would send you up and down the street.

Despite those few glitches, Google has done an amazing job in introducing bicycle transportation to Google Maps – very exciting!

Wrapping up Gov 2.0 Camp New England

Gov 2.0 Camp New EnglandI had a fantastic time yesterday at Gov 2.0 Camp New England and enjoyed the event a lot. In particular I was very impressed by the interest of the public sector. I wouldn’t say it’s typical that an event, held on a Saturday, which happens to be the first nice spring day in Boston, attracts so many government employees, ranging from the governor’s office to local town administrations, and affiliated organizations. Throughout the event you could feel the commitment to work on better and more inclusive governance at all levels.

My personal highlight was the first session I attended, about Open 311. It’s clearly not my core area, but I’m interested in the current development and felt that I learned a lot in that session. One of the most interesting points during the discussion was the evaluation of “Resistat”. Resistat is an initiative to include residents in 311 statistics in Somerville. It works very simple: a mailing-list facilitates communication and statistics and results are sent out to residents as powerpoint files. Not rocket science, but it’s enough communication technology that even though only about 25% of involved residents have been to in-person meetings, 85% of them say that they feel better engaged with their local government (by sending powerpoints to a mailing list, it’s as simple as that!). Anyway, great insights when talking about the “town hall meeting divide”, can’t wait to read the entire study about the program.

Our session about Open Data Strategies was “merged” together with I think 5 other sessions that had the word “data” in the title. Unfortunately it didn’t really work out as we intended it and the discussion went somehow all over the place. Better luck next time I guess.

The last session I picked was all about data visualization. Two young IBM researchers showed and demoed amazing data visualization tools – Many Eyes and sense.us among them. Their latest project, called “IBM Visual Bill explorer”, should make it easy for citizens to visually explore and analyze legal texts. Tremendously valuable when trying to understand or to find potential pitfalls in 1000+ pages documents written by lawyers, as bills usually are.

On a side note: I had absolutely no idea that IBM is running such a great research department. Where are marketing departments when you really need them?

Happy hour – free beer was involved – went straight to talking GIS and Open Source. As it turned out, there are very similar problems across gov agencies (surprising, huh!). One notion during the discussion was, that, instead of throwing money individually at our problems, why not join forces and contribute to and customize Open Source projects where we all benefit from? Interesting thought, will be continued…

Python Flickr API geo-search example

I just started using the wonderful flickrapi Python interface for, well, searching Flickr for geocoded photos around given locations. It’s fairly easy to use and does most things for you. You start with a Flickr API object…


import flickrapi
api_key = '1234567890'
flickr = flickrapi.FlickrAPI(api_key, cache=True)

…replace some dots with underscores in the Flickr API methods…


photos = flickr.photos_search(tags='boston', lat='42.355056', lon='-71.065503', radius='5')

…and loop through the parsed results…


for photo in photos[0]:
	print photo.attrib['title']
	photoLoc = flickr.photos_geo_getLocation(photo_id=photo.attrib['id'])
	print photoLoc[0][0].attrib['latitude']
	print photoLoc[0][0].attrib['longitude']
	photoSizes = flickr.photos_getSizes(photo_id=photo.attrib['id'])
	print photoSizes[0][1].attrib['source']

…done.

The above code example lists title, latitude, longitude and thumbnail-source of photos found in a 5km radius search around the Boston Common.

Crowdsourcing bicycle routes

If I had to think of a solution to start creating a bicycle routing system, I’d do exactly what The San Francisco County Transportation Authority has done: create smart phone apps, gather information where cyclists are riding, data mine those tracks and build route suggestions on top of that knowledge.

Bicycle routing is in my opinion far more complex than car routing. Car routing is mostly based on well known and documented rules, also known as road traffic regulations. Mix in estimated traffic figures, average speeds and fuel consumptions and you get pretty decent car directions.

For cyclists, a similar rule set exists, but it’s maybe a little more, let’s call it, elastic. Cyclists use short-cuts, turn where cars can’t, go against traffic, ride through parks and on poorly documented trails. High traffic doesn’t mean slowdown for cyclists. They ride by on the bike lane on the right side of a traffic jam at almost the same speed as without traffic. But high traffic creates a security risk some cyclists aren’t comfortable with taking and rather choose a different route.

A perfect route from A to B for speedy messengers doesn’t necessarily mean it’s also an ideal route for kids. For your daily commute you probably pick another route than for weekend rides, even though it connects the same points.

Bicycle routing criteria is manifold, sometimes psychological, hard to measure and to quantify. Researching how cyclists are going, for what purpose and under what conditions, is a very smart way to get started on that topic.

Re-projecting vectors in JavaScript

I know, it eventually all boils down to maths. But it still blows my mind that you can re-project geographic features on-the-fly with a few lines of JavaScript in a web browser.

How?

There is this great library PROJ.4, that does everything you’d ever want in terms of cartographic projections. A few smart people have ported PROJ.4 to JavaScript, called Proj4js then.

Proj4js works great in combination with OpenLayers, a popular JavaScript web mapping framework, and allows on-the-fly projections between any spatial reference systems browser applications.

<script src="proj4js-compressed.js"></script>
<script src="http://openlayers.org/api/OpenLayers.js"></script>

Define the spatial reference you’re planning to use. Check Spatial Reference for the exact projection parameters and include them in your code.

Proj4js.defs["EPSG:26986"] = "+title=Massachusetts Mainland NAD83 +proj=lcc +lat_1=42.68333333333333 +lat_2=41.71666666666667 +lat_0=41 +lon_0=-71.5 +x_0=200000 +y_0=750000 +ellps=GRS80 +datum=NAD83 +units=m +no_defs";

Adding all desired projections to the OpenLayers script…

projOSM = new OpenLayers.Projection("EPSG:900913");
projWGS84 = new OpenLayers.Projection("EPSG:4326");
projMassGIS = new OpenLayers.Projection("EPSG:26986");

map = new OpenLayers.Map ("map", {
	maxExtent: new OpenLayers.Bounds( -20037508.34, -20037508.34, 20037508.34, 20037508.34),
	maxResolution: 156543.0399,
	units: 'm',
	projection: projOSM,
	displayProjection: projWGS84,
	allOverlays: false
});

osm = new OpenLayers.Layer.OSM(
	"OpenStreetMap",
       "http://tile.openstreetmap.org/${z}/${x}/${y}.png"
);

openspace = new OpenLayers.Layer.WFS("Open space", "http://giswebservices.massgis.state.ma.us/geoserver/wfs", {
	typename: "massgis:GISDATA.OPENSPACE_POLY"
}, {
	projection: projMassGIS
	attribution: "<a href='http://www.mass.gov/mgis/'>MassGIS</a>"
});

…results in an interactive map with MassGIS Open Space WFS vector features overlayed on an OpenStreetMap base layer, using WGS84 lat/lon as display coordinates.

On a sidenote: OpenLayers comes with a Python proxy to retrieve information from remote servers via an XMLHttpRequest. Here is a good how-to get Python play well with IIS 6 on Windows Server 2003, which was quite useful.

Don’t forget to add the domains you’re trying to access to the Python proxy. For MassGIS you would add following string for instance:

allowedHosts = ['giswebservices.massgis.state.ma.us']

Heating up SVG

Last week I came over Raphaël, a great JavaScript library for vector graphics visualizations, and I started playing around with maps and SVG again. Long time no see!

To bring some map content from ArcMap to Raphaël I used the VBA Macro I wrote 4 years ago in ArcMap. It still does the job and gives me clean vector graphics the way I want them. I couldn’t find a decent SVG export option for QGIS, although there are some efforts to improve that kind of functionality.

AsSVG, a Python geoprocessing script for ArcGIS is pretty good too. It provides some nice export options, such as pick style and data attribute fields, and I actually ended up using it a lot.

However, it’s 2009 and there are other ways available for sharing code then just providing a plain text file. So I ended up wrapping a bitbucket repository around it. Just in case if somebody is interested in working on or improving the script…