Archive for the 'Geotech' Category

Quantum GIS and Homebrew, Pt. 1

A few weeks ago I decided to give Homebrew as package manager a try and move some of my projects to Mac OS X as development platform. Until now I really liked the advantages of using Ubuntu as virtual machine, have development dependencies separated from my day-to-day system, safely try stuff out and be able to rollback to a snapshot if something goes terribly wrong. Sharing hardware resources is obviously the drawback of given setup, which basically made me think about alternatives.

Homebrew is great so far: very easy to setup, integrates well with Mac OS X, large repository and a great “Formula” system to add your own packages.

When it comes to GIS applications and libraries on Mac OS X, William Kyngesburye’s site is one of the best resources you’ll find out there. I used his’ downloads quite a bit. But with the move to Homebrew, I also had the intention to manage my GIS packages with it. Although Homebrew doesn’t provide user friendly double-click-to-install packages, like William does, its Terminal commands are simple enough:

$ brew install mypackage

Drawback of having Homebrew manage my GIS dependencies was, as I noticed immediately, that William’s Quantum GIS package didn’t work any longer. So I decided to build Quantum GIS on my own, from source, against dependencies managed by Homebrew. The entire build process is very well documented on QGIS’s wiki. Using Homebrew for getting all dependencies is as simple as:

$ brew install python --framework --universal
$ brew install postgis qt gdal geos proj \
sqlite libspatialite libiconv gsl pyqt qwt cmake

I edited the existing SIP-Formula slightly (added a “–universal” option) to meet QGIS’ requirements on Mac OS X.

$ brew install sip --universal

Next step is to configure and make QGIS. Some paths and libs weren’t found automatically by cmake, so I had to specify them:

$ cmake -D CMAKE_INSTALL_PREFIX=~/Applications \
-D CMAKE_BUILD_TYPE=Release \
-D CMAKE_BUILD_TYPE=MinSizeRel \
-D WITH_INTERNAL_SPATIALITE=TRUE \
-D QWT_LIBRARY=/usr/local/lib/libqwt.dylib \
-D QWT_INCLUDE_DIR=/usr/local/include \
-D GEOS_INCLUDE_DIR=/usr/local/include \
-D GEOS_LIBRARY=/usr/local/lib/libgeos_c.dylib \
-D GDAL_INCLUDE_DIR=/usr/local/include \
-D GDAL_LIBRARY=/usr/local/lib/libgdal.dylib \
-D GSL_INCLUDE_DIR=/usr/local/include \
-D GSL_LIBRARIES=/usr/local/lib/libgsl.dylib \
-D POSTGRES_INCLUDE_DIR=/usr/local/include \
-D POSTGRES_LIBRARY=/usr/local/lib/libpq.dylib \
-D PYTHON_EXECUTABLE=/usr/local/bin/python2.7 \
..
$ make
$ make install

…produces the QGIS.app in your ~/Applications folder.

The Homebrew Formula for QGIS that I’ve started, doesn’t make a successful QGIS build yet. It fails somewhere at

Linking CXX shared library libqgis_analysis.dylib
[ 33%] Built target qgis_analysis
make: *** [all] Error 2

…although Homebrew is basically building against the same dependencies as my “manual” steps did – working on it.

So far so good. However, the bad news, maybe just in my case, is that QGIS seems to be linked to Mac OS 10.6 original Python version 2.6.1 and throws following error at startup:

Couldn’t load SIP module. (note: SIP is installed in /usr/local/bin)
Python support will be disabled.

Traceback (most recent call last):
File “”, line 1, in
ImportError: No module named sip

Python version:
2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)]

Major downer here is the second line, “disabled Python”, which means most plugins won’t work. This is major since one of Quantum GIS’ strength is its plugin-architecture.

To be continued…

Snowboard traces in Vail

We had a blast last week visiting friends in Denver, Colorado. For three days we went up to Vail, which is one of the best skiing regions I’ve ever been too – great slopes and awesome back country terrain!

All mine

The tiny green geonerd sitting on my shoulder made me turn my GPS on while being out on the mountain. Carrying the GPS receiver in the pocket of my jacket worked surprisingly well and resulted in about 160km of neat slope and lift traces. Here they are, shown as KML overlay on a Google Map…

…or here, uploaded to OpenStreetMap, waiting to be mapped and tagged for OpenPisteMap.

Distortions

An interesting piece – Medusa, 2001, by William Kentridge – that we recently saw here at MassArt, reminding me of map projections and how distorted and often misunderstood the pictures are that we are drawing of our planet.

R visualizations

New Year’s resolution: harvest some knowledge from those colleagues who are digging through R. Apparently it’s not only an excellent tool for number crunching, but also can be used for neat geographic data visualizations.

Visualizing Facebook Friends: Eye Candy in R

Facebook visualization using R

Topology rules with GeoDjango

I’m still very new to GeoDjango and the Django world in general, but I have to say that, based on what I’ve seen so far, I’m very impressed by how easy, fast and clean you can build powerful geographic web applications with it.

Django 1.2 is the current version and one thing that caught my eye in the release notes was Model validation. Among other benefits, model validation allows you to “perform custom validation on your Model” everytime a user wants to save data through a ModelForm. Translated to GeoDjango, that basically means that one can access the unleashed power of PostGIS (or the functionality of your supported geo-database of choice) to validate geometries and provide feedback for users through a nice web interface. Best part of it: it’s all done only by a few lines of additional code in your data model.

I’m not going into much detail how Django’s model validation works, you should check out the documentation if you’re interested, but here’s a quick example of creating something like a simple topology rule with GeoDjango:

Let’s say I have users entering and modifying point data and need to make sure that each point falls inside a polygon from another layer.

# models.py
from django.contrib.gis.db import models
from django.core.exceptions import ValidationError

class Poly(models.Model):
	geometry = models.PolygonField()
	objects = models.GeoManager()

class Point(models.Model):
	geometry = models.PointField()
	objects = models.GeoManager()

	# simple topology rule
	def clean(self):
		# find Polys that contain the Point
		pq = Poly.objects.filter(geometry__contains=self.geometry)
		if not pq: # no Poly has been found
			raise ValidationError('Point is not within a Poly.')

That’s it. You only need to add a custom clean() method and put your validation rules there. Those few lines of additional code check if the new point falls inside a polygon and tell the user if it’s not through the Django admin interface.

GeoDjango Model Validation

I’ve posted the project on bitbucket if you want to give it a shot. Add a little more PostGIS extravaganza and you can have a pretty sophisticated set of topology rules in your web application, bundled with the great capabilities Django apps offer.

GeoDjango on a Windows server

Deploying GeoDjango projects on a Windows server is obviously possible, but not necessarily the most seamless process one can think of. To get started, there are some great tutorials – at the django project site and here, with some more information about the correct PATH settings – available, that guide you step-by-step through a GeoDjango installation.

Below are some things that I did in addition, did differently, or simply couldn’t do because I never got them to work.

The basics: Django and IIS

IIS is was the web server of choice on the Windows machine in question. Serving Django through IIS is of course doable, there is even a dedicated page on the Django project about it. However, I totally failed in getting Django to work well with IIS. It seemed painfull and eventually made me switch the entire web server to Apache. Django integration with WSGI under Apache is a piece of cake compared to PyISAPIe and IIS if you ask me, up and running in less than 10 minutes.

Virtual Environments

Virtual Environments are great when developing with Python. They give you separated Python installations for every project you work on and avoid interferences with dependencies and installed packages across your projects. Virtual environments is something I’ll really miss on Windows, because I never got it to work correctly. Virtualenv basically creates the correct folder structure and activates a Virtual Environment to work in, but for some reason dependencies in my projects were attempting to access the main Python installation. Which obviously defeats the purpose of Virtual Environments.
What I ended up doing was, using one Django installation and being extremely careful about what goes there. Basically I have to manually validate, check and test that new packages for new projects don’t mess with existing once. Painful, I tell ya.

win-psycopg headache

I ran into serious troubles with psycopg 2.2.1 (the PostgreSQL adapter for Python). My project wouldn’t connect to the PostgreSQL database server at all. I found this entry at stackoverflow, which describes exactly the same issue. The posted solution worked for me too: use psycopg 2.0.14 instead of 2.2.1, no problems so far.

Multiple Django projects in one Apache/WSGI instance

This was the most annoying issue: turned out that only one project at a time was working on the server. That might be an option for the iPhone, but it clearly was unacceptable in my case. The problem caused by the “Python for Windows extensions” is documented in this thread, and my solution, similar to the one posted at the end of the thread, was to install pywin32-212 instead of pywin32-214.

Lessons learned

If you have a choice, get a Linux box to deploy GeoDjango. Anything but .NET seems painful to serve on Windows.

Boston mapping folks!

… join the

Boston OSGeo Enthusiasts Meetup

next Tuesday, May 18th, 6:30-8pm

at the Bocoup Loft (319 A St, Boston, MA 02210)


View Larger Map

… and meet other Open Source mapping software users and developers, share experiences, learn about tools, discuss projects and developments.

To RSVP, please just add your name to the Doodle.

For more information and to participate in planning the meetup, please join the mailing list and follow the May meetup thread.

See you there!

Playing games

Last week has been pretty intense. My department finalized a, for a regional planning agency, rather unusual project and ran a couple of community meetings during the week. What happened in those meetings wasn’t the typical PowerPoint presentation followed by a Q&A. People, mostly of younger age, played a 3D video game and participated that way to Boston’s Chinatown Masterplan process.

Emerson students created the excellent video below, that captures the very core of the project in about 3 minutes:

I only got during the final phase of the project involved, helped testing the game, setting up the website and organizing our community meetings, but it was an absolutely exciting experience to take part in. Especially the first community meeting, hosted for Chinatown residents, stood out with an highly energized atmosphere.

The game helped in the meeting as ice breaker, facilitated engaging discussions about the neighborhood and provided a fun environment for the audience to work with. We all were somehow surprised when we actually saw game mechanics kicking in. At the point where scores and winners are announced, people started cheering, got all excited, compared scores and discussed strategies why one couldn’t complete all tasks or what would have been a better way to play.

The good part: it all had serious, real world backgrounds – well, real world compressed into 30 minute tasks. However, the game deals with existing challenges of Chinatown residents. The game allows players to walk in somebody else’s shoes through Chinatown and complete tasks as another person, equipped with different skills and opportunities. It’s an educational game, used to gather feedback on and engage in an urban planning processes.

But it’s still fun to play, and fun is the one aspect I wouldn’t underestimate here. It helps to broaden the audience of the community meeting, attracts especially younger people and provides an experience you wouldn’t expect.

Although the game can be played online, it is designed for meetings, to be played by people located in the same room. The game serves as facilitator for participation and gets people talking to each other, very actively as we found out during the meeting. The game is not designed as standalone solution.

The online game version should extend the dialog started during the meeting and provide opportunities for others to chime in at any point (we’re working on it, almost there).

Further readings:

Bike with Google

It’s all over the (map/bike) news: Google added the long anticipated Bicycling layer to Google Maps, which shows bicycle facilities and enables bicycle directions. Excellent!

It’s a great move for bicycle advocacy and helps increasing awareness about cycling as serious urban transportation mode. According to that blog post on the Google Lat Long Blog, the Google Maps engineers developed some solid bicycle routing algorithms. The data is apparently coming from Rails-to-Trails, public sources and Google’s Street View and Imagery. In the areas I viewed, all data was (c) Google, which is, after we saw the parcel layer (c) by Google, not surprising at all.

Nice side effect: bicycling is available in Google Maps API V3 (which recently made it into OpenLayers).

Even though the bicycle layer looks pretty impressive at a first glance – lot’s of green lines in the Boston area – it’s clearly in beta stage. For my daily commute it suggested three basically realistic alternatives, only with 1 to 3 blocks detours from the route I usually take. My usual route includes going up a quiet one-way street which Google doesn’t suggest, instead it would send me to the car packed 4-lane highway.

Playing around with the marker, the routes get a little funkier:

  1. It’s not the shortest/quickest route. Bike trails are clearly weighted very high in the algorithm.
  2. You’re allowed to ride through the Boston Common, but not through the Public Garden as the algorithm suggests; missing restriction.
  3. The shown bike facilities in Cambridge are pretty messy and Boston looks way better than it actually is – Mass Ave preferred for bicycling? That’s one of the craziest streets you could possibly ride on.
  4. At Kendall the algorithm clearly gets confused with some turn restrictions and would send you up and down the street.

Despite those few glitches, Google has done an amazing job in introducing bicycle transportation to Google Maps – very exciting!

Wrapping up Gov 2.0 Camp New England

Gov 2.0 Camp New EnglandI had a fantastic time yesterday at Gov 2.0 Camp New England and enjoyed the event a lot. In particular I was very impressed by the interest of the public sector. I wouldn’t say it’s typical that an event, held on a Saturday, which happens to be the first nice spring day in Boston, attracts so many government employees, ranging from the governor’s office to local town administrations, and affiliated organizations. Throughout the event you could feel the commitment to work on better and more inclusive governance at all levels.

My personal highlight was the first session I attended, about Open 311. It’s clearly not my core area, but I’m interested in the current development and felt that I learned a lot in that session. One of the most interesting points during the discussion was the evaluation of “Resistat”. Resistat is an initiative to include residents in 311 statistics in Somerville. It works very simple: a mailing-list facilitates communication and statistics and results are sent out to residents as powerpoint files. Not rocket science, but it’s enough communication technology that even though only about 25% of involved residents have been to in-person meetings, 85% of them say that they feel better engaged with their local government (by sending powerpoints to a mailing list, it’s as simple as that!). Anyway, great insights when talking about the “town hall meeting divide”, can’t wait to read the entire study about the program.

Our session about Open Data Strategies was “merged” together with I think 5 other sessions that had the word “data” in the title. Unfortunately it didn’t really work out as we intended it and the discussion went somehow all over the place. Better luck next time I guess.

The last session I picked was all about data visualization. Two young IBM researchers showed and demoed amazing data visualization tools – Many Eyes and sense.us among them. Their latest project, called “IBM Visual Bill explorer”, should make it easy for citizens to visually explore and analyze legal texts. Tremendously valuable when trying to understand or to find potential pitfalls in 1000+ pages documents written by lawyers, as bills usually are.

On a side note: I had absolutely no idea that IBM is running such a great research department. Where are marketing departments when you really need them?

Happy hour – free beer was involved – went straight to talking GIS and Open Source. As it turned out, there are very similar problems across gov agencies (surprising, huh!). One notion during the discussion was, that, instead of throwing money individually at our problems, why not join forces and contribute to and customize Open Source projects where we all benefit from? Interesting thought, will be continued…