How not to test changes …

Yesterday I found out the hard way that the neighborhood POI frontend did no longer work, and must have been broken for quite a while already.

Why didn’t I spot this in testing? Well …

  • The problem was a typo in the OCitysMap renderer backend code, not in the frontend
  • I tested the change back then locally
  • But due to another typo in the test setup the local config for the POI frontend was copied to the wrong folder, and so not actually used
  • So the frontend fell back to default settings for the rendering host, and that happened to be the public, not the local server
  • Meaning that when testing the change I actually tested the frontend, which didn’t really change, against the not yet changed public render backend
  • As test results looked good I pushed the changes, pulled them on the public server, but didn’t test there once more
  • So the typo in the renderer went unnoticed as I actually didn’t test the local instance of that, and after the push/pull the public instance was broken, too 🙁

Renderer code and local test setup are fixed now, so something like this should hopefully not happen again.

Now I need to work on making the alternative frontend send email notifications about rendering errors to me, like the main frontend already does, so that a failure like this can’t go unnoticed for this long …

130K – Status updates


130000+ maps rendered

Some day in September my MapOSMatic instance crossed the line of 130K maps rendered, since it started in May 2016. At this rate it will get to 150K even before it’s fourth anniversary 🙂


Database reimport

I had to do a database re-import recently as it had turned out that some data was missing, probably due to recent general problems with minutely changesets on the OSM side. At that point I also took the opportunity to upgrade osm2pgsql and the import style, so that the database is now ready for v5 releases of the OpenStreetMap Carto style. Older styles should continue to work as before with this.


Email setup

So far the MapOSMatic setup used my GMail account for sending out error notifications to me, and “your map is ready” notifications.

I now switched to running my own mail server instead, so reducing the number of external dependencies and increasing privacy a little bit.

During this I discovered something embarrassing: looks as if for years, ever since I introduced the feature, “your map is ready” notifications were actually never sent to the notification address given by users, but to the server administrators address (so mine) only. I never really noticed as I used my own address for testing anyway … this is now fixed, too.


Style updates

I have been a bit sloppy with style upgrades in recent month, but I’m going to catch up with this again. For a start I’m now going over the style installation scripts in my MapOSMatic Vagrant test project, once that’s finished and all tests pass I will upgrade styles on the public instance, too. This will affect:

  • The OpenStreetMap Carto style
  • The Humanitarian style
  • The Belgian style
  • The OpenArdenne style

and probably a few others, too.

I’m also still on the hunt for more open source MapnikXML and CartoOSS styles to support, so if you know of any that I’m missing that is based on the osm2pgsql schema, please let me know.

Styles that use imposm instead of osm2pgsql may also be supported in the future, but right now this is not possible due to disk space constraint on the current server.


What’s next?

Once I’m done with bringing all styles up to date I plan to gradually increase the maximum area size that can be rendered. Currently this is limited to 20x20km², but there’s no real technical reason for this. The limit is just there to put a limit to rendering time.

As all render requests are processed by a single queue one by one so far, this was meant to keep the total queue waiting time low, together with a hard 60 minute limit after which render jobs are killed.

My plan is to set up at least two separate rendering queues for the future, a “quick” and a “slow” queue. Based on area covered, base and overlay style choices, and single vs. multi page layout, the system will try to predict whether a requested render job will be fast or slow, and will put it in the quick or slow queue accordingly.

This way the typical quick requests that only take a few minutes or less to render will no longer have to wait for e.g. multi page requests with many pages, and the one hour limit can be lifted a bit for the slow queue. At the same time I can lower the time limit for the quick queue, so that in cases where the render time estimate turns out to be wrong a request can be canceled early, and be moved to he slow queue for re-rendering.

Once that’s done I will then probably focus on the paper size dialog once more, finally making it possible to choose a specific map scale there, so that things like “I want a 1:10,000 map on DinA2 paper” will finally become possible.

Another thing I’m working on on the side is to try to create — or extend one of the existing — Mapnik symbolizers for rendering arcs. This would allow for rendering things like the camera viewing angles in the “Surveillance under Surveillance” overlay — which right now uses the Python Cairo Bindings directly instead of Mapnik — or like the nautical navigational light visibility arcs on OpenSeaMap — which uses its own custom renderer implementation for this.

130K – Status updates


130000+ maps rendered

Some day in September my MapOSMatic instance crossed the line of 130K maps rendered, since it started in May 2016. At this rate it will get to 150K even before it’s fourth anniversary 🙂


Database reimport

I had to do a database re-import recently as it had turned out that some data was missing, probably due to recent general problems with minutely changesets on the OSM side. At that point I also took the opportunity to upgrade osm2pgsql and the import style, so that the database is now ready for v5 releases of the OpenStreetMap Carto style. Older styles should continue to work as before with this.


Email setup

So far the MapOSMatic setup used my GMail account for sending out error notifications to me, and “your map is ready” notifications.

I now switched to running my own mail server instead, so reducing the number of external dependencies and increasing privacy a little bit.

During this I discovered something embarrassing: looks as if for years, ever since I introduced the feature, “your map is ready” notifications were actually never sent to the notification address given by users, but to the server administrators address (so mine) only. I never really noticed as I used my own address for testing anyway … this is now fixed, too.


Style updates

I have been a bit sloppy with style upgrades in recent month, but I’m going to catch up with this again. For a start I’m now going over the style installation scripts in my MapOSMatic Vagrant test project, once that’s finished and all tests pass I will upgrade styles on the public instance, too. This will affect:

  • The OpenStreetMap Carto style
  • The Humanitarian style
  • The Belgian style
  • The OpenArdenne style

and probably a few others, too.

I’m also still on the hunt for more open source MapnikXML and CartoOSS styles to support, so if you know of any that I’m missing that is based on the osm2pgsql schema, please let me know.

Styles that use imposm instead of osm2pgsql may also be supported in the future, but right now this is not possible due to disk space constraint on the current server.


What’s next?

Once I’m done with bringing all styles up to date I plan to gradually increase the maximum area size that can be rendered. Currently this is limited to 20x20km², but there’s no real technical reason for this. The limit is just there to put a limit to rendering time.

As all render requests are processed by a single queue one by one so far, this was meant to keep the total queue waiting time low, together with a hard 60 minute limit after which render jobs are killed.

My plan is to set up at least two separate rendering queues for the future, a “quick” and a “slow” queue. Based on area covered, base and overlay style choices, and single vs. multi page layout, the system will try to predict whether a requested render job will be fast or slow, and will put it in the quick or slow queue accordingly.

This way the typical quick requests that only take a few minutes or less to render will no longer have to wait for e.g. multi page requests with many pages, and the one hour limit can be lifted a bit for the slow queue. At the same time I can lower the time limit for the quick queue, so that in cases where the render time estimate turns out to be wrong a request can be canceled early, and be moved to he slow queue for re-rendering.

Once that’s done I will then probably focus on the paper size dialog once more, finally making it possible to choose a specific map scale there, so that things like “I want a 1:10,000 map on DinA2 paper” will finally become possible.

Another thing I’m working on on the side is to try to create — or extend one of the existing — Mapnik symbolizers for rendering arcs. This would allow for rendering things like the camera viewing angles in the “Surveillance under Surveillance” overlay — which right now uses the Python Cairo Bindings directly instead of Mapnik — or like the nautical navigational light visibility arcs on OpenSeaMap — which uses its own custom renderer implementation for this.

Server performance issues, new server coming soon …

Update: new server not fully ready yet, while old one falls apart. Hope to have everything ready for switchover by tomorrow, June 30th


My MapOSMatic server ran into some performance issues last week 🙁

This already started on Monday, but being on vacation away from home last week, and mostly offline, I only really noticed this late Thursday, and couldn’t do much about it before today.

Service should mostly be restored by now, but there were a few casualties:

  • contour line and hillshade layers are unavailable for now
  • there will be no further OSM data updates on this machine, so latest changes will not show on generated maps

I did not put too much effort into bringing the server back into a fully working state as I had planned to replace it with a slightly more powerful (and at the same time a bit more affordable) machine by the end of the month anyway.

So for now a few features will be missing, but by end of the week latest everything should be fully working again as before, plus some small additional things, when I switch over to the new server.

Right now the new machine is still busy doing the initial full OSM planet imports for the main and the waymarked trails databases, after that I’ll have to move over the full history of rendered jobs, and perform some testing.

My most optimistic estimate, if nothing goes wrong, would be that the new server can take over by tomorrow, Tuesday June 23rd in the european evening …

Coming soon: multi file uploads

During the Karlsruhe Hack Weekend hosted by Geofabrik, I made it possible to upload more than just one GPX track or Umap export file per MapOSMatic map render request. The screenshot shows a preview with three GPX tracks, colored in red, green and blue.

align=right

The file upload user interface has also changed slightly: instead of separate form tabs for GPX and Umap files, there’s now just one form tab where multiple files can be uploaded. The file types will be auto detected now so there’s no need for multiple upload tabs anymore.

The code still needs some minor polishing, but all the major steps are in place, so it will probably go live on the public instance tomorrow …

Data area bounds revisited

Do you remember my earlier post on data area bounds ?

Turns out I was a bit too naive on believing that the largest administrative bounds polygon in an extract always equals what the extract is actually supposed to be containing.

This assumption can be false for many different reasons:

  • there may be a larger admin polygon that just happens to be part of the extract as it intersects with its bounding box, but extends far beyond it
  • the largest polygon may not actually be included as it’s extending beyond the bounding box slightly, e.g. due to the bounding box being based on an older version of this border polygon
  • the extract may just be a real rectangular area extract after all

This sometimes lead to a wrong admin polygon being shown as assumed data bounds, often one that only represents a small subset of the actual data.

So the bounds logic has now been changed to only assume that the largest contained admin polygons bounding box width and height are both witin 75% and 110% of the total data bounding box width and height. Otherwise the bounds shown on the slippy maps will just be that of the full data bounding box.