Using MSR MapCruncher

Documentation Version 3.0.0

 

Contents

Purpose. 2

HOWTO.. 3

Advanced features. 10

Editable bounding regions. 10

Multiple source maps in a single layer 10

Multiple layers. 12

Automatic Legend Extraction. 14

Transparent color knockouts. 15

Sharing options. 18

Troubleshooting. 18

Why does my map look tortured?. 18

No matter how many points I add, I always have to fine-tune a little. 19

How many correspondence points do I need?. 21

Tips and Other Features. 22

Supported source map types. 22

Error distances and the error wand. 23

Nudge map. 23

Source Info. 24

Recursive registration. 24

Update button. 24

Insets. 24

Multiple page source documents. 24

Source maps from URIs. 24

Scripted rendering. 25

Limitations. 25

Schematic (not to-scale) maps. 25

Approximate reprojection. 25

High zoom levels cause long UI stalls. 26

Browser support 26

Release Notes. 26

Next Steps. 27

 

Purpose

MapCruncher is a tool designed to quickly convert existing maps into an online format that’s as fast and easy to use as Virtual Earth (VE). MapCruncher transforms maps into a common Mercator projection used by VE. By putting a large collection of maps into a common projection, users can mix-and-match maps to use them in new ways. MapCruncher will take your input map (such as a PDF), and generate a collection of small, correctly-aligned tiles that can be viewed in VE.

There are a few things that MapCruncher is not. MapCruncher is not designed to handle schematic (not to-scale) maps. MapCruncher is not geographically perfect: it uses an approximation to transform maps, and thus sacrifices some precision in feature location. MapCruncher is not a tool for drawing maps; instead, it is designed to convert existing maps into a web-viewable form.

HOWTO

1.      Install MapCruncher from http://research.microsoft.com/mapcruncher/.

2.      Find a map you want to crunch in a supported file type. Be aware that maps may be under copyright, which restricts your ability to republish them.

3.      Download your map to your local disk.

4.      Launch MapCruncher. It begins by creating a New Mashup document.

5.      Select FileAdd Source Map…, and select a source map from the local filesystem.

6.      Create the first three correspondences as follows:

a.       Navigate around the source map to find an identifiable feature. Drag the mouse to pan. Double-click to center and zoom, or use the mouse wheel to zoom in and out. Place the feature under the crosshairs.

b.      Navigate around the Virtual Earth map to find the corresponding feature.

c.       (optional) Label the point.

d.      Click Add.

Repeat this process to establish three correspondences. (See How many correspondence points do I need?).

7.      Now you have a rough registration between the maps. The procedure for adding more correspondences is easier, and each correspondence you add reduces error.

a.       Click Lock.

b.      Find a new point on the source map. Notice that as you manipulate the source map, the VE map moves along with you.

c.       Click Unlock.

d.      Fine-tune the position on the VE map.

e.       Label and add the new point.

Repeat this process until you find that new points do not seem to require very much tuning.
Tip: Error distances and the error wand
Troubleshoot: How many points do I need?
Troubleshoot: Why does my map look tortured?

8.      (optional) Tell the visitors to your page where to find the original maps.

9.      Select the Render button.

a.       Select an output folder where the rendered results should be stored. Point it right at a folder served by your web server, if you like.


b.      Click Start Estimating; MapCruncher estimates the rendered output size and starts rendering tiles.

c.       The maximum zoom level can be adjusted to change output size.

§         MapCruncher guesses a reasonable default zoom level based on your source map’s size.

§         MapCruncher renders in order from far-away tiles to zoomed-in tiles, so you can just ask for a higher zoom level than you require, and simply cancel rendering when you’re tired of waiting for it.

§         To set the zoom level yourself, adjust the Maximum Zoom box. As you adjust the box, MapCruncher previews the chosen zoom level.

§         Each additional zoom level quadruples the number of tiles generated, and so also the time taken to render and the space required to store the tiles. At level 1, four tiles cover the surface of the Earth; street level is around level 19. (Before you spin the zoom much past 19, be aware of this known limitation.) At high zoom levels, estimation can take minutes, and rendering can take hours.

10.   (optional) Use the links to view the rendered tiles inside MapCruncher, or with your web browser.

11.  (if required) Copy the rendered folder onto your web server, and open the SamplePage.html file in a browser.

12.  (optional) Use the SamplePage.html file as a basis to create your own customized page. Be sure to place your customized data outside the folder rendered by MapCruncher, so that if you re-render your maps from MapCruncher, you don’t lose your custom work. Your custom HTML page should point to the automatically generated .crunched.xml file; that way, when you re-render your map, the new contents appear automatically in your custom HTML page.

Advanced features

Editable bounding regions

By default, MapCruncher makes everything beyond the bounding box of the original source map transparent. You can trim out more of the source map to cut out insets, legends, or to make better-fitting boundaries between adjacent source maps in a single layer. The round corner points of the blue bounding region can be adjusted by dragging.

To make more complicated shapes, right-click on a blue edge and choose Add Corner.

Multiple source maps in a single layer

You can specify multiple source maps as part of a single MapCruncher layer. A layer is a single tile set that is always viewed at once. It may comprise a single source map, or multiple source maps. Here, four maps from adjacent counties become one continuous layer:

To add source maps to a layer, right-click on the layer name in the Map Layers pane and choose Add Source Map:

The source maps in a layer overlap in the order they appear in the Map Layers pane. You can drag and drop a source map to reorder it. Selecting the layer name itself will change the Source Map to a preview of the composite layer. (Because it is composed of so many pieces, this preview can take a long time to render. MapCruncher provides a progress bar so you can distinguish between an preview of an undesirable layer and a simply incomplete preview.)

Multiple layers

Rendering transforms all of the source maps in a layer into an atomic unit. However, for some applications, you may wish to provide distinct layers which can be individually activated in the browser.

 

The Add Layer context menu makes a new layer:

Browser performance note: if you don’t need the ability to composite maps dynamically, then incorporate the maps as multiple source maps in a layer rather than as separate layers. The browser must download a separate set of images for every layer, so viewing several layers simultaneously can be unpleasantly slow.

Automatic Legend Extraction

You can tell MapCruncher where legends appear on your source map:

Use Editable bounding regions to clip out the legend:

The generated output page will include a pop-up legend dialog that shows links just to the legends for maps that are currently visible:

Transparent color knockouts

There are five ways that a mashup can include transparency:

§         Regions beyond the bounds of each source map are transparent.

§         Regions inside the source map that are marked transparent (in a .PNG file) or not drawn (in a .PDF file) appear transparent.

§         Regions that you exclude using the Editable bounding regions facility appear transparent.

§         The VirtualEarth JavaScript API provides a facility for making an entire layer translucent.

§         You can define a color set, and then either knock out all regions of those colors, or all regions not of those colors.

To define transparent color knockouts, select a source map and then select the Transparency tab:

Use the source map crosshairs to identify colors to add to the color list (just as you would add a correspondence point), and select Add Color Under Crosshairs. The source map will show the effect immediately:

   

Fuzziness controls how nearly a color must match the selected color to be knocked out; a larger number includes more nearby colors. This can help knock out “almost-white” regions, such as artifacts created by JPG compression.

Halo expands the opaque region by the chosen number of pixels, typically allowing a contrasting neighboring color in the source map to show through. This can help make map features more visible when displayed against various backgrounds. For example, a black source map label may look fine superimposed over a light-colored road map, but be very difficult to read when the background is changed to aerial photography of a dark forest. (Halo is only barely implemented: the entire source map adopts the maximum halo value of any color, and rendering with halos is quite slow. Thus, this feature should be considered for the brave.)

Sharing options

The Mashup Render Options panel has two checkboxes that affect how others can modify or reuse your mashup.

Copy source maps and crunchup data to output folder

This option copies the original source map files plus the .yum file itself to the rendered output folder, which you will typically publish to the web. By doing this, any web visitor that views your mashup can reconstruct it from these original files, and perhaps modify it. If you prefer to only publish the crunched version of your data but keep the source map files (.pdf and .jpg inputs) private, uncheck this box.

Permit composition

This option adds a tag to the MapLayers.crunched.xml file marking it as public domain. This enables others to build mashups that build on your crunched tiles.

Technical motivation: In essence, the .xml file is basically a big, complicated link to your crunched tile set. Ideally, one could build a composed mashup by writing a web page that referred to MapLayers.crunched.xml files on two or more other web servers. Unfortunately, due to security restrictions in modern browsers, such remote references to xml files are disallowed. The second best approach is for the composing site to build one big .xml file that combines the contents of those others, and refers to the tile sets on the other web servers.

Effects on rights: To ensure that this combining process is clearly authorized, this checkbox labels your MapLayers.crunched.xml as in the public domain. Notice that this only marks the xml description file as public domain, not the crunched tiles, .yum file, or the source map images. If you prefer to discourage others from building new mashups that compose with your mashup data, uncheck this box, and perhaps insert a comment to that effect in the MapLayers.crunched.xml file.

Troubleshooting

Why does my map look tortured?

One likely possibility is that the map is schematic (not to-scale), like this Washington DC subway map. In that case, MapCruncher cannot transform the map.

 

Another possibility is that the map is mostly to-scale, but the cartographer has taken a few liberties. We had a hard time registering points on this island until we read this little note next to it:

 

The New York MTA Subway map gave us fits for a while until we noticed this remark in the key:

Another possibility is that you have selected some dramatically incorrect correspondences.

Try checking the “Force Affine” checkbox. This usually straightens the map out enough to see, and you can examine the green error wands to get an idea whether the map is schematic, or the correspondences are just broken.

No matter how many points I add, I always have to fine-tune a little.

If you have to fine-tune a lot, you may have a not-to-scale map.

Perhaps one or two points are incorrectly matched, and throwing the rest of the approximation off. Use the error column, and re-check the correspondences with the worst errors. Replace suspect correspondences.

Make sure you are using correspondence points that are widely separated on the source map. If you start with a dense cluster, MapCruncher has to extrapolate for far-away points, and extrapolation produces bigger errors than interpolation. In this LA Transit map, all of the correspondences are in the city, which leaves features at the edge of the map not tied down:

 

Sometimes the road atlas data in VE doesn’t match the aerial photography. We recommend registering to features in the Hybrid View.

Ultimately, you may never get a perfect match. This can happen due to limitations of MapCruncher’s approximate transformation, or due to errors in the source map itself:

How many correspondence points do I need?

Some source maps, such as aerial photographs or floor plans, are drawn in “conformal projections”, and can be registered with as few as two points.

Other maps require a third point to get pinned down. In the example below, the two correspondences are correct, but the source map is obviously locked in an incorrect position: the Northern border of Washington is not East-West, as it appears on the right. That’s because the chosen source map happens to not be a conformal projection.

Adding one additional point tells MapCruncher how to fix the problem:

We have found that 15 to 20 correspondences seem to be about as many as we need, even on detailed maps such as a county-wide map with street-level detail. Beyond that, other sources of error usually limit the fidelity of the results.

Tips and Other Features

Supported source map types

MapCruncher understands several input file types:

PDF

The highest-quality results are usually obtained from a PDF file. PDF files often contain vector graphics, which look good even when they are rendered at high zoom levels.

JPG, PNG, TIFF, GIF, BMP

These raster formats can be used with MapCruncher, but unless the source map is at high resolution, the results will look poor at high zoom levels. Scanned maps (such as historical maps) are often available as rasters.

WMF, EMF

These file formats seem to be less common, but they can include vector data, and can produce high-quality results.

Error distances and the error wand

When the maps are locked, the correspondence table includes an “error” column that shows how well the approximation could match up your points; it is sorted with the greatest error at the top. If the topmost point has a substantially greater error than those below, this suggests that you may have misidentified that point.

Double-click the suspect point. The green wand leads from the point you selected towards where MapCruncher suspects that the correct point should go. For this point, we thought we had the intersection matched up…

…but you can see that the pin in the source map is a block too far North:

Nudge map

Use Ctrl-Arrows to nudge the source or VE map views under the crosshairs by small steps.

Source Info

On the Source Info tab, you can enter the URL, home page, and description of the source map. This information will be included in the sample output page generated by MapCruncher. By including this information here, it’s easier for users of the site you make to find the original map.

Recursive registration

ViewShow Rendered Layer lets you place your own rendered tiles over the VE map. You can use this function to register one map into VE, and then register a second map to features only visible in your own first map.

Update button

When fine-tuning a correspondence point, you can select the point in the correspondence list, nudge the maps under the crosshairs, and then click Update to adjust the stored correspondence. Update can also be used to rename a point.

Insets

If a source document contains multiple maps that must be registered separately, such as insets, you may Add Source Map multiple times, specifying the same source file repeatedly. Use editable bounding regions to select different parts of each added instance of the map.

 

Multiple page source documents

(for the brave) If your source map appears in a PDF file with multiple pages, you can Add Source Map, then save the .yum XML file, and use a text editor to change the PageNumber attribute.

Source maps from URIs

(for the brave) You can tell MapCruncher to fetch a source map from a URI each time MapCruncher starts. To do so, replace the Document tag in the .yum file with one like this:

        <Document>

          <UriDocument Uri="http://adds.aviationweather.gov/data/satellite/latest_WMC_vis.jpg" PageNumber="0" />

        </Document>

Scripted rendering

MapCruncher can be invoked from the command line with the name of a .yum file to open. If you also supply a –render flag, MapCruncher will render the .yum file and exit when it’s done. Combined with Source maps from URIs, this enables the creation of a scripted pipeline.

Limitations

Schematic (not to-scale) maps.

MapCruncher is not designed to handle schematic (not to-scale) maps. In a to-scale map, MapCruncher can infer from only a few correspondences how all of the rest of the data should be transformed. In a schematic map, the cartographer has taken arbitrary liberty with the world geometry, and no set of correspondences say anything certain about other unlabeled points.

Approximate reprojection.

MapCruncher uses an approximation to reproject to-scale maps from their original projections into VE’s Mercator projection, and thus sacrifices some precision in feature location. Disciplined cartography involves understanding both the relationship between angular measures (latitude and longitude) and the surface of the Earth, and the relationship between those angular measures and the surface of the map (projection geometry). MapCruncher understands neither. Instead of modeling source projection, source datum, VE datum, and VE projection, MapCruncher uses a simple function to directly approximate the relationship between the original map and the VE map.

Mathematically speaking, MapCruncher uses either an affine or a quadratic polynomial to approximate the projection. The quadratic polynomial admits curvature in the approximation, which helps handle the curves that appear in conic projections.

 

Polynomial fitting can only be approximate, because a polynomial can never quite match the trigonometric curves produced by geographic projections.

We have found, however, that the approximation is suitable for a wide variety of maps. Frequently, the source map exhibits more internal error than the approximation. Approximation error is apparent in very detailed maps (maps whose smallest feature is tiny compared to the extent of the map): In the case of the King County Bike Map, the approximation seems to be responsible for errors of up to about a quarter-block.

High zoom levels cause long UI stalls

If you zoom far into a vector map with text, the UI can stall (including a frozen mouse pointer) for quite some time. I theorize that the renderer requests a font at an enormous point size, and rather than just rendering the very corner of that giant letter ‘n’, Windows locks the UI and dutifully allocates an enormous chunk of memory to rasterize the entire font. This is a great strategy for rapidly blting lots of ordinary-sized text, but falls over pretty badly in this corner case. We don’t yet have a way to detect when this will occur and prevent it; sorry.

Update in Version 3.0.0: This problem should no longer occur in PDF files, because FoxIt provides a way to render that doesn’t involve interactions with Windows GDI. That means that this limitation should only affect vector maps in .WMF and .EMF formats.

Browser support

MapCruncher’s JavaScript:

§         is known to work in Firefox 1.5.x on Windows, Linux, and MacOS X.

§         is known to work in Internet Explorer 6 and 7 on Windows.

§         is known not to work on Firefox 1.0.x (MacOS X) and Safari 1.3.x (MacOS X).

IE presentation bug

A bug in IE can cause strange black streaking on transparent tiles, as in this example:

We’re not really sure why, nor do we presently have a workaround.

Release Notes

MapCruncher 3.0 includes the following improvements:

Next Steps

If you make an exciting mashup, please drop us a line at . We’d love to hear how MapCruncher is getting used. Enjoy!