Tools for ArcGIS Editor for OpenStreetMap

This section is divided into three parts: Description of the Geoprocessing Tools, Advanced Tools (for the brave and super geeky) and Description of Editing Tools.

Description of the Geoprocessing Tools & Models

The interaction with the OSM server is accomplished using geoprocessing tools to download, manage, and upload data. To learn more about the role of geoprocessing within the ArcGIS framework please refer to http://resources.arcgis.com/content/geoprocessing.

OSM_on_AGS Toolbox

If you have installed the ArcGIS Editor for OpenStreetMap Feature Service component available with version 2.0 or above, you will see an additional toolbox installed in your ArcMap environment, called "OSMonAGS". The scripts in this toolbox are for running the Feature Service components, and are not for use in ArcMap. This toolbox will not appear if you have not installed the Feature Service component.

OpenStreetMap Toolbox

Geoprocessing tools are provided for the interaction with the OpenStreetMap data, and referenced in models also provided with the OpenStreetMap Toolbox. An additional tool – Create OSM Network Dataset – will appear if you’ve installed ArcGIS Editor for OSM version 2.1.
tools10.png

The tools and models are described below.

Add OSM Editor Extension

If you have downloaded data from OSM using the ArcGIS Editor for OpenStreetMap, your downloaded data is already mapped to the OSM data model and Esri tools. If you want to use or share this downloaded data in another context where the ArcGIS Editor for OpenStreetMap is not installed, you would need to run the Remove OSM Editor Extension tool on the data first to remove this mapping. Conversely, if you wanted to bring data from another environment into ArcMap and upload it to OSM using the ArcGIS Editor for OpenStreetMap tools, you would need to run this Add OSM Editor Extension tool first so the correct connections to the data model can be made.

This is a geoprocessing tool that adds a custom feature class extension to the provided feature class or feature layer. The input parameter needs to follow the data schema as defined by the Download OSM Data tool. The feature class extension is used to trigger OSM-specific editor functionality.

Combine Layers

This tool can be used with other data in ArcGIS, not just OSM data. This tool provides the ability to group layers, by combining any number of input layers into a single group layer. The first parameter, Input Layers, is the collection of layer files that should be combined into a single group layer. The second parameter, Output Group Layer, is the name of the new group layer. If you would like to persist the layer file on disk, the full path where to store the lyr file should be included.

Combine OSM Attributes

If you are working with OSM data in an environment that does not have ArcGIS Editor for OSM installed, and in that environment have added or edited standalone attributes not regularly in the Esri OSM data model, this tool can be used to port those standalone attributes back to the data and then upload back to the OSM servers. In effect, this allows you to map other data into an OSM structure, and could be used for importing data to OSM.

This tool allows you to combine user-defined attribute fields into a field containing the OpenStreetMap tags. The first parameter, Input OSM Layer, is the feature class containing the attribute fields to be combined. The second parameter, Select OSM Attributes, are the attribute fields that can be translated into tag key/value pairs for OpenStreetMap attribution.

The converse of this tool is the OSM Attribute Selector.

Create OSM Network Dataset - version 2.1 only

This tool provides the ability to create a network dataset from a given OSM dataset. This capability requires a configuration file to define the barriers, junctions, turns, and roads. This configuration file can be generated from a script also provided upon installation. For detailed instructions for using this tool in a workflow, see Create a Network Dataset from OSM Data. The first parameter, ‘Input OSM Feature Dataset’, is the path to an OSM-enabled feature dataset. You can obtain an OSM-enabled feature dataset by running the Download, Extract, and Symbolize OSM Data or the Load OSM File tools. The second parameter, ‘Network Configuration File’ is the path to a configuration file that defines the barriers, junctions, turns, roads, and other information for generating the network dataset. Sample files you can use come with the version 2.1 installation. The final parameter, ‘Output Network Dataset’, is the name of the network dataset that will be created. This field is auto-populated after filling out the first parameter, but is editable if desired.
The result of running this tool will be that your OSM feature dataset is transformed into a Network dataset. In addition to the new network feature class that is createed, it will also include a feature class for barriers, turns, junctions, and roads. After generating the network dataset, you can use the ArcGIS Network Analyst extension tools to work with the dataset.

Download OSM Data

This tool allows you to download vector data from the OpenStreetMap server and store it locally inside a geodatabase. The first parameter is the URL for the OpenStreetMap server hosting the data. The data are downloaded from this location. The second parameter is the extent for which you would like to download data. For more detailed information about the concept of extents please refer to the ArcGIS Desktop 10 Web help on Output Extents (http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Output_Extent/001w00000009000000/). Since this a request against the live OpenStreetMap server, the area of interest needs to be smaller than 0.25 square degrees or have less than 50,000 nodes in the server response (per the OSM API - http://wiki.openstreetmap.org/wiki/API_v0.6). The next parameter is optional and allows you to download all references that are mentioned in the initial download request. Due to size and time considerations the initial OpenStreetMap download optimizes the data download. This optimization can result in including only references to data as opposed to the actual data itself. For the majority of editing downloads this behavior is desired but in cases where you would like to request the all-inclusive dataset, an option is provided for the tool to resolve all the referenced data. Caution: checking this option could result in a great number of requests and a longer download time, possibly multiple hours. The fourth parameter is the target feature dataset where you would like to store the downloaded data from the OpenStreetMap server. File and enterprise geodatabases are supported as the download containers.

The result of the download are three feature classes containing the point, line, and polygon features translated from the node and ways from the OSM notation. Relations are stored as a standalone table in the geodatabase and are currently not explicitly exposed to the user.

Download, Extract and Symbolize OSM Data

Download and Symbolize OSM Data is one of the most often used models in the toolbox. Using the 'Download OSM Data' tool and the 'Symbolize OSM Data' models, it is used to make a connection to OSM, download the OSM XML describing the features, and then parse and symbolize those features into a feature dataset you can use in your map document. To know more about the resulting feature dataset, see Details on the OSM Feature Dataset.

PostgreSQL Note
There is a toolset specific for when you want to Download , Extract, and Symbolize OSM data to a feature dataset stored in SDE for PostgreSQL. This different set of models is necessary because of the PostreSQL reserved word for 'natural', which is also a value used in the definition queries for symbolization. If you are downloading into a PostgreSQL SDE repository and do not use the PostgreSQL-specific model, you will see errors when the data is symbolized in the ArcMap interface.

Download OSM Data XAPI

The Download OSM Data (XAPI) tool will download data from an OSM xapi endpoint and converts it into an .osm file (known as a planet file). It also supports querying the OSM data that is downloaded such that only certain features are downloaded. The features to download are defined using a predicate; for example amenity=hospital would retrieve only features where the amenity attribute is set to hospital. After the script finishes running, use the Load OSM File tool to load the newly created .osm file into the map document as a feature dataset.

NOTES:
  • This tool can only be used when the map document's Data Frame uses the Geographic WGS84 Coordinate System.
  • Check that the xapi server in the 'Download URL' parameter is responding - see a list of xapi servers at http://wiki.openstreetmap.org/wiki/Xapi#jXAPI. If the server is not responding, you can choose another server from the list. The trailing forward slash for this parameter is important (e.g., http://open.mapquestapi.com/xapi/api/0.6/ rather than http://open.mapquestapi.com/xapi/api/0.6 - note no forward slash).

Load OSM Diff Files

This tool is used to synchronize your downloaded OSM data with the planet diff files available from the OSM server. Two possible use cases are:
  • You've loaded and symbolized an OSM file into your ArcMap interface, and want to get the latest updates from the OSM server without downloading all of the features again.
  • You want to synchronize your data with the latest updates programatically. The ArcGIS Editor for OpenStreetMap Feature Service uses this capability for this purpose.
When you use this tool, you will be prompted to enter the following parameters:
  • OpenStreetMap planet diff files base download URL: the URL to the planet diff file location. The parameter specifies the base URL for the hour-replicate and minute-replicate directories, but this tool uses both to determine what changes need to be retrieved.
  • Input OSM Feature Dataset: the target feature dataset you've been working with that you want to be synchronized with the OpenStreetMap server data.
  • Start Time for Diff Files (optional): If you want to specify a start time for the synchronization, indicate it here. Time must be provided in UTC and expressed according to ISO 8601 (e.g., 2012-03-04T14:00:00, means March 4th, 2012 at 2 p.m.). If no argument is provided the synchronization will be made for the last hour.
  • Load updates only inside current AOI: Indicates if only new or updated data within the current area of interest for the current dataset with be recorded. Recommended that this be checked.
  • Verbose geoprocessing logging: The parameter specifies the amount of logging information provided by the tool.

Load OSM File

This tool allows you to load stand alone OpenStreetMap XML files also known as planet files. The OSM database extracts do not have a size and feature limitation as the direct download tool and the extract can encompass producer-defined geographic areas. For more information about planet files and custom OSM database extracts please refer to the OpenStreetMap Planet file documentation (http://wiki.openstreetmap.org/wiki/Planet). The result is a feature dataset you can use in ArcMap - to know more about the resultant feature dataset, see Details on the OSM Feature Dataset.

Important: If you want to load an .osm file that encompasses the entire earth, this can take a long time (e.g., many days). For example, loading an .osm file that contains Europe alone took 6.5 days in a test environment. The expanded .osm file was 350 GB in size. Therefore, it is recommended that you load the smallest .osm file that meets your needs. If you are interested in a country, state or county, it is easier to load such .osm files than load the whole planet with plans to zoom in to your country, state, or county of interest.

The first parameter is the location of the OSM file on disk. It is the user’s responsibility to download a planet file of choice from one of the known download locations. The second parameter is the target feature dataset where you would like to store the data from the stand alone OpenStreetMap file. File and enterprise geodatabases are supported as the download containers. The result of the loading process are three feature classes containing the point, line, and polygon features translated from the node and ways from the OSM notation. Relations are stored as a standalone table in the geodatabase and are currently not explicitly exposed to the user.

OSM Attribute Selector

Due to the flexible nature of the OpenStreetMap data with its key/value pairs as attribute information, it can be a challenge to present the information in the normalized way of a relational database. The downloaded OSM data is optimized for editing as well as keeping the number of feature attributes manageable. The geodatabase schema for points, lines and polygons has the following default attributes outlined in the table below.

Attribute Name Attribute Type
OBJECTID ESRI unique ID field
SHAPE ESRI Geometry field
highway OSM Theme type (user defined)
barrier OSM Theme type (user defined)
waterway OSM Theme type (user defined)
railway OSM Theme type (user defined)
aeroway OSM Theme type (user defined)
aerialway OSM Theme type (user defined)
power OSM Theme type (user defined)
man_made OSM Theme type (user defined)
building OSM Theme type (user defined)
leisure OSM Theme type (user defined)
amenity OSM Theme type (user defined)
shop OSM Theme type (user defined)
tourism OSM Theme type (user defined)
historic OSM Theme type (user defined)
landuse OSM Theme type (user defined)
military OSM Theme type (user defined)
natural OSM Theme type (user defined)
geological OSM Theme type (user defined)
route OSM Theme type (user defined)
boundary OSM Theme type (user defined)
place OSM Theme type (user defined)
OSMID Unique OSM ID
osmTags OSM key/value pair storage container
osmuser OSM user (metadata)
osmuid OSM user unique ID (metadata)
osmvisible OSM visibility (metadata)
osmversion OSM version (metadata)
osmchangeset OSM changeset (metadata)
osmtimestamp OSM user (metadata)
osmMemberOf OSM is a member of what relations
osmSupportingNode Has attributes (key/value pairs) (points only)


With this schema some of the core cartographic and analytical functionalities like labeling or computing feature statistics are inaccessible.

An example is if you wanted to create labeling in your map using an attribute such as speed in the Highway Lines feature. Because extracting such information for all features would result in an unwieldy and sparsely populated table, you must run the OSM Attribute Selector tool to get this additional attribute information. This tool allows you to extract keys from the OSM key/value collection field and add them to the selected keys as separate attributes in the feature class.

The first parameter, Input OpenStreetMap Feature Class, should be the feature class from which the OSM tags are to be extracted - e.g., Highways (lines). The second parameter, Select OSM Tag Keys, is the tag keys to extract from the tag collection and store as attributes in the feature class table – e.g., maxspeed. After running the tool, the additional feature information is added to the end of the attribute table for your selected feature class.

The converse of this tool is the Combine OSM Attributes tool.

Remove OSM Editor Extension

If you want to use data downloaded from OSM from the ArcMap environment in another environment where the ArcGIS Editor for OSM tools are not installed, you need to run this tool first. The Remove OSM Editor Extension tool removes the ArcGIS Editor for OSM custom feature class extension from the selected feature class or feature layer. Attempting to open and view the data on machines not containing the ArcGIS Editor for OSM components will result in an error message.

The converse of this tool is the Add OSM Editor Extension tool.

Upload OSM Data

This geoprocessing tool allows you to upload local changes stored inside a geodatabase back to the OpenStreetMap server.
  • The first parameter specifies the URL for uploading the local edit changes. The URL needs to specify a valid OpenStreetMap server instance. The default is to publish to http://www.openstreetmap.org. However, you can also consider publishing to the OSM development endpoint, especially when you are first getting used to publishing to OSM. The development endpoint is http://api06.dev.openstreetmap.org/.
  • The second parameter is the local revision table storing the changeset information that is submitted to the OSM server. This table will be created in your geodatabase when you download the data from OSM or run the Load OSM File tool, and autopopulated as you edit the OSM dataset in ArcMap. Do not edit this table manually - all changes will be done automatically by the software. This table also stores the upload status, i.e. if the data upload was successful or if the upload resulted in conflicts. The conflict editor will use the same table to guide you through the conflict resolution. The table name has the schema of <name>_osm_revision where <name> is the name of the feature dataset where the OSM data resides.
  • The third parameter should be a brief yet descriptive statement about the nature of the local changes. This statement is stored on the server and will be used in the review process of all changes.
  • The fourth parameter 'Bundle Data Upload(s)' is an option that allows you to upload all the changes as one change request to the OSM server. Generally, this reduces the overhead on your client and the server because all changes are uploaded at one time. If you leave the 'Bundle Data Upload(s)' option unchecked, the changes will be uploaded one at a time. The advantage of uploading them one at a time is that if one change has issues, you can then resolve the issue with the Conflict Editor, whereas when all are uploaded in a bundle, if there is an issue with one change then the whole upload will fail. Recommended practice is to attempt to upload as a bundle, and if something in the bundle fails, attempt again with this option unchecked.
  • The fifth parameter is the user name and the password of your editor credentials. To be able to submit changes to the OSM server you need to have a valid editor account. The credentials are part of the upload, and authenticate the submissions against the server instance. An editor account can be easily created by completing the editor sign-up page at https://www.openstreetmap.org/user/new.

Note: The upload geoprocessing tool cannot be run during an edit session. Before running the tool you have to Stop Editing.

Advanced Tools

To enable Advanced tools, open ArcMap (in Windows 7, you'll need to open it as an Administrator) and click on the Toolbox icon to show all toolboxes. At the top level toolbox, right-click and select "Add Toolbox...". Browse to your OpenStreetMap toolbox file and add it.

Once added to the tree, right-click on the OpenStreetMap toolbox and select "Add", and then "Tools...". Browse again to the OpenStreetMap toolbox and click the box next to it. Now you see two sub-toolboxes, and within them the OpenStreetMap tools, including some hidden advanced tools. The advanced tools are described below.

advanced_tools.png

Feature Comparison

This tool allows the user to compare two feature classes to determine the similarity between the two feature representations.
  • Input (source) features: The source feature class assumed to contain the data to be compared to the OpenStreetMap data.
  • Field to store number of intersections: An attribute field in the source feature class to store the number of intersections between the source and the matching feature. The field needs to be of type integer. A number of 0 indicates that the source does not intersect any other feature, suggestiong that the source has no matching entity in the match feature class, making it ok to import. The higher the number of intersections, the more features are geometrically similar and special considerations for imports are required.
  • Field to store intersected feature OIDs: An attribute field in the source feature class to store the enumerations of object IDs of intersected features from the match feature class. The fields needs to be of type string. The list of object IDs is comma separated.
  • Match feature class: A feature class to which to compare the source. In the context of the ArcGIS Editor for OpenStreetMap the match feature class is one of created feature classes containing the OSM data.

Export to OSM File

Export OpenStreetMap feature classes into the OSM XML format.

Description of Editing Tools

The ArcGIS Editor for OpenStreetMap is an extension to the existing editor environment for the desktop. Please be sure to read the introduction to editing data at ArcGIS 10 in the Web help - http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/What_is_editing/001t00000001000000.
Here we will discuss the Attribute Window and the Conflict Editor .

The Attribute Window

As you start editing, the Create Features window lists all of the available feature templates. The choice of templates can be filtered down by geometry type, by layer, and by frequency.
tools20b.png

Using one of the available templates creates a feature with the specified “default” OSM attributes. To set additional attributes (key/value pairs), open the attribute window (tools21.png).

The attribute window for OpenStreetMap is a little different from the default ArcGIS attribute dialog and it aims to guide you in collecting the appropriate tags for each feature. The attribute window has three columns containing the name of the tag in the left most column, the value of the tag in the middle column and potentially a link to the OSM wiki page for further information about the type and the collection rules for each attribute. If the cell value has a combo box it means that a list of suggested values and tag names does exists. The list of values is merely a list of suggestions and user defined entries can be added at any time. Tip: You can modify the suggested list such that a suggested data model - like the Humanitarian Data Model described earlier - could be used for an ad-hoc/emergency data automation effort. Different configuration files can be set in the Editor Options using the OpenStreetMap tab. For more information on how to represent a different data model please refer to Change the Data Model.

tools23.png

Conflict Editor

At the time of the data automation process, the extension checks if the feature is a member of a relation. Deleting a point, line, or polygon can have an effect of changing the relation in which the feature participates. A conflict message will appear telling the user about the modifications and which features are affected by the changes.

A number of things can happen between the time you download data from OpenStreetMap, make the edits locally and then submit the changes back to the OSM server. Other OSM users could be working in the same geographic area and submit the changes at different times. As a result, you might encounter a data state which is different from the time when you initially downloaded the data. In that case, the server might reject the submission and indicate a conflict between the server and the client state. To mitigate the conflict the ArcGIS Editor for OpenStreetMap offers a Conflict Editor - shown below - to help you resolve this situation.
tools25.png

In the top left of the Conflict Editor window is a list of datasets and their noted conflict. In this example there is one downloaded dataset called seattle_osm_revision in the current map. This one dataset has one type of conflict with respect to a line feature. When you hover over the node, a tooltip with the conflict description will be shown.
tools26.png

The top right section of the conflict editor shows the attributes of the local feature (edits that were made on the local machine) and the attributes of the feature as the OSM Server currently sees it. In our example we have a race track in downtown Seattle and the local version has the name OpenStreetMap First Edition. However due to the popularity of the race track someone had already changed the name to OpenStreetMap Second Edition and submitted the changes back to the server. This change now causes a conflict because the server assigned the already submitted changes the version number 2 whereas the local attributes still refer to version 1.
tools27.png

In addition to the attribute change then was also a change in the geometry of the feature shown in the bottom area of the Conflict Editor. The geometry of the local feature is shown on left side and the geometry on the OSM server is shown in red on the right side. The colors (green for local and red for server) are fixed but the symbology derives from the default drawing symbology settings in ArcMap.
tools28.png

You now have to choose which information is accurate and should be kept: the local attributes (“OpenStreetMap First Edition”) and the server geometry (red line) or the server attributes (“OpenStreetMap Second Edition”) and the local geometry (green line). When you do a mouse right click on the listed error and choose on the suggested solutions for this error. In the screen capture below the user decided to keep the server attributes (highlighted in green for the attribute table) and the local geometry (green check mark for accepted in the left map and red cross-out for rejection in the right map for the server geometry).
tools29.png

Not all conflicts have a resolution associated with them, and there are always two entries to revise the state of the conflict itself. The first option, Reset the current revision status, clears the conflict status but leaves the error description in the database. It is then up to you to determine a conflict resolution based on the description before the data can be submitted back to the server. The second option, Remove the revision incident removes the conflict altogether so that the local state of the feature becomes disassociated from the server feature they are no longer influencing each other.

Back To: Documentation

Last edited Dec 12, 2012 at 12:45 AM by eggwhites, version 53

Comments

No comments yet.