Loading data and additional attributes

Jun 22, 2012 at 11:14 PM

Hello, I would like to download the large data set (for example, all of Germany) and add multiple attributes.  What is the most efficient way of doing it?  I tried to load data and then use Attribute selector, but that takes more than a week on a very powerful machine.  Sometimes it crashes in the process. Download, Extract, & Symbolize tool only extracts by coordinates and not by the region so I don’t see how that can be used.

Thank you,

-Lana.

 

Developer
Jun 24, 2012 at 1:33 AM

Lana,

as it happens Germany is one of our test cases. Large dataset are only available as extract and the usual sites are listed at http://wiki.openstreetmap.org/wiki/Planet.osm#Country_and_area_extracts.

On a regular desktop machine (about 3 yrs old) the import for us took a little more than 48 hrs. A powerful CPU isn't really helpful as most of the action is actually an I/O bound processes.

If you only need a handful of attributes I would first run the OSM File Loader tool and then write a little python script in the like of

import arcpy

arcpy.OSMGPAttributeSelector('c:\temp\germany.gdb\germany\germany_osm_ln','name;maxspeed;ref')

.....

Unfortunately there are some performance hits in the Attribute Selector tool at the 2.0 release and most time is spent doing a full table scan. For the next release this limit has been removed and it resulted in 10-20x faster execution performance.

- Thomas

Jun 25, 2012 at 9:52 PM

Hi Thomas, thank you.  That is exactly what I was doing, but The AttributSelector call even with a few attributes takes days.  When is the next version available? 

Thank you.

-Lana.

Coordinator
Jun 26, 2012 at 12:34 AM

Hello Lana,

We are planning to have a beta for users to work with around the timeframe of the Esri International User Conference (July 23-27). If you're willing, it'd be great to get your feedback on this part of the functionality. I can let you know when the beta is ready; shall I send you a note at that time?

Christine

Jun 29, 2012 at 6:59 AM

Hello Christine,

Please let me know when beta is available, I would like to try it out as soon as possible.  I just tried to download 4 additional attributes and it took 162 hours, so 10-20x improvement will be very welcome.

Thank you.

-Lana.

Coordinator
Jun 30, 2012 at 12:24 AM

We will add you to the list of folks we ask to look at the beta (although everyone will be able to download it from the site). Thanks for being willing to provide feedback; yes you should see improvement with this feature.

Christine

Jul 24, 2012 at 10:14 PM

Hello Christine,

Do you have a beta available at this time?  if no, when do you expect to release it?

Thank you.

-Lana.

Developer
Jul 24, 2012 at 10:46 PM

Lana,

we just released beta1 for the desktop. Not sure if we can maintain the previous claims of up to 30x times but it is certainly an improvement and we are eager to hear the "new" loading times from you.

- Thomas