Load OSM File Issue

Feb 18, 2013 at 9:52 AM
Hi,

I am trying to load a large OSM file downloaded from Geofabrik using the Load OSM File tool. I have downloaded the Africa.bz2 file from Geofabrik and extracted the file to osm using 7zip - the resulting file is approx 7Gb.

I have tried to run the Load OSM File tool on numerous machines without success on this dataset, including writing to SDE geodatabases (SQL Server and PostgreSQL) and file geodatabase. I am able to load smaller datasets, for example Libya and Burkina Faso osm files to file geodatabase without issue.

When loading the Africa.osm file to file geodatabase, the process begins loading nodes but stops before reaching the total number of nodes. (at around 29900000) No error message is given. Looking at the resource monitor during the process, RuntimeLocalServer.exe is using around 7.7Gb RAM (all that is available).

At this point the file geodatabase contains empty feature classes. If I cancel the geoprocessing task after the process has hung the database is populated with point data (29900000 points), but nothing else. The full dataset contains 38286956 nodes.

Does anyone have any suggestions? I have tried downloading the osm file again and writing to an SDE geodatabase however I cannot get the process to complete even on small OSM files when using an SDE gdb.

My primary machine config is as follows:
Windows Server 2008 R2 SP1 64bit
Core i5 M520
8Gb RAM
ArcGIS Desktop SP1 + 64bit geoprocessor
ArcGIS Editor for OSM 2.1 64bit

Thanks
Coordinator
Feb 19, 2013 at 10:29 PM
Hello drinki,
Quick question - will you verify that Background Geoprocessing is not enabled? See http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//00210000003q000000 for how to enable/disable. If you've already unchecked it and are still getting the issue, let us know.

Christine
Feb 22, 2013 at 2:12 PM
Hi Christine,

I have now disabled Background Geoprocessing, however I am still getting a similar issue. I have tested on two machines (details below), including writing to file and SDE geodatabases (PostgreSQL and SQL Server).

ArcGIS 10.1 (build 3143) on Windows Server 2008, i5 M520, 8Gb RAM
ArcGIS Editor for OSM 2.1 64bit
10950000 nodes loaded...
Buffer cannot be null.
Parameter name: buffer
General error in the node loading routine (Exception of type 'System.OutOfMemoryException' was thrown.).

ArcGIS 10.1 (build 3143) on Windows 7 X5650 CPU with 16Gb RAM
ArcGIS Editor for OSM 2.1 64bit
8150000 nodes loaded...
Exception of type 'System.OutOfMemoryException' was thrown.
General error in the node loading routine (Exception of type 'System.OutOfMemoryException' was thrown.).

Your suggestion to disable background geoprocessing has improved the output slightly. I am now getting some line and polygon features written to the geodatabase, whereas before only point data was being output.

Any further suggestions would be appreciated.
Coordinator
Feb 22, 2013 at 4:03 PM
Edited Feb 22, 2013 at 4:03 PM
Hello drinki,
Thanks for the update. Ok, lets look at some other ways to optimize the process. One thing to keep in mind: as you know, Africa is going to take some time. Probably at least a couple of days to load - when we did Europe, it took about 5-6, and about a month for the whole world. Since it sounds like there are memory issues, probably best to load it on a dedicated machine (well, dedicated at least until you get the file loaded). Sounds like you have two environments, and so this may be an option for you.

On that dedicated machine, please make sure no other memory intensive programs are running. Also, verify that in ArcMap, on the Geoprocessing menu at the top, in the "Geoprocessing Options" dialog, the 'Add results of geoprocessing operations to the display' option is not checked. Otherwise, ArcMap will be adding to the display during the operation and this will take additional time/memory resources.

Another option is to load a portion of Africa instead of the whole continent. Have you tried with a couple of the country .osm files from geofabrik? Do those work ok for you?

Christine
Apr 24, 2015 at 7:48 PM
Hi,

I saw this thread and was wondering what the solution was?
I currently get the same error after loading 14 050 000/~74 000 000 nodes for South America. Is the only option loading the data on a per country basis?

Thanks for any guidance.

Alex
Coordinator
Apr 24, 2015 at 10:52 PM
Hello Alex - First just a housekeeping issue. We've recently moved this whole project to GitHub, so that is the best place to post questions (https://github.com/Esri/arcgis-osm-editor/issues). Also, there is a new version of the toolbox that has better performance for loading large areas. So with the new version you shouldn't have to do the load country-by-country, although for South America the load might still take a few days. Also, please do not try to load the whole world - that will not work, the OSM dataset is too large.

So - here is where you can download the latest installers with the performance improvements (improvement not available before 10.2):

ArcGIS Editor for OSM 10.3.x - http://www.arcgis.com/home/item.html?id=75716d933f1c40a784243198e0dc11a1
ArcGIS Editor for OSM 10.2.x - http://sdi.maps.arcgis.com/home/item.html?id=16970017f81349548d0a9eead0ebba39
Apr 27, 2015 at 1:36 PM
Thank you!

I will continue on github as I am still experiencing issues (won't load past 16-20 million nodes).
Developer
Apr 27, 2015 at 5:24 PM
Alex,

please leave the option "Conserve Memory" as checked. This way we have loaded continents without problems.

The "memory" parameter might be a little misleading - it is not a sophisticated balancing mechanism and for large dataset it is most likely going to run into memory exhaustion issues.
  • Thomas