ArcGIS vs QGIS etc Clipping Contest Rematch revisited

Earlier this Last year, in June, Don Meltz wrote an interesting blog “ArcGIS vs QGIS Clipping Contest Rematch” where he let compete ArcGIS and Quantum GIS in a clipping contest. The benchmark contest data set in question is a 878MB ZIP file (ContourClipTest.zip). The blog page gained a lot of comments, even from ESRI since some ArcGIS versions crashed on this test data set.

Find below the various timings compiled from the blog and the comments:

1) Proprietary software

Software Processing time Hardware/Software
ArcGIS 9.3 crash after 1h 9min: ERROR 999999: Error executing function. Invalid Topology [4gb file limit.] Failed to execute (Clip) unknown
ArcGIS 10.0 crash likewise unknown
ArcGIS 10.1 ESRI promise to calculate it in 34 seconds in this updated version (did anyone test?) unknown
GlobalMapper (version?) 30 mins unknown
GlobalMapper v11.02 49 sec Windows XP w/ 3.5GB RAM
Manifold 8 (64bit) 31 min Windows XP64 16 gb. RAM and 2.33 GHz

Note: The two GlobalMapper results are a bit funny, perhaps always minutes?

2) Free and Open Source Software

Software Processing time Hardware/Software
Quantum GIS (version?; Simple features) 4-5 min unknown
GRASS GIS 7 (topological GIS) 5 min Dell PowerEdge 2950 from 2008, Intel Xeon 2.66GHz, 8GB RAM
gvSIG to be done
PostGIS to be done

Notes: Hope volunteers will test this also on gvSIG and PostGIS (and other FOSSGIS)! Please report…

18 thoughts on “ArcGIS vs QGIS etc Clipping Contest Rematch revisited

  1. Andrew Kemp

    Just to add to this…
    Cadcorp SIS MapModeller 8.0
    3 minutes 40.
    Dell Core2 laptop that at the time was being used as a web server, had chrome open and was being used for email. Will post the result later when this test is carried out on a proper desktop.

  2. Anonymous

    PC: Intel core i7, 4 GB RAM; Win 7, 64-bit

    gvSIG 1.11 – 3-4 min.
    gvSIG 1.11 with SEXTANTE – 2-3 min.

    QGIS 1.8 – 4-5 min.

    MapWindow GIS 4.8.6 – 12 min.

  3. Emilio

    My two cents.

    Intel Core2 Quad CPU Q6600 @ 2.40GHz 64 bits; 4 GB RAM; Kubuntu 12.04 Kernel 3.2.0-32-generic

    QGIS 1.8
    Loading shapefiles: 3:35 min.
    Clipping: 3:38 min.

    gvSIG 1.12
    Loading shapefiles: 0:20 min.
    Clipping: 3:27 min. (core geoprocess)

    OpenJUMP 1.5.2
    Loading shapefiles: “Out of Memory Error”

  4. markusN

    Sorry for being carelss with using the word “benchmark”, of course it is a “contest” as stated in the title. It may become a benchmark when executed on the same machine with most software packages in question being installed on it.

  5. Giovanni Manghi

    Just tested on my new laptop (i7, 8gb ram) with Windows 64 and qgis-dev installed with osgeo4w.

    It tool 0:30 to open the shape and 2:50 to do the clip.

    I will try test also using Linux, GRASS and other clipping tools available in the Sextante toolbox.

  6. Markus Metz

    and the winner is …

    JTS/GEOS because a number of different applications could successfully use JTS/GEOS to perform the task at hand within a reasonable amount of time.

    The real winner is IMHO Alessandro Furieri because he described the contest (apples and oranges, even though it was mostly JTS/GEOS vs. JTS/GEOS) most accurately.

    Posted by the person fiddling around with GRASS vector topology.

  7. Markus Metz

    and the winner is …

    JTS/GEOS because a number of different applications could successfully use JTS/GEOS to perform the task at hand within a reasonable amount of time.

    The real winner is IMHO Alessandro Furieri because he described the contest problems (apples and oranges, even though it was mostly JTS/GEOS vs. JTS/GEOS) most accurately.

    To do it in a more comparable way, the contest should IMHO be run on a (one single dedicated) virtual machine with 1 GB RAM, a 32-bit OS, and contestants would have to provide some publicly available software version, for free or for money, which would then be installed on that testing system.

    Posted by the person fiddling around with GRASS vector topology.

  8. Anonymous

    I ran the whole clip on a 64 Win7 laptop (8 Gb Ram / i7 running at 3.4 Ghz) using ArcGIS 10.1 SP1 and 64 Bit Background processing in 20s.

  9. matthias

    Dell Latitude, Win7 Professional (SP1) 64Bit, Intel Core i5 2,5 GHz, 4GB RAM;

    with ArcGIS 10.1 (SP1)
    Loading shape: 1 min
    Clipping: 57s

    with QGIS 1.8
    Loading shape: 3-5 min
    Clipping: 2:55 min

  10. Carlos Grohmann

    MacBookPro, i7 2.7 GHz, 8 GB RAM
    OSX 10.8.2 Mountain Lion

    QGIS 1.8.0
    clip: 1:50 min

    GRASS 6.4.2 (from William Kyngesburye):
    clip: process killed after 45 min 🙁

    GRASS 7.0.svn (from Michael Barton)
    clip: process killed after 45 min (progress bar was at about 20%)

    Note: the GRASS 7.0 package used in the test is dated from 01-Nov-2012, so it is not including the latest vector improvements:
    http://trac.osgeo.org/grass/log/grass/trunk/lib/vector

    GDAL 1.9.2
    ogr2ogr -clipsrc StudyArea1MileBuffer.shp clip.shp Contours20Ft.shp
    clip: 2:16 min

  11. markusN

    Based on the currently latest GRASS 7 (r53935) with fresh speed-ups: using the same old machine from 2008 as used earlier in this original blog (see above):

    – import: 26 seconds including building the topology
    – topological overlay: 4:02 min

  12. Vladimir Naumov

    Here’s my test with PostGIS:

    4Gb RAM, Intel Core i3 540 @ 3.07 Ghz

    Ubuntu 11.10

    PostgreSQL 9.0.5 on i686-pc-linux-gnu, compiled by GCC gcc-4.5.real (Ubuntu/Linaro 4.5.2-8ubuntu4) 4.5.2, 32-bit

    POSTGIS=”2.0.1 r9979″ GEOS=”3.3.3-CAPI-1.7.4″ PROJ=”Rel. 4.8.0, 6 March 2012″ LIBXML=”2.7.8″

    Done in 3 minutes!

  13. Giovanni

    For what is worth… I tested again on the same machine (i7, win 64, 16GB ram), QGIS master, ArcGIS 10 and PostGIS 2 (default configuration):

    clipping:

    2:10 on QGIS

    crash on arcgis (after more or less 1 hour)

    2:14 on PostGIS

  14. John Lindsay

    I’ve just run this benchmark using the clip tool in Whitebox Geospatial Analysis Tools (http://www.uoguelph.ca/~hydrogeo/Whitebox/), which is based on JTS and runs in parallel. The average of 10 iterations was 13.5178 seconds using a 3 GHz 8-core Xeon E5 Mac Pro with 64 GB of RAM.

    Benchmark Results:
    Iteration 1 = 12.04 sec.
    Iteration 2 = 11.47 sec.
    Iteration 3 = 17.305 sec.
    Iteration 4 = 11.62 sec.
    Iteration 5 = 11.489 sec.
    Iteration 6 = 11.543 sec.
    Iteration 7 = 18.781 sec.
    Iteration 8 = 11.657 sec.
    Iteration 9 = 17.611 sec.
    Iteration 10 = 11.662 sec.

    Average = 13.5178 sec.

Comments are closed.