In my presentation I briefly review 3 decades of Open Source GIS development, from the 1980th to the present.

See my slides:

Scaling up globally: 30 years of FOSS4G development. Keynote at FOSS4G-CEE 2013, Romania by Markus Neteler

 

Presentation file: Download presentation file (ODP) to get all the clickable links working!

The GRASS GIS team will organize a GRASS GIS Community Sprint from 2-7 Feb, 2013 in Genova, Italy. The sprint is at the same time of the “XIV Meeting degli Utenti Italiani GRASS e Gfoss” at the University of Genova.

We would like to invite you to financially support this upcoming Community Sprint! The past sprints have been very successful as we expect for the upcoming one.

Important Web page:
https://grass.osgeo.org/wiki/GRASS_Community_Sprint_Genova_2013

Please consider to donate:
https://grass.osgeo.org/donations/

Background info
The GRASS GIS Community Sprint is a great occasion for folks to support the development by actively contributing to the source code, manuals or likewise. The community sprint is a get-together for GRASS project members and supporters and related OSGeo projects to take decisions and tackle larger problems. For this meeting, we welcome people committed to improving the GRASS GIS project and the interfaces to QGIS, GDAL, PostGIS, R-stats. Sextante. gvSIG, OGC Services and more. This includes developers, documenters, bug reporters, translators and other OSGeo supporters. Not only the “C Tribe” will be addressed but also Python or whatever the participants prefer.

In order to prepare the upcoming GRASS GIS 6.4.3 release, a major bugtracker cleanup has been done for GRASS 6 over the past few days. More than open 370 trac tickets (back to GRASS 6.4.0) were revisited, updated or closed: the GRASS GIS bugsquashing team submitted over 140 code changes, and subsequently 88 tickets could be closed in these few days. The few remaining critical tickets are being worked on, leading to a new stable GRASS GIS 6.4.3 release to be expected soon.

The next “GRASS GIS Community Sprint” will take place from May 23 to May 28, 2012 in Prague, Czech Republic directly following the Geoinformatics FCE CTU 2012 conference.

This GRASS Community Sprint is a great occasion for you to support the development by actively contributing to the source code, manuals or likewise. It is a get together for GRASS project members and supporters to make decisions and tackle larger problems. For this meeting, we welcome people committed to improving the GRASS GIS project. This includes developers, documenters, bug reporters, translators and others.

Timing and Duration:

May 23, 2012 (day of arrival) – May 28, 2012 (day of departure)

Venue:

Department of Mapping and Cartography Faculty of Civil Engineering, Czech Technical University in Prague

For more detailed information, please visit
https://grass.osgeo.org/wiki/GRASS_Community_Sprint_Prague_2012

GRASS GIS 6.4.2 released
19 February 2012
https://grass.osgeo.org

We are pleased to announce the release of a new stable version of GRASS GIS. This release fixes bugs discovered in version 6.4.1 of the program and adds a number of new features. This release includes over 760 updates to the source code since 6.4.1. As a stable release series, the 6.4 line will enjoy long-term support and incremental enhancements while preserving backwards-compatibility with the entire GRASS 6 line.

The new wxPython graphical user interface (wxGUI) has been updated with many new features and tools. Python is now a fully supported scripting language, including an updated Python toolkit to simplify the authoring of personal scripts, support for NumPy based array calculations, and a Python application interface for the GRASS C libraries. Additionally, MS-Windows support continues to mature.  GRASS 6.4.2 debuts ten new modules, a new GUI cartographic composer tool, a new GUI object-oriented modeling environment, and improved infrastructure for installing community supplied add-on modules.

Read the full story at
https://grass.osgeo.org/announces/announce_grass642.html

About GRASS GIS

The Geographic Resources Analysis Support System, commonly referred to as GRASS, is an Open Source Geographic Information System (GIS) and geospatial analysis toolkit. For nearly three decades, GRASS has provided powerful raster, vector, and geospatial processing engines in a single integrated software suite. GRASS includes tools for spatial modeling of raster and vector data, visualization, the management and analysis of geospatial information, and the processing of satellite and aerial imagery. It also provides the capability to produce sophisticated presentation graphics and publication-quality hardcopy maps. GRASS has now been translated into twenty languages and supports an extensive array of data formats. It is distributed under the terms of the GNU General Public License (GPL).

GRASS differs from many other GIS software packages used in the academic and professional worlds in that it is developed and distributed by users for users, mostly on a volunteer basis. Its code and spatial processing algorithms are open and transparent, and the software is distributed free of charge. The source code is also freely available, allowing for immediate customization, examination of the underlying algorithms, the addition of new features, and faster identification and patching of bugs.

Watch how the community based GRASS GIS software development works! You can see how the individual contributors modify and expand the source code.

GRASS GIS 6.4 development visualization from 1999 to 2011 with Gource

The corresponding timeline is available at https://grass.osgeo.org/devel/grasshist.html

Download the high resolution version from https://grass.fsv.cvut.cz/video/

A first release candidate of GRASS 6.4.1 is now available.

Source code:
https://grass.osgeo.org/grass64/source/
https://grass.osgeo.org/grass64/source/grass-6.4.1RC1.tar.gz

Windows Binaries:
https://grass.osgeo.org/grass64/binary/mswindows/native/WinGRASS-6.4.1RC1-1-Setup.exe

More binaries will become available shortly.

To get the RC1 source code from SVN:
svn checkout https://svn.osgeo.org/grass/grass/tags/release_20110103_grass_6_4_1RC1

An announcement has been drafted at
https://trac.osgeo.org/grass/wiki/Release/6.4.1RC1-News

All RC news will be merged into the final announcement later.

Since the 6.4.0 release in September 2010 almost 390 source code
modifications have been made to the 6.4.x release branch. Key
improvements of the GRASS 6.4.1 release include enhanced
portability for MS-Windows (native support), fixes for the new
wxPython based portable graphical interface, and new functionality.

Release candidate management at
https://trac.osgeo.org/grass/wiki/Grass6Planning

Please join us in testing this release candidate for the final release.

Thanks to all contributors!

A second release candidate of GRASS 6.4.0 is now available:

https://grass.osgeo.org/grass64/source/
https://grass.osgeo.org/grass64/source/grass-6.4.0RC2.tar.gz

To get the RC2 source code from SVN:
svn checkout https://svn.osgeo.org/grass/grass/tags/release_20090112_grass_6_4_0RC2

An announcement has been drafted at
https://trac.osgeo.org/grass/wiki/Release/6.4.0RC2-News

All news will be merged into the final announcement later.

Key improvements of the GRASS 6.4.0 release include enhanced
portability for MS-Windows (native support), hundreds of fixes,
the new wxPython based portable graphical interface and much
new functionality.

Release candidate management at
https://grass.osgeo.org/wiki/GRASS_6.4_Feature_Plan

Please test, test, test…

Thanks to all contributors!

Markus

Nov 2008

Lucky to have (access to) a cluster? Here some notes on how to do geospatial number crunching on it a.k.a. HPC (High Performance Computing).

Preparing the disks
We decided to use the ext3 file system. An initial problem was the formatting of the RAID5 disk set since it exceeded the file system specifications. Then, setting the ext3 block size to 4k instead of 1k we could format it.

Storage: a home for GIS data
The disks are available via NFS to all nodes (blades in our case). All raw/original data sets and the GRASS database are sitting in an NFS exported directory which I even link on my laptop to easily add/access/modify stuff.

Front-end machine and blades configuration
The cluster is a (currently) 56CPU blades system, we’ll expand to 128 CPUs later this year (16 blades with 2 procs a 4 core and 16GB RAM per blade). Additionally, we have a front-end machine to run the job manager and to link in further disks, all via NFS.
The blades are configured diskless, i.e. that once started, they receive their operating system from the front end machine via network (10GB/s ethernet). Like this, we have a single directory on the front end which contains all software, this is then propagated to all blades. Very convenient. We use Scientific Linux (the LiveDVD copied onto the disk, there is a special directory to store your modifications which are then merged in on the fly once you boot the blades, pretty cool concept). The job software is (SUN) Grid Engine, also free software. Job control with GRASS I have described here:

https://grass.osgeo.org/wiki/Parallel_GRASS_jobs
-> Grid Engine

GRASS: Avoiding replicated import of large data files through virtual linking
New in GRASS 6.4 is that you can just register a geodata file on the fly with r.external. Altogether I have 1.4TB of new GIS data from our province, naturally I didn’t want to by a new disk array just for my provincial GRASS location! Here r.external comes handy to minimize the “import” to a few bytes. As expected, it leverages GDAL to get data into GRASS, the overhead is minimal.

Power consumption
Power consumption is measured, too: The entire system consumes around 2000W (each blade less than 200W), so it’s going into the direction of “green” computing (there is no such thing!). If we had a solar panel at least…

Outcome
All in all a very nice solution. I made a stress test and removed all internal switches and shut down the blades while I was processing 8000 MODIS satellite maps. Everything survived and the Grid engine job manager collected the crashed jobs and restarted them without complaining. All resulting maps are collected in the target GRASS mapset and could be even exported to common GIS formats, if needed.
If you want to run Web Processing Services (e.g., pyWPS), you can likewise send each session to a node, giving you enormous possibilities for your customers.

Edit 2014:

“Big data” challenges on a cluster – limits and our solutions to overcome them:

  • 2008: internal 10Gb network connection way too slow
    • Solution: TCP jumbo frames enabled (MTU > 8000) to speed up the internal NFS transfer
  • 2009: hitting an ext3 filesystem limitation (not more than 32k subdirectories allowed but having more files in cell_misc/ as each GRASS GIS raster map consists of multiple files)
    • Solution: adopting XFS filesystem [yikes! …. all to be reformated, i.e. some terabyes had to be “parked” temporarily]
  • 2012: Free inodes on XFS exceeded
    • Solution: Update XFS version [err, reformat everything again]
  • 2013: I/O saturation in NFS connection between chassis and blades
    • Solution: reduction to one job per blade (queue management), 21 blades * 2.5 billion input pixels + 415 million output pixels
  • 2014: GlusterFS saturation
    • Solution: Buy and use a new 48 port switch, ti implement 8-channel trunking (= 8 Gb/s)

After getting mad with Lidar points colorizing which till now
required a DB table with GRASSRGB attributes, I have
modified d.vect to support colors directly from z height (geometry).
Works for 3D points, lines (eg, 3D contours) and 3D polygons
(eg delaunay triangles):

# Spearfish:
g.region rast=elevation.10m
r.random elevation.10m n=5000 vector=random3d -d
d.mon x0
# display as black points
d.vect random3d
# display 3D points colorized according to z height
d.vect -z random3d zcol=gyr

# 3D contour lines
r.contour elevation.10m out=contour20m step=20
d.vect -z contour20m zcol=gyr

# generate 3D triangles
v.delaunay random3d out=random3d_del
# display 3D polygons colorized according to z height
d.vect -z random3d_del type=area zcol=gyr

Enjoy,
Markus