Category Archives: GRASS development

GRASS GIS 6.4.2 released

GRASS GIS 6.4.2 released
19 February 2012
http://grass.osgeo.org

We are pleased to announce the release of a new stable version of GRASS GIS. This release fixes bugs discovered in version 6.4.1 of the program and adds a number of new features. This release includes over 760 updates to the source code since 6.4.1. As a stable release series, the 6.4 line will enjoy long-term support and incremental enhancements while preserving backwards-compatibility with the entire GRASS 6 line.

The new wxPython graphical user interface (wxGUI) has been updated with many new features and tools. Python is now a fully supported scripting language, including an updated Python toolkit to simplify the authoring of personal scripts, support for NumPy based array calculations, and a Python application interface for the GRASS C libraries. Additionally, MS-Windows support continues to mature.  GRASS 6.4.2 debuts ten new modules, a new GUI cartographic composer tool, a new GUI object-oriented modeling environment, and improved infrastructure for installing community supplied add-on modules.

Read the full story at
  http://grass.osgeo.org/announces/announce_grass642.html

About GRASS GIS

The Geographic Resources Analysis Support System, commonly referred to as GRASS, is an Open Source Geographic Information System (GIS) and geospatial analysis toolkit. For nearly three decades, GRASS has provided powerful raster, vector, and geospatial processing engines in a single integrated software suite. GRASS includes tools for spatial modeling of raster and vector data, visualization, the management and analysis of geospatial information, and the processing of satellite and aerial imagery. It also provides the capability to produce sophisticated presentation graphics and publication-quality hardcopy maps. GRASS has now been translated into twenty languages and supports an extensive array of data formats. It is distributed under the terms of the GNU General Public License (GPL).

GRASS differs from many other GIS software packages used in the academic and professional worlds in that it is developed and distributed by users for users, mostly on a volunteer basis. Its code and spatial processing algorithms are open and transparent, and the software is distributed free of charge. The source code is also freely available, allowing for immediate customization, examination of the underlying algorithms, the addition of new features, and faster identification and patching of bugs.

First release candidate of GRASS 6.4.1 now available

A first release candidate of GRASS 6.4.1 is now available.

Source code:
http://grass.osgeo.org/grass64/source/
http://grass.osgeo.org/grass64/source/grass-6.4.1RC1.tar.gz

Windows Binaries:
http://grass.osgeo.org/grass64/binary/mswindows/native/WinGRASS-6.4.1RC1-1-Setup.exe

More binaries will become available shortly.

To get the RC1 source code from SVN:
svn checkout http://svn.osgeo.org/grass/grass/tags/release_20110103_grass_6_4_1RC1

An announcement has been drafted at
http://trac.osgeo.org/grass/wiki/Release/6.4.1RC1-News

All RC news will be merged into the final announcement later.

Since the 6.4.0 release in September 2010 almost 390 source code
modifications have been made to the 6.4.x release branch. Key
improvements of the GRASS 6.4.1 release include enhanced
portability for MS-Windows (native support), fixes for the new
wxPython based portable graphical interface, and new functionality.

Release candidate management at
http://trac.osgeo.org/grass/wiki/Grass6Planning

Please join us in testing this release candidate for the final release.

Thanks to all contributors!

GRASS GIS 6.4.0 RC2 released

A second release candidate of GRASS 6.4.0 is now available:

http://grass.osgeo.org/grass64/source/
http://grass.osgeo.org/grass64/source/grass-6.4.0RC2.tar.gz

To get the RC2 source code from SVN:
svn checkout http://svn.osgeo.org/grass/grass/tags/release_20090112_grass_6_4_0RC2

An announcement has been drafted at
http://trac.osgeo.org/grass/wiki/Release/6.4.0RC2-News

All news will be merged into the final announcement later.

Key improvements of the GRASS 6.4.0 release include enhanced
portability for MS-Windows (native support), hundreds of fixes,
the new wxPython based portable graphical interface and much
new functionality.

Release candidate management at
http://grass.osgeo.org/wiki/GRASS_6.4_Feature_Plan

Please test, test, test…

Thanks to all contributors!

Markus

Building a cluster for GRASS GIS and other software from the OSGeo stack

Nov 2008

Lucky to have (access to) a cluster? Here some notes on how to do geospatial number crunching on it a.k.a. HPC (High Performance Computing).

Preparing the disks
We decided to use the ext3 file system. An initial problem was the formatting of the RAID5 disk set since it exceeded the file system specifications. Then, setting the ext3 block size to 4k instead of 1k we could format it.

Storage: a home for GIS data
The disks are available via NFS to all nodes (blades in our case). All raw/original data sets and the GRASS database are sitting in an NFS exported directory which I even link on my laptop to easily add/access/modify stuff.

Front-end machine and blades configuration
The cluster is a (currently) 56CPU blades system, we’ll expand to 128 CPUs later this year (16 blades with 2 procs a 4 core and 16GB RAM per blade). Additionally, we have a front-end machine to run the job manager and to link in further disks, all via NFS.
The blades are configured diskless, i.e. that once started, they receive their operating system from the front end machine via network (10GB/s ethernet). Like this, we have a single directory on the front end which contains all software, this is then propagated to all blades. Very convenient. We use Scientific Linux (the LiveDVD copied onto the disk, there is a special directory to store your modifications which are then merged in on the fly once you boot the blades, pretty cool concept). The job software is (SUN) Grid Engine, also free software. Job control with GRASS I have described here:

http://grass.osgeo.org/wiki/Parallel_GRASS_jobs
-> Grid Engine

GRASS: Avoiding replicated import of large data files through virtual linking
New in GRASS 6.4 is that you can just register a geodata file on the fly with r.external. Altogether I have 1.4TB of new GIS data from our province, naturally I didn’t want to by a new disk array just for my provincial GRASS location! Here r.external comes handy to minimize the “import” to a few bytes. As expected, it leverages GDAL to get data into GRASS, the overhead is minimal.

Power consumption
Power consumption is measured, too: The entire system consumes around 2000W (each blade less than 200W), so it’s going into the direction of “green” computing (there is no such thing!). If we had a solar panel at least…

Outcome
All in all a very nice solution. I made a stress test and removed all internal switches and shut down the blades while I was processing 8000 MODIS satellite maps. Everything survived and the Grid engine job manager collected the crashed jobs and restarted them without complaining. All resulting maps are collected in the target GRASS mapset and could be even exported to common GIS formats, if needed.
If you want to run Web Processing Services (e.g., pyWPS), you can likewise send each session to a node, giving you enormous possibilities for your customers.

Edit 2014:

“Big data” challenges on a cluster – limits and our solutions to overcome them:

  • 2008: internal 10Gb network connection way too slow
    • Solution: TCP jumbo frames enabled (MTU > 8000) to speed up the internal NFS transfer
  • 2009: hitting an ext3 filesystem limitation (not more than 32k subdirectories allowed but having more files in cell_misc/ as each GRASS GIS raster map consists of multiple files)
    • Solution: adopting XFS filesystem [yikes! …. all to be reformated, i.e. some terabyes had to be “parked” temporarily]
  • 2012: Free inodes on XFS exceeded
    • Solution: Update XFS version [err, reformat everything again]
  • 2013: I/O saturation in NFS connection between chassis and blades
    • Solution: reduction to one job per blade (queue management), 21 blades * 2.5 billion input pixels + 415 million output pixels
  • 2014: GlusterFS saturation
    • Solution: Buy and use a new 48 port switch, ti implement 8-channel trunking (= 8 Gb/s)

d.vect: support for z height (geometry) colors added

After getting mad with Lidar points colorizing which till now
required a DB table with GRASSRGB attributes, I have
modified d.vect to support colors directly from z height (geometry).
Works for 3D points, lines (eg, 3D contours) and 3D polygons
(eg delaunay triangles):

# Spearfish:
g.region rast=elevation.10m
r.random elevation.10m n=5000 vector=random3d -d
d.mon x0
# display as black points
d.vect random3d
# display 3D points colorized according to z height
d.vect -z random3d zcol=gyr

# 3D contour lines
r.contour elevation.10m out=contour20m step=20
d.vect -z contour20m zcol=gyr

# generate 3D triangles
v.delaunay random3d out=random3d_del
# display 3D polygons colorized according to z height
d.vect -z random3d_del type=area zcol=gyr

Enjoy,
Markus

Fifth release candidate of GRASS 6.3.0 now available

A fifth release candidate of GRASS 6.3.0 is now available:

Source code download:
http://grass.osgeo.org/grass63/source/grass-6.3.0RC5.tar.gz

An initial announcement has been drafted at
http://trac.osgeo.org/grass/wiki/Release/6.3.0RC5-News

Key fixes include improved portability for MS-Windows (native support), several module fixes, and especially the introduction of a new, Python based portable graphical user interface which includes a new vector digitizer.

About GRASS GIS:
Commonly referred to as GRASS, this is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization. GRASS is currently used in academic and commercial settings around the world, as well as by many governmental agencies and environmental consulting companies. GRASS is official project of the Open Source Geospatial Foundation.

Web site: http://grass.osgeo.org/

New GRASS Book published: Open Source GIS: A GRASS GIS Approach

The new edition of Open Source GIS: A GRASS GIS Approach is now available! With this third edition, we enter the new era of GRASS 6, the first release that includes substantial new code developed by the International GRASS Development Team. The dramatic growth in open source software libraries has made GRASS 6 development more efficient, and has enhanced GRASS interoperability with a wide range of open source and proprietary geospatial tools. The book is based on GRASS 6.3.

Thoroughly updated with material related to GRASS6, the third edition includes new sections on attribute database management and SQL support, vector networks analysis, lidar data processing and new graphical user interfaces. All chapters are updated with numerous practical examples using the first release of a comprehensive, state-of-the-art geospatial data set. This new OSGeo Educational data set along with additional material can be downloaded from http://www.grassbook.org/

First release candidate of GRASS 6.3.0 now available

A first release candidate of GRASS 6.3.0 is now available:

http://grass.itc.it/grass63/source/
-> grass-6.3.0RC1.tar.gz

An initial announcement has been drafted at
http://grass.itc.it/announces/announce_grass630.html
(state April 2007, further fixes need to be merged in)

Key fixes include improved portability for MS-Windows
(native support), hundreds of fixes, TclTk based
portable graphical interface and much new functionality.

Release candidate management at
http://grass.gdf-hannover.de/wiki/GRASS_6.3_Feature_Plan

Please test, test, test…

Community based software development: The GRASS GIS project

Tomorrow I’ll give a talk at University of Trento about the question why free software development works. Focus will be on GRASS, a Geographic Information System (GIS) used for geospatial data management and analysis, spatial modeling, and visualization. The project has an outstanding history as it was originally started a military project, but in the public domain. With the advent of Internet and related network based development tools a community was formed. Nowadays, both users and developers are spread over the globe and manage to develop this software with low administrational overhead and a good communication culture. This seminar reviews the technical and social aspects of this Free GIS project.

You can get the seminar slides here.