Poor Lost Grid
This post covers importing a raster layer from a text file and tidying it up for analysis using Clip Raster and Con
Data can take many different forms, some obvious and easy to work with, others less so. In this post we look at a special case of a raster layer in the guise of a text file. But first, a bit about file formats.
Raster grids can come in a wide range of formats: jpgs, tifs, NetCDF layers and more than a few others. At their most basic level, rasters have a very simple structure – a set number of rows and columns, a place in the world, a cell size and values for each cell. By comparison, vector layers are much more complex and hold a lot more data. Consider a polygon – the file is keeping track of all the points that make up the polygon’s corners, and how they’re all linked together with lines, and then all the attributes that go along with the polygon. Depending on the level of detail and extent, they can become quite large. Rasters are relatively “light” by comparison.
One unexpected format for rasters is as a text file. We’ve started out a new project recently on finding the best locations for hops in Aotearoa, and one of the many factors to consider is the number of frost free days across the country. The good people at NIWA have done the work and have shared the data with us. Into the Inbox one day came a note with a 3 Mb text file attached:
When downloaded and opened, it looked like this:
What on earth am I supposed to do with this? We’ve got a file with a header of sorts and then a whole bunch of values separated by a space. Let’s decode this thing:
- ncols = the number of columns in the raster grid
- nrows = the number of rows
- xllcorner = the x-coordinate of the lower left hand corner of the grid
- yllcorner = the y-coordinate
- cellsize = well, cell size, the resolution
- NODATA value = a value that denotes no data, or unknown
A bit of experience tells me that the x- and y-coordinate values are in the New Zealand Transverse Mercator coordinate system, so it should play nicely with all our other data layers. Units of the cellsize for this coordinate system would then be metres.
Piecing this all together we have a recipe to construct a grid. We know where to start, how big to make each cell, and how many rows and columns to use. Plus, the sequence of values after the header relates to the value for each cell (it’s a matrix, basically – each line is one row with 688 values). At this point it looks like we’ve got a layer full of no data values, but if we look further down, we start to see different values:
While we’re here, notice that the non -9999 values are floating point – they have a decimal and a certain number of places past the decimal.
To make sense of the -9999s, picture a raster grid with 909 rows and 688 columns, each 1,500 m on each side and covering the landmass of Aotearoa. Since the islands sit diagonally within that grid there are going to be some cells that are covering the sea surface – those are the ones that get a -9999. Anywhere on land gets a floating point number that here represents the median number of frost free days per year.
Enough of this…how do I get this on a map? It used to be that there was a tool called ASCII to Raster which would take a text file and convert it to a normal raster after reading the header lines. It even likes text files!
That tool seems to have gone the way of the dinosaurs and is no longer available in Pro (though there is a Raster to ASCII tool). Why is there a need for these tools? Emailing a text file is often easier than zipping up a raster grid.
So what’s my plan? I do recall that Pro recognises files that end in .ASC (file type 4 here) so I sneakily make a copy of the text file and simply change its extension to .asc.
When I add this to a map like an ordinary layer, voila! I get this:
Now we’re getting somewhere. The extent of this layer clearly goes beyond Aotearoa’s shoreline so I’ll tidy this by clipping the raster to the coastline:
To get this:
Just about to feel like I’m done when I notice something odd in the Contents:
If I understand these data properly, it’s telling us about how many frost free days an area gets annually. So why have I got negative values and values greater than 365? I’ll play around with the symbology a bit to see just where these places are:
So all my negative values are clustered around the St Arnaud Range while the Sounds has the ones greater than 365, plus two hanging out around Kaikoura (not a bad choice) oh, and one at Cape Campbell, too. I suspect that there are some artefacts of the process of creating this layer (what we refer to as interpolation in the trade). A quick email to NIWA confirms this and I should be safe to reclassify all the negative values to 0 and the > 365 values to 365.
All good, but how am I going to do this? If I use the Reclassify tool I can certainly take care of the lower and upper values, but I would have trouble leaving the others as they are, especially since they’re floating point values. (How do I know this? Two ways – looking at the text table above we saw floating point values but I can also click on one of the cells on the map and peer into its value:
Clearly floating point with a ridiculous amount of precision. I need another option. The one that comes to mind is Con, which basically changes values in a cell if it meets certain conditions. So in this case I might use it change all the cells with a negative value to 0 (i.e. if cell value < 0, change it to 0; if not, leave it alone). I’ve got two changes to make subject to different conditions:
- if Value < 0, change it to 0, and
- if Value > 365, change it to 365
In general, I’ve got to be careful about the < (less than) being an actual < and not a <= (less than or equal to) but it’s not critical here, really. Can I do them both at once? Let’s look at the tool – here I’ve set it up with both conditions linked by an OR:
Set up this way, if a value is less than 0 OR greater than 365, the value will be reset to 0 (Input true raster or constant value). If it’s not, the original value from nz_ffp50_Clip at that cell will be used.
This ain’t gonna work because I can only assign one true or false value, so I’ll have to do it in two steps. First I do the negative values (for not particular reason):
Then I do the > 365 values, careful to reset the input conditional and false rasters to the output of the above Con:
And I’ve now got my analysis ready output raster from the original text file:
Notice that the range of values in this new layer runs from 0 to 365. Happy days.
Last thing I’ll do here is to bend this layer to fit my analysis. My sources tell me that hops do best in areas with greater than 120 frost free days, so a quick reclass gets me those areas:
Nice. So in this post we’ve seen how we can work with a text file version of a raster grid, plus we saw how to clip rasters and also how to use the Con tool to make conditional changes. I’m sad to admit that I’ve been keeping my eyes open for a good use of Con for quite a while now…sad testament to how I spend my time, but here it fit in very nicely and was the best tool for the job.
All this talk of hops has made me thirsty. Now what icy cold beverage would possibly make use of hops in its production? Hmmmmm?