{"id":2126,"date":"2018-05-25T10:49:23","date_gmt":"2018-05-24T22:49:23","guid":{"rendered":"http:\/\/blogs.lincoln.ac.nz\/gis\/?p=2126"},"modified":"2023-05-07T03:31:17","modified_gmt":"2023-05-07T03:31:17","slug":"a-humble-table-join-saves-the-day","status":"publish","type":"post","link":"https:\/\/blogs.lincoln.ac.nz\/gis\/a-humble-table-join-saves-the-day\/","title":{"rendered":"A Humble Table Join Saves the Day"},"content":{"rendered":"<p><em>Table joins are an easy and convenient way to link data in spreadsheeets to spatial layers.\u00a0 Here we cover a good example of one simple table join deciphering some codes in a soils layer.<\/em><\/p>\n<p>Those familiar with the J: drive will know that we hold many national scale datasets, from soils to land use to elevation,\u00a0in a mixture of raster and vector formats.\u00a0 Many will also know that sometimes the attributes are less than forthcoming in terms of meaning.\u00a0 There are several examples of this, but let&#8217;s look at one in particular: S-Map.<\/p>\n<p><a href=\"https:\/\/smap.landcareresearch.co.nz\/\" target=\"_blank\" rel=\"noopener noreferrer\">S-Map<\/a> comes to us through <a href=\"https:\/\/www.landcareresearch.co.nz\/home\" target=\"_blank\" rel=\"noopener noreferrer\">Mannaki Whenua<\/a> (Landcare Research).\u00a0 This a vector polygon layer\u00a0of soil families with 238,231 polygons.\u00a0 Here&#8217;s a look at the raw data layer:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapfull2.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2129\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapfull2.jpg\" alt=\"\" width=\"1008\" height=\"760\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapfull2.jpg 1008w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapfull2-300x226.jpg 300w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapfull2-768x579.jpg 768w\" sizes=\"auto, (max-width: 1008px) 100vw, 1008px\" \/><\/a><\/p>\n<p>Too much detail to see at this scale so I&#8217;ll zoom in to somewhere closer to home:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapzoom2.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2130\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapzoom2.jpg\" alt=\"\" width=\"1004\" height=\"762\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapzoom2.jpg 1004w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapzoom2-300x228.jpg 300w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/smapzoom2-768x583.jpg 768w\" sizes=\"auto, (max-width: 1004px) 100vw, 1004px\" \/><\/a><\/p>\n<p>There&#8217;s clearly not full coverage but what&#8217;s there looks quite detailed.\u00a0 And here&#8217;s a peek at the attribute table:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/table.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2131\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/table.jpg\" alt=\"\" width=\"750\" height=\"505\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/table.jpg 750w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/table-300x202.jpg 300w\" sizes=\"auto, (max-width: 750px) 100vw, 750px\" \/><\/a><\/p>\n<p>For those areas mapped, this is a valuable resource with lots of quantitative information about soils.\u00a0 But the view of table above shows us an issue.\u00a0 If you can make sense of these values, you&#8217;re doing better than I am &#8211; these are coded values rather than human-readable text descriptions.\u00a0 I recently had some students who wanted to map soil texture.\u00a0 It&#8217;s easy enough to symoblise the polygons by the texture code (TextureDom, or dominant texture), but without knowing what it means, it&#8217;s next to useless.\u00a0 Knowing (hoping) that there was some sort of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Rosetta_Stone\" target=\"_blank\" rel=\"noopener noreferrer\">Rosetta Stone<\/a> somewhere that would allow some translation, I set out on a search.\u00a0 What we&#8217;re in need of here are <a href=\"https:\/\/www.opendatasoft.com\/2016\/08\/25\/what-is-metadata-and-why-is-it-important-data\/\" target=\"_blank\" rel=\"noopener noreferrer\">metadata<\/a> &#8211; data about data, essentially.<\/p>\n<p>My first port of call was the <a href=\"https:\/\/lris.scinfo.org.nz\/layer\/48440-s-map-a-new-soil-spatial-information-system-for-new-zealand-current\/\" target=\"_blank\" rel=\"noopener noreferrer\">Manaaki Whenua data portal<\/a>:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/LRIS.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2132\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/LRIS.jpg\" alt=\"\" width=\"1435\" height=\"865\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/LRIS.jpg 1435w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/LRIS-300x181.jpg 300w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/LRIS-1024x617.jpg 1024w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/LRIS-768x463.jpg 768w\" sizes=\"auto, (max-width: 1435px) 100vw, 1435px\" \/><\/a><\/p>\n<p>First off &#8211; I&#8217;m in for a shock!\u00a0 S-Map is no longer available for downloads to commercial users&#8230;\u00a0 <em>(Good thing I had already downloaded it to J: some time ago&#8230;)<\/em>\u00a0 Second, I couldn&#8217;t find anything that would allow me to translate the texture codes.\u00a0 Notice the &#8220;Metadata&#8221; tab in the screenshot above &#8211; for many layers, you can often get more detail on the data and attributes from here, either is text for or as downloadable documents, but in this case, nothing.<\/p>\n<p>Back to square one, or rather back to the J: drive.\u00a0 In perusing the S-Map folder (with Windows Explorer rather than ArcCatalog) I did manage to find this little gem:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/metadata.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2133\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/metadata.jpg\" alt=\"\" width=\"797\" height=\"560\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/metadata.jpg 797w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/metadata-300x211.jpg 300w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/metadata-768x540.jpg 768w\" sizes=\"auto, (max-width: 797px) 100vw, 797px\" \/><\/a><\/p>\n<p>Which contained\u00a0this:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/texture.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2134\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/texture.jpg\" alt=\"\" width=\"948\" height=\"511\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/texture.jpg 948w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/texture-300x162.jpg 300w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/texture-768x414.jpg 768w\" sizes=\"auto, (max-width: 948px) 100vw, 948px\" \/><\/a><\/p>\n<p>So here&#8217;s my translation of what the codes mean.\u00a0 Recall that I&#8217;ve got over 328,000 polygons so I have no intention of going through line by line and adding in the description &#8211; I wouldn&#8217;t even make a summer scholarship student to that!\u00a0 I could add a new text attribute and then run a series of Select by Attributes and Field Calculations, but instead I&#8217;ll opt to use a <a href=\"http:\/\/desktop.arcgis.com\/en\/arcmap\/10.3\/manage-data\/tables\/joining-attributes-in-one-table-to-another.htm\" target=\"_blank\" rel=\"noopener noreferrer\">table join<\/a>, one of the most powerful ways of enriching existing data layers.\u00a0 The basic idea is that I&#8217;ll create a simple table with the Texture Codes and Descriptions shown above and then join it to the spatial layer based on the Texture Codes.\u00a0 As long as I have attribute values in both my spatial layer and my spreadsheet (this could also be a text or csv table) that match then I can link them together.\u00a0 So here&#8217;s my simple table, done in Excel:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/ss.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2135\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/ss.jpg\" alt=\"\" width=\"533\" height=\"539\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/ss.jpg 533w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/ss-297x300.jpg 297w\" sizes=\"auto, (max-width: 533px) 100vw, 533px\" \/><\/a><\/p>\n<p>Three things to note here:<\/p>\n<ul>\n<li>the first line is what will become my field names;<\/li>\n<li>No spaces, no odd characters like % or &amp; and no numbers at the beginning of the name;<\/li>\n<li>The field names between table and layer don&#8217;t need to match but the actual values must in both format (e.g numbers\u00a0versus text) and value.<\/li>\n<\/ul>\n<p>Next, I can add that sheet to my map using the Add Data button &#8211; it will appear in the Table of Contents though you&#8217;ll only see it if the List by Source button,<a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/source.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2136\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/source.jpg\" alt=\"\" width=\"258\" height=\"59\" \/><\/a>, is ticked (this will happen automatically when you do the join).\u00a0 The figure below shows the table added and open:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/Onthemap.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2137\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/Onthemap.jpg\" alt=\"\" width=\"998\" height=\"760\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/Onthemap.jpg 998w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/Onthemap-300x228.jpg 300w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/Onthemap-768x585.jpg 768w\" sizes=\"auto, (max-width: 998px) 100vw, 998px\" \/><\/a><\/p>\n<p>Next step is to join this table to the S_Map_All_Attributes attribute table.\u00a0 Do so by right-clicking on the layer name and going to Joins and Relates &gt; Join&#8230;:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/join.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2138\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/join.jpg\" alt=\"\" width=\"1005\" height=\"756\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/join.jpg 1005w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/join-300x226.jpg 300w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/join-768x578.jpg 768w\" sizes=\"auto, (max-width: 1005px) 100vw, 1005px\" \/><\/a><\/p>\n<p>(The link between the files is done on the next window:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/tool.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2139\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/tool.jpg\" alt=\"\" width=\"427\" height=\"613\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/tool.jpg 427w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/tool-209x300.jpg 209w\" sizes=\"auto, (max-width: 427px) 100vw, 427px\" \/><\/a><\/p>\n<p>Here I&#8217;m joining the data from TextureCode field in Sheet1$ based on the TextureDom attribute in the spatial layer.\u00a0 If I click the Validate Join button it will go through and check the validity of the join and how many connections it can make &#8211; not a bad thing to do:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/vaildate.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2140\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/vaildate.jpg\" alt=\"\" width=\"799\" height=\"649\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/vaildate.jpg 799w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/vaildate-300x244.jpg 300w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/vaildate-768x624.jpg 768w\" sizes=\"auto, (max-width: 799px) 100vw, 799px\" \/><\/a><\/p>\n<p>This tells me that 237,082 of the 238231 records can be matched.\u00a0 The records that didn&#8217;t match likely don&#8217;t have a value for TextureDom.\u00a0 After ticking Close &gt; OK, the TextureCodes are now joined to the S-Map layer table at the far right:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/joined.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2141\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/joined.jpg\" alt=\"\" width=\"714\" height=\"315\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/joined.jpg 714w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/joined-300x132.jpg 300w\" sizes=\"auto, (max-width: 714px) 100vw, 714px\" \/><\/a><\/p>\n<p>Great!\u00a0 Now I&#8217;ve translated those codes to something useful &#8211; I can use this for analysis and for mapping:<\/p>\n<p><a href=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/mapped.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2142\" src=\"https:\/\/d-blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/mapped.jpg\" alt=\"\" width=\"972\" height=\"601\" srcset=\"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/mapped.jpg 972w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/mapped-300x185.jpg 300w, https:\/\/blogs.lincoln.ac.nz\/gis\/wp-content\/uploads\/sites\/3\/2018\/05\/mapped-768x475.jpg 768w\" sizes=\"auto, (max-width: 972px) 100vw, 972px\" \/><\/a><\/p>\n<p>Before we move on though, there is an important final step in this process.\u00a0 The join is a temporary thing and really belongs to the map document.\u00a0 To make it permanent, I should export this layer as a new one (right-click the S-Map layer &gt; Data &gt; Export Data&#8230; and save as a new layer).\u00a0 Then it will always be a part of the new layer.<\/p>\n<p>I rank table joins as amongst the top five most useful tools in ArcMap.\u00a0 Lots of useful data exist as spreadsheets or text files and as long as you can establish a link to a layer&#8217;s attributes, you can make use of those data.\u00a0 In fact we&#8217;ve talked about table joins before with regard to <a href=\"http:\/\/blogs.lincoln.ac.nz\/gis\/mapping-the-census\/\">mapping census data<\/a>.\u00a0 The example above is a slight different type of join.\u00a0 With the census data, these is a one-to-one match between the spatial data (e.g. meshblocks or suburbs or regions) and the census data.\u00a0 With the S-Map data, there were many instances of each Texture Code, so this was a one-to-many match.\u00a0 You saw how simple my spreadsheet table is &#8211; ArcMap essentially used it as a lookup table to match the correct description to the correct code.<\/p>\n<p>There are many examples of attributes in data such as this (the <a href=\"https:\/\/soils.landcareresearch.co.nz\/index.php\/soil-data\/nzlri-soils\/\" target=\"_blank\" rel=\"noopener noreferrer\">Land Resource Inventory<\/a> and <a href=\"https:\/\/www.landcareresearch.co.nz\/resources\/maps-satellites\/lenz\" target=\"_blank\" rel=\"noopener noreferrer\">Land Environments of New Zealand<\/a>, both on J:,\u00a0come to mind).\u00a0 If you&#8217;ve got some metadata that translates the codes, it&#8217;s often quick and easy to create you&#8217;re own file and then join it to your spatial data, thereby making hard to understand data easier to interpret. The tricky part if often finding the right metadata.<\/p>\n<p>C<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Table joins are an easy and convenient way to link data in spreadsheeets to spatial layers.\u00a0 Here we cover a good example of one simple table join deciphering some codes in a soils layer. Those familiar with the J: drive will know that we hold many national scale datasets, from soils to land use to [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2126","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/posts\/2126","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/comments?post=2126"}],"version-history":[{"count":1,"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/posts\/2126\/revisions"}],"predecessor-version":[{"id":4118,"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/posts\/2126\/revisions\/4118"}],"wp:attachment":[{"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/media?parent=2126"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/categories?post=2126"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.lincoln.ac.nz\/gis\/wp-json\/wp\/v2\/tags?post=2126"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}