JSON and the Argonauts
We look at extracting data from the Christchurch City Councils SmartView maps.
As local government uses maps to make their data more and more available, we’ll look in this post at extracting some useful data from the Christchurch City Councils SmartView maps. Let start with a quick look at what’s available:
You can type in an address int the Search window or click on one of the subsections – let’s go to Find section and see what’s there:
Clicking on any one of these takes you eventually to a map – Fruit trees look interesting, yes?
This shows us locations and additional information if we click on a tree point. While this is great stuff, I might like to use this layer in my analysis, but in its current form, it is shackled to the browser. How could I extract it?
I’ll admit at the start that I had no idea how to do this, let alone that SmartView was a thing until some of my ERST606 students brought it to my attention (hat tip to Estela and Grace, as well as the Argonauts). There’s always something new to learn with GIS. Anyway, here’s how; gird your loins – it gets a bit involved.
First navigate up to the top level page where we clicked on Find. Right-click somewhere on the blue area and choose “Inspect” – you’ll get a funny looking window at the right:
Select Network tab from the top menu and make sure XHR is selected:
On the SmartView page, click Find > Fruit Trees. In one of the windows, an item called “map_feature.php…” appears (NB – I had to use (gulp) Edge to do this – please don’t tell anyone I used Edge by choice):
Right-click on this and choose ‘Copy link address’
Paste this link in a new tab and bunch of text appears:
Talk about gobbeldygook – what a mess. But there’s gold in them thar characters – on closer inspection, useful bits appear:
This kind of file is what’s known as a JSON file (JavaScript Object Notation) – it’s basically a text file format for exchanging data in a lighter, more portable way. We could, for instance, convert a polygon layer to a JSON file and send that off, knowing that all the features and their geometries are recorded in text format. It’s sort of human readable, but takes a bit of practice to decode. In any event, Pro can work with them.
Select all this text and copy it into a text file using something like Notepad or WordPad. Save the file, giving it a .json extention, e.g. “fruittrees.json”.
In Pro now, use the JSON To Features tool – the Input JSON is the fruittrees.json file and enter a name for the Output feature class in a geodatabase (or you could save it as a shapefile):
Click ‘Run’ and voila they should appear as points!
Nice! This worked for most of the choices in Find, but not all; Street Art, for instance, didn’t, but with a bit of perseverance I’m sure you can get them to work too.
This is a bit involved to get at some data but sometimes one must be prepared to do what’s necessary to get points on the map.
C
Hi, I am SmartView’s developer. The good thing about the method you describe is that it is not bound to SmartView. You can use it more or less on most websites that use data, sometimes it takes a bit of time to figure out when and where is the data located, but it is a very similar approach in most situations.
But if there would be interest in having an easier way to get the data from SmartView, I am happy to look into it and add a button or something in that sense that would allow you to download the GeoJSON without the hassle of going through the Dev Tools.
Thanks Michael – couldn’t agree more and thanks for making that point. Being aware of this makes getting data from similar sites possible – always good!
C
Great idea to make the download more accessible Michael. A few months ago I thought I’d be clever (with my limited GIS skills) and make a map of fruit trees near my house (selected for the species that I was most interested in and upload it to Avenza for ease of searching for the trees on the go with my phone.
Now I know (thanks to this post) how to get the data, but it would be handy in future to have the data available for others thinking the same thing- or students who want to do some analysis and identify where there are gaps of fruit species for future tree planting plans.
Now there is a new button in each of the map views (bottom right corner), clicking the button downloads all the map data points that are currently loaded. It downloads as a geojson file since it is the easiest to work with (in my experience anyway!) . I can’t promise it will work in all browsers and settings, since some devices might block Script triggered .geojson blob downloads, if it doesn’t on some common setup let me know and I’ll look into it, but please don’t ask me to make it work on Internet Explorer! 😀
Btw, some datasets are more complicated and the downloaded data might make not that much sense, but hopefully the ones you are after will be fine.
Let me know if that helps and if you have any other feedback.
Cheers
Very nice blog post. I definitely love this site. Stick with it! .
Some really excellent info Sword lily I detected this.
Thank you for starting this up. This website is something that is needed on the internet someone with a little originality!
I very delighted to find this internet site on bing just what I was searching for as well saved to fav
I like the efforts you have put in this regards for all the great content.
very informative articles or reviews at this time.
You re so awesome! I don t believe I have read a single thing like that before.