Python: Geospatial Environment Setup (Part 2)

zero to pandas

There are alot of Python courses out there that we can jump into and get started with. But to a certain extent in that attempt to learn the language, the process becomes unbearably long and frustratingly slow. We all know the feeling of wanting to run before we could learn how to walk; we really wanna get started with some subtantial project but we do not know enough to even call the data into the terminal for viewing.

Back in August, freeCodeCamp in collaboration with Jovian.ai, organized a very interesting 6-week MOOC called Data Analysis with Python: Zero to Pandas and as a self-proclaimed Python groupie, I pledged my allegiance!

If there are any expectation that I've managed to whizz myself through the course and obtained a certificate, nothing of that sort happened; I missed the deadline cause I was busy testing out every single code I found and work had my brain on overdrive. I can't...I just...can't. Even with the extension, I was short of 2 Pythonic answers required to earn the certificate. But don't mistake my blunders for the quality of the content this course has to offer; is worth every gratitude of its graduates!

Zero to Pandas MOOC is a course that spans over 6 weeks with one lecture webinar per week that compacts the basics of Python modules that are relevant in executing data analysis. Like the play on its name, this course assumes no prior knowledge in Python language and aims to teach prospective students the basics on Python language structure AND the steps in analyzing real data. The course does not pretend that data analytics is easy and cut-corners to simplify anything. It is a very 'honest' demonstration that effectively gives overly ambitious future data analysts a flick on the forehead about data analysis. Who are we kidding? Data analysis using programming language requires sturdy knowledge in some nifty codes clean, splice and feature engineer the raw data and real critical thinking on figuring out 'Pythonic' ways to answer analytical questions. What does it even mean by Pythonic ways? Please refer to this article by Robert Clark, How to be Pythonic and Why You Should Care. We can discuss it somewhere down the line, when I am more experienced to understand it better. But for now, Packt Hub has the more comprehensive simple answer; it simply is an adjective coined to describe a way/code/structure of a code that utilizes or take advantage of the Python idioms well and displays the natural fluency in the language.

The bottom line is, we want to be able to fully utilize Python in its context and using its idioms to analyze data.

The course is conducted at Jovian.ai platform by its founder; Aakash and it takes advantage of Jupyter-like notebook format; Binder, in addition to making the synchronization available at Kaggle and Google's Colab. Each webinar in this course spans over close to 2 hours and each week, there are assignments on the lecture given. The assignments are due in a week but given the very disproportionate ratio of students and instructors, there were some extensions on the submission dates that I truly was grateful for. Forum for students is available at Jovian to engage students into discussing their ideas and question and the teaching body also conducts office hours where students can actively ask questions.

The instructor's method of teaching is something I believe to be effective for technical learners. In each lectures, he will be teaching the codes and module requires to execute certain tasks in the thorough procedure of the data analysis task itself. From importing the .csv formatted data into Python to establishing navigation to the data repository...from explaining what the hell loops are to touching base with creating functions. All in the controlled context of two most important module for the real objective of this course; Numpy and Pandas.

My gain from this course is immensely vast and that's why I truly think that freeCodeCamp and Jovian.ai really put the word 'tea' to 'teachers'. Taking advantage of the fact that people are involuntarily quarantined in their house, this course is something that should not be placed aside in the 'LATER' basket. I managed to clear my head to understand what 'loop' is! So I do think it can solve the world's problem!

In conclusion, this is the best course I have ever completed (90%!) on data analysis using Python. I look forward to attending it again and really finish up that last coursework.

Oh. Did I not mention why I got stuck? It was the last coursework. We are required to demonstrate all the steps of data analysis on data of our choice, create 5 questions and answer them using what we've learned throughout the course. Easy eh? Well, I've always had the tendency of digging my own grave everytime I get awesome cool assignments. But I'm not saying I did not do it :). Have a look-see at this notebook and consider the possibilities you can grasp after you've completed the course. And that's just my work...I'm a standard C-grade student.

And the exciting latest news from Jovian.ai is that they have upcoming course at Jovian for Deep Learning called Deep Learning with PyTorch: Zero to GANS! That's actually yesterday's news since they organized it earlier this year...so yeah...this is an impending second cohort! Tentatively, the course will start on Nov 14th. Click the link below to sign-up and get ready to attack the nitty-gritty. Don't say I didn't warn ya.

And that's me, reporting live from the confinement of COVID pandemic somewhere in a developing country at Southeast Asia....

Society of Conservation GIS Scholarship Award 2020

Malaysia Forest Cover 2020 | WWF-Malaysia

Tool: ArcGIS Pro 2.9.3, Operations Dashboard ArcGIS & ArcGIS Online Technique: Data transformation and geometric calculation

WWF-Malaysia Forest Cover Baseline is a dashboard of forest cover extent status in selected land uses across Malaysia's region, methodology of analysis and resources involved in the exercise.

The WWF-Malaysia Forest Cover Baseline and Forest Cover Key Performance Index (KPI) is a task undertaken by the Conservation Geographical Information System (CGIS) Unit to amass the discrete information of forest cover extent across Malaysia's 3 main region of legislation: Peninsular Malaysia, Sarawak and Sabah. This exercise produces a concise dashboard report in an online platform that describes the processed information on the forest cover status as well as their prospective areas identified for conservation work.

Report can be interactively accessed at the following:

The dashboard can be accessed at Malaysia Forest Cover 2020.

📌 Availability: Retracted in 2021

[2022] 30 Day Map Challenge -- FAILED

Last year, I participated once again in the 30 Day Map Challenge that was going around in Twitter-ville come November. It is the 3rd attempt at the marathon and 2022 served as a reminder that progressed too despite getting stuck at Day 3 as life caught up with me.

I don't like the idea that I have left the challenge incomplete, again. It was not my priority and I work better with clear goals or visions of expected output. If it does not add to my need to learn something new ...it will be a task bound to head straight to the backburner. Let's resolve to make it a long-term routine instead of a spurt of stress trying to make the deadline.

As a consequence, I am attuning this task into one that actually gives me the benefit out putting into record the techniques and tools I used to make the maps in writing. I believe that will serve more purpose and added value other than visuals. And perhaps, have some stock ready for submission this year instead.

Anyone else participated in this challenge back in November? How did you do and what would you like to do better for the next one? Don't be shy and do drop a word or two.

🧰 Publicly available data

Hunting for spatial data comes naturally now. There seems to be less and less opportunity for doubts when we could attach a pair of coordinates to some places.

For work and hobby, hunting for data take almost half of the usable hours I set aside to execute certain objectives; if not 100%. Although the internet is a vast plain of data, not all of them are usable. The democratization of data is a subject that is to translucent to discuss but to solid to argue with. Thus, with differing opinions, we get different versions of them online. Here are some of the interesting data platforms I manage to scour based on their thematic subject

🌳 Nature and Environment

Delta at Risk - Profiling Risk and Sustainability of Coastal Deltas of the World. I found this while lamenting on how people love asking for data addition into their maps at the eleventh hour. I find their confidence in my skills quite misleading but flattering nonetheless. But it does not make it any less troublesome.

Protected Planet - Discover the world's protected and conserved areas. This platform includes not just data of protected areas, but also other effective area-based conservation measures like ICCAs IUCN listing and as the website claims, it is updated regular via submissions from agencies. So far, I found this platform to be the most convenient since it rounds up all possible conservation-based themes which also includes World Heritage Sites.

Global Forest Change (2000-2020) - The global forest extent change since 2000 to the current year or lovingly referred to as the Hansen data by most forestry RS specialist. This data is updated annually and to be honest, the platforms are literally everywhere. But this platform is legitimate under Earth Engine Apps and you can refer to Google Earth Engine for future data updates to ease your search.

👩‍⚖️ Administrative Data

GADM - Map and spatial data for all countries and their sub-divisions.

🏦 Built-environment Data

OpenStreet Map - This database is the most amazing feat of tech-aware crowdsourcing. A little more than 2 decades ago, some 'experienced' gate-keeping professionals would have refuted its legitimacy within an inch of their lives but OSM has proven that time prevails when it comes to bringing the accessibility and network data into practical use. I am not that adept with downloading from this website so I go directly to a more manual data download. My favorite is the Geofabrik Download but you can also try Planet OSM.

🎮 Other Cool Data

OpenCell ID - Open database platform of global cell towers. Cleaning the data is a nightmare but I think it is just me. I have little patience for cerebral stuff.

So, those are some of the data I managed to dig for personal projects. Hope it helps you guys too!

code & run

Coding is one of the things I have aspired to do since like...forever! But finding a resource in-sync with my comprehension, schedule and able to retain my interest long enough is a challenge.

I have the attention span of a gnat so, I jumped everywhere! If I am not actively engaged with the learning, I just can't do it. And I know...we have DataCamp, Udemy, Khan Academy and even Kaggle...but I either can't keep up, too poor to pay for the full course or it couldn't sync with me enough. I believe I can say that most of the exercise doesn't 'vibe' with me.

Recently, I committed myself to my one passion; running. It's one of my favorite activities when I was back in school but the will to really run died a decade ago. I have recently picked up my running shoes and ran my little heart out despite having the speed of a running ant; aging perhaps? And I owe my hardcore will to the motivation of earning what I paid when I decided to join a 1-month long virtual run of 65km. It is called the 'Pave Your Path' virtual run organized by

Running Station

. Nailed it 2 days ago after 13 sessions of 5km - yes, you can accumulate the distance from multiple runs. It made me realize that...it's not that bad. The 'near-death' experience while running kinda turned me into a daredevil these days when it comes to undertaking some things I'd whine about doing a few months back.

"If I can go through dying every single evening for 5km long run...I can handle this,"

My thoughts exactly every time I feel so reluctant to finish some tasks I believe I could hold off for some time.

Naturally, I plan my work rigorously and despite the flexibility of my schedule and my detailed plans, I still have a hard time trying to nail the last coffin to my projects. Usually, it's due to my brain's exhaustion from overthinking or I am just truly tired physically. Which is a weird situation given I do not farm for a living. Even so, I was lethargic all the time.

But when I started running a month ago, things kind of fall into places for me. Maybe...just maybe...I've become more alert than I used to. I still have my ignorance of things that I believe do not concern my immediate attention but I seem to be able to network my thoughts faster than I used to.

It might be just me, feeling like a new person due to my sheer willpower to not burn my RM60 paid for the virtual run, but it did feel like there was a change.

For that, I managed to confirm what I have suspected all along - I am one of those people who love drills. I like things to be drilled into my head until I by-heart it into efficiency and then focus on polishing the effectiveness.

Thus...for coding, I committed myself to

freeCodeCamp

. By hook or by crook, I'll be coding by first quarter next year or someone's head is gonna roll!

It's an interactive learning experience simple enough for me to start, straightforward enough to not make me waste my time searching for answers and it's free. God bless Quincy Larson.

Going back to the program outlined in freeCodeCamp, I find it fascinating that they start off with HTML. I have no arguments there. My impatience made me learn my lesson - you run too fast, you're going to burn out painfully and drop dead before you halfway through. HTML is a very gentle introduction to coding for newbies since it's like LEGO building blocks where you arrange blocks and match two to create something. I didn't have to go crazy with frustration is I don't 'get' it. Yes, we would all want some Python lovin' and I think alot of coders I came to know have raved about how simple it is to learn. But I think, it is an opinion shared by 'experienced' coders who wished Python was there when they first started coding. Someone once told me, what you think is the best based on others' experiences may not be the best for you...and I agree with this. After alot of deliberations and patience at my end, starting over again this time feels, unlike the dreaded looming doom I've always had back then.

Are you into coding? What do you code and what's you're language preference? Where did you learn coding? Feel free to share with me!

diy geo app

Esri has been releasing more and more MOOC over the span of 2 years to accommodate its increasingly large expanse of products within the ArcGIS ecosystem. 

But of all the MOOCs that I've participated in, 'Do-It-Yourself Geo App MOOC' must be the most underrated ones produced by Esri Training. The functionalities highlighted within the MOOC took the anthem right off their recent Esri UC 2020 that went virtual. The curriculum includes:

The creation of hosted feature layer (without utilizing any GIS software medium like ArcMap or ArcGIS Pro).

The basics of the ArcGIS Online platform ecosystem:

hosted feature layer >  web map > web app 

Basically, to view a hosted feature layer, you will need to drag it onto a 'Map' and save it as a web map.

Conventionally, web map suffices for the visualization and analytical work for the likes of any geospatialist who are familiar with Web GIS. 

But this time, Esri is highlighting a brand new web map product called 'Map Viewer Beta'. Why beta? Cause it is still in beta version but so sleeky cool that they just had to let every have a shot at using it. Truth be told, Map Viewer Beta did not disappoint.

Even so, Map Viewer Beta still has some functionalities that have yet to be implemented. 

Using web map to visualize data, configure pop-up, execute simple analysis and extending it to Map Viewer Beta interface 

Utilizing Survey123 for crowdsourcing data; the first level of citizen science and creating a webmap out of it.

Creating native apps using AppStudio for ArcGIS; no coding required. 

Some tidbits on accessing the ArcGIS API for JavaScript

I love how cool it is that this MOOC actually shows you step-by-step on how to use the new Map Viewer Beta and explain the hierarchy of formats for the published content in the ArcGIS Online platform

I have established my understanding of ArcGIS Online ecosystem 3 years back but I do find it awkward that such powerful information is not actually summarized in a way that is comprehensible for users that have every intention of delving into Web GIS. And Web GIS is the future with all the parallel servers that could handle the processing/analysis of large amount of data. ArcGIS Online is a simplified platform that provides interfaces for the fresh-eyed new geospatial professionals. 

It is quite well-know for the fact that there has been some criticism as to the domination of Esri within the GIS tools/resources within the geospatial science industry, but I believe it is something we could take as a pinch of salt. Not everything in Esri's massive line of commercial products are superior to other platforms but it is a starting point for any new geospatialists who wants to explore technologies there are not familiar with. 

All in all, this MOOC is heaven-sent. For me, I have been playing with the web apps and web maps for close to 4 years and I can attest to the fact that it covers all the basics. For the developer's bit, maybe not so much as going through it in a distinct step-by-step but it does stoke the curiosity as to how it works. The question is, how do we make it work. Now that's a mystery I am eager to solve. 

I'm going to put this on my ever-expanding to-do list and think JavaScript for another few more months of testing out this ArcGIS API for JavaScript implementation. Tell me if you wanna know how this actually works and I'll share what I find out when I do.

For those who had missed out on this cohort, fear not. This MOOC runs twice a year and the next cohort is going to be from Feb 17 to March 17 2021. The registration is already open, so don’t hold back and click the link below:

Do-It-Yourself Geo Apps

Do register for a public account before signing up or just click 'Register' at the MOOC's page and it's open the open to either sign in or 'Create a public account'. It was a blast and I'm sure, if you've never used any of the feature I've mentioned above, you'll be as wide-eyed as I was 3 years ago. :D

Till then, stay spatially mappy comrades!

P/S: If you complete all the assignments and quizzes, you'll get a certificate of completion from Esri. Which is pretty rad!

Yes peeps. I’ve been studying and on contrary to all my previous attempts to make beautiful notes, I say f it and just work with what helps me clear my head the fastest 🏃🏻‍♀️. I love writing notes, but I realize, to gather my thoughts properly, I need some sort of way to not waste paper just to arrange and rearrange my ideas or comprehension of things. 

What better way of doing that than using a mind map!

So you kiddos out there who are starting out with Python and just can’t wait to get into deep learning or machine learning, I’d say, hold your horses for a minute and have some preview of that pond you’re trying to jump into. And don’t be scared, cause we’re all friends here in the hell-hole of learning plateau. Will it get better? I believe so. I am positive I understand more of the principles of deep learning and the relevance of Python libraries associated with it. Yes...this is a Python bar, darling. 👩🏻‍💻

There’s no real shortcut if you ask me since we have different way of comprehending things; my pre-existing mold may have harder time grasping the things I am learning right now than you would. So don’t be afraid to doodle while you think. No amount of paper will be enough to help you understand things, so better start being sustainable by using some digital platforms and saving those papers to when you’re truly ready to pen out your understanding of things; not what you read. There’s a difference!

Check out the mind map of some essential Python libraries you can get started with before you start doing some deep learning. It’s worth reviewing all that prior, I promise. 

Have fun! 🙆🏻‍♀️

survey123 offline

Survey123 for ArcGIS is perhaps, one of those applications that superficial nerds like me would like; it's easy to configure, kiddie-level degree of customization with 'coding' (for that fragile ego-stroke) and user-friendly template to use. 

No app development/coding experience is required to publish a survey form and believe it or not, you can, personalize your survey to not look so meh. 

It took me some time to stumble through the procedures of enabling this feature before I understand the 'ArcGIS Online' ecosystem to which this app is chained to. 

So how do we do it? And why doesn't it work pronto?

This issue may be due to the fact that when we first start creating our forms, we go through the generic step-by-step procedures that leave little to imagination what was happening. Most of the time, we're too eager to find out how it really work. 

When we publish a Survey123 form; be it from the Survey123 website portal or the Survey123 Connect for ArcGIS software, we are actually creating and publishing a folder that contains a hosted feature layer and a form. It is on that hosted feature layer that we add, delete, update or edit data it. From ArcGIS Online, it looks like any feature service that we publish out of ArcGIS Desktop or ArcGIS Pro, save for the special folder it is placed in with a 'Form' file. 

To enable any offline function in any hosted feature layer in ArcGIS Online, you will need to enable the 'Sync' feature. So far, in many technical articles that I have gone through to learn how to enable this offline feature always goes back to 'Prepare basemaps for offline use'. It is a tad bit frustrating. But my experience when deal with 'Collector for ArcGIS' gave me the sense of epiphany when it comes to Survey123. So when you have prepared your Survey123 form for offline usage and it still doesn't work...do not be alarmed and let's see how to rectify the issue. 

1. Locate your survey's hosted feature layer

At your ArcGIS Online home page, click 'Content' at the main tab. We're going to go directly to your hosted feature layer that was generated for your survey when you published. 

Locate your survey folder. Click it open 

In the survey folder, navigate to the survey's hosted feature layer and click 'Options' button; the triple ellipses icon

At at the dropdown, click 'View item details'. Please refer to the screenshot below: 

2. Change the hosted feature layer settings

At the item details page, navigate to the 'Settings' button at the main header and click it. This will prompt open the settings page for the feature layer. Refer to the screenshot below:

At the 'Settings' page, there are two tabs at the subheader; 'General' and 'Feature layer (hosted)'. Click 'Feature layer (hosted)' to configure its settings.

At the 'Feature layer (hosted)' option, locate the 'Editing' section. Here, check the 'Enable sync' option. This is the option that will enable offline data editing. Please refer to the following screenshot: 

Don't forget to click 'Save'

With this, your hosted feature layer which serves as the data model is enabled for synchronization. Synchronization helps to sync back any changes you've made when you're out on the field collecting data; editing, adding, deleting or update...depending on what feature editing you've configured. 

It's pretty easy once you get the hang of it and just bear in mind that the data hierarchy in the ArcGIS Online universe are as follows:

Feature layer (hosted) > Web map > Web application

Once you get that out of the way, go crazy with your data collection without any worries!