IDS Applied Project (AP)

Jess Sullivan

Contents:

The academic evolution described in “Network Driven, Open Source Collaboration and Higher Education: Academic Parallels” (my RA) can be exemplified through a variety of current projects forming my Applied Project (AP). Open Source Software (OSS) may be described as a geographically dispersed continuation of the academic goals found in brick and mortar universities. Perhaps the most common manifestation of OSS methodologies and features- shared commentary and work organization, a Version Control System, and a way to ask for and give advice- is the sharing of scripts.

Scripting may be generalized as the practice of writing small utilities and programs to make technology-specific tasks simpler. Regardless of an Individual developer’s project goals or needs, many tasks pertaining to a specific set of tools or language will be the same for everyone. Common tasks- such as installing complex software prerequisites, managing project dependencies (the layers of tools on which a project is built), or setting up a development environment (a reproducible “sandbox” in which one can work, fiddle, and make mistakes without disrupting anything else) are different for every technology, operating system and language, and are often subject to change as time goes on.

The following are a variety of READMEs (in English; the actual scripts are in scripting languages like Python or Bash Shell!) from scripts of mine contributed over the course of this semester (also in the form of ePort blog posts).

return to index
Link- Scripting for rJava || https://github.com/Jesssullivan/rJDKmanager

Quickly & forcefully manage extra JDKs in base R
Simplify rJava woes

# get this script:
wget https://raw.githubusercontent.com/Jesssullivan/rJDKmanager/master/JDKmanager.R

rJava is depended upon by lots of libraries- XLConnect, OpenStreetMap, many db connectors and is often needed while scripting with GDAL.

library(XLConnect)   # YMMV

Errors while importing a library with depending on a JDK are many, but can (usually) be resolved by reconfiguring the version listed somewhere in the error.

On mac OSX (on Mojave at least), check what you have installed here- (as admin, this is a system path) :

sudo ls  "/Library/Java/JavaVirtualMachines/

I seem to usually have at least half a dozen or more versions in there, between Oracle and openJDK. Being Java, these are basically sandboxed as JVMs and are will not get in each others way.

However:

Unlike JDK configuration for just about everything else, aliasing or exporting a specific release to $PATH will not cut it in R. The shell command to reconfigure for R:

sudo R CMD javareconf

seems to always choose the wrong JDK. Renaming, hiding, otherwise trying to explain to R the one I want (lib XLConnect currently wants none other than Oracle 11.0.1) is futile.
The end-all solution for me is usually to temporarily move other JDKs elsewhere.
This is not difficult to do now and again, but keeping a CLI totally in R for moving / replacing JDKs makes for organized scripting.

return to index
Link- Scripting for Raspian on QEMU || https://github.com/Jesssullivan/QEMU-Raspian

Using QEMU for Raspian ARM

single ARM guest configuration tested on Mac OSX 10.14.6 host
experimental multi-guest support for Ubuntu 18.04 host (Gnome / Budgie) over virtual network bridge

Emulates a variety of Raspian releases on proper ARM hardware with QEMU.

Prerequisites:

QEMU and wget (OSX homebrew)

# OSX:
brew install qemu wget 

# Ubuntu:
sudo apt-get install qemu-system-arm -y

Get the Python3 CLI in this repo:

wget https://raw.githubusercontent.com/Jesssullivan/USBoN/master/QEMU_Raspian.py

info – multiple guests over virtual bridge (.deb distros only):

  • multiple guests get unique MAC addresses
  • see /setupHostDepends.sh for required packages 

# permiss shell scripts with ``` sudo chmod u+x ... ``` 

sudo ./setupHostDepends.sh

# switch to bridge br0:
sudo ./UPbrctl.sh

# back to normal host network:
sudo ./DOWNbrctl.sh

# add network bits for host:
rm /etc/network/interfaces  # existing lo file
cp interfaces /etc/network/  # replace from this repo
cp qemu-ifup /etc/

usage – single guest:

After the first launch, it will launch from the persistent .qcow2 image.

With no arguments & in a new folder, Raspian "stretch-lite" (no desktop environment) will be:

  • downloaded as a zip archive with a release kernel
  • unarchived –> to img
  • converted to a Qcow2 with 8gb allocated as disk
  • launched from Qcow2 as administrator 
sudo python3 QEMU_Raspian.py

optional guest configuration arguments:

  • -h prints CLI usage help
  • -rm removes ALL files added in dir with QEMU_Raspian.py
  • stretch uses standard graphical stretch release with GUI
  • stretchlite for stretchlite release [default!]
  • buster for standard graphical buster release [YMMV]
  • busterlite for busterlite release [YMMV] 
# examples:
sudo python3 QEMU_Raspian.py busterlite
python3 QEMU_Raspian -h  # print help
# YMMV: burn / backup as .img:     

to burn the new image back to an SD card for a hardware Pi:     

# bash
qemu-img convert -f qcow2 -O raw file.qcow2 file.img

return to index
Link- Scripting GDAL for GIS Students || https://github.com/Jesssullivan/GIS_Shortcuts

GIS Shortcuts

GDAL on OSX, Ubuntu & Windows WSL

Bash is great: It is a shell scripting language that is baked into every new Apple computer and many, many Linux distributions too. It is fun to learn and easy to find your “insert repetitive task here” to Bash translation on Stack Exchange, just a search away. As you uses the command prompt more and more for increasingly cool GIS tasks- be it for Python QGIS OR ESRI, R language for data cleaning and analysis, or just because you noticed you can get “mad amounts of work done” at increasingly faster rates- your vocabulary in the UNIX shell and bash will naturally grow.

I wanted to make a GIS post for Mac OS because it is both under-represented (for great reasons) in GIS and arguably the number 1 choice for any discerning consumers of computer hardware.

Many Linux OS options are faster for much of this “UNIX for GIS”, as quite a few of the things we need are already included with many Debian / Ubuntu distros, and come forms that have been stable for a long, long time.

If you are looking to setup a system primarily for GIS / data science (disregarding ESRI of course), See my initial notes on the Ubuntu variant Pop_OS by System76. If you like the ChromeOS vibe for multitasking and simplicity and the familiarity of Mac OS, it is a keeper (and also a sleeper, seeing how many folks are still on Windows for GIS…….).

Open Terminal
(The less you have in your way on your screen, the faster you can go!) xD

Note: in my opinion, homebrew and macPorts are good ideas- try them! If you don’t have it, get it now:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

(….However, port or brew installing QGIS and GDAL (primarily surrounding the delicate links between QGIS / GDAL / Python 2 & 3 / OSX local paths) can cause baffling issues. If possible, don’t do that. Use QGIS installers from the official site and build from source!)

if you need to resolve issues with your GDAL packages via removal:
on MacPorts, try similar:

sudo port uninstall qgis py37-gdal

# on homebrew, list then remove (follow its instructions):

brew list
brew uninstall gdal geos gdal2

!!! NOTE: I am investigating more reliable built-from-source solutions for gdal on mac.

Really!

There are numerous issues with brew-installed gdal. Those I have run into include:

  • linking issues with the crucial directory “gdal-data” (libraries)
  • linking issues Python bindings and python 2 vs. 3 getting confused
  • internal raster library conflicts against the gdal requirements
  • Proj.4 inconsistencies (see source notes below)
  • OSX Framework conflicts with source / brew / port (http://www.kyngchaos.com/software/frameworks/)
  • Linking conflicts with old, qgis default / LTR libraries against new ones
  • Major KML discrepancies: expat standard vs libkml. 
    brew install gdal
#
# brew install qgis can work well too.  At least you can unbrew it!
#

Next, assuming your GDAL is not broken (on Mac OS this is rare and considered a miracle):

# double check CLI is working:
gdalinfo --version
# “GDAL 2.4.0, released 2018/12/14”
gdal_merge.py
# list of args

Using Ubuntu GDAL on Windows w/ WSL

LINK: get the WSL shell from Microsoft

# In the WSL shell:

sudo apt-get install python3.6-dev -y
sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get install libgdal-dev -y
sudo apt-get install gdal-bin -y

# See here for more notes including Python bindings:
# https://mothergeo-py.readthedocs.io/en/latest/development/how-to/gdal-ubuntu-pkg.html

In a new Shell:

# Double check the shell does indeed have GDAL in $PATH:
gdalinfo --version

To begin- try a recent GIS assignment that relies on the ESRI mosaic system and lots of GUI and clicking but use GDAL instead.

Data source: ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/d-10m/

!! Warning! These files are not projected in a way ESRI or GDAL understands. They WILL NOT HAVE A LOCATION IN QGIS. They will, however, satisfy the needs of the assignment. 

# wget on mac is great.  This tool (default on linux) lets us grab GIS data from
# most providers, via FTP and similar protocols.

brew install wget

make some folders

mkdir GIS_Projects && cd GIS_Projects

use wget to download every .dem file (-A .dem) from the specified folder and sub-folders (-r)

wget -r -A .dem ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/

cd ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem

make an index file of only .dem files.
(If we needed to download other files and keep them from our wget (more common)
this way we can still sort the various files for .dem)

ls -1 *.dem > dem_list.txt

use gdal to make state-plane referenced “Output_merged.tif” from the list of files
in the index we made.
it will use a single generic "0 0 255" band to show gradient. 

gdal_merge.py -init "0 0 255" -o Output_Merged.tif --optfile dem_list.txt

copy the resulting file to desktop, then return home

cp Output_Merged.tif ~/desktop && cd

if you want (recommended):

rm -rf GIS_Projects  # remove .dem files.  Some are huge!

In Finder at in ~/desktop, open the new file with QGIS. A normal photo viewer will NOT show any detail.

Need to make something like this a reusable script? In Terminal, just a few extra steps:

mkdir GIS_Scripts && cd GIS_Scripts

open an editor + filename. Nano is generally pre-installed on OSX.

nano GDAL_LiveMerge.sh

COPY + PASTE THE SCRIPT FROM ABOVE INTO THE WINDOW

  • ctrl+X , then Y for yes

make your file runnable:

chmod u+x GDAL_LiveMerge.sh

run with ./

./GDAL_LiveMerge.sh

You can now copy + paste your script anywhere you want and run it there. scripts like this should not be exported to your global path / bashrc and will only work if they are in the directory you are calling them: If you need a global script, there are plenty of ways to do that too.

See /Notes_GDAL/README.md for notes on building GDAL from source on OSX

return to index
Link- Scripting for Mesh Networking || https://github.com/Jesssullivan/motioneye-mesh/tree/master/batman

Decentralized Video Monitoring over a Pi ‘Zero W’ Mesh Network

Using motioneye video clients on Raspbian & a BATMAN-adv Ad-Hoc network

link: motioneyeos
link: motioneye Daemon
link: Pi Zero W Tx/Rx data sheet:
link: BATMAN Open Mesh

The Pi Zero uses an onboard BCM43143 wifi module. See above for the data sheet.
We can expect around a ~19 dBm Tx signal from a BCM43143 if we are optimistic. Unfortunately, "usable" Rx gain is unclear in the context of the Pi.

This implementation of motioneye is running on Raspbian Buster (opposed to motioneyeos).

Calculating Mesh Effectiveness w/ Python:

Please take a look at dBmLoss.py- the idea here is one should be able to estimate the maximum plausible distance between mesh nodes before setting anything up. It can be run with no arguments- 

python3 dBmLoss.py

…with no arguments, it should use default values (Tx = 20 dBm, Rx = |-40| dBm) to print this: 

you can add (default) Rx Tx arguments using the following syntax:
                 python3 dBmLoss.py 20 40
                 python3 dBmLoss.py <Rx> <Tx>                 

 57.74559999999994 ft = max. mesh node spacing, @
 Rx = 40
 Tx = 20

Try a few values from a shell or take a look in the file for more info.

Order of Operations:

All shell scripts should be privileged to match chmod u+x and assume a Raspbian Buster environment.

See ./motionConfig.sh to configure motioneye.
See ./batmanConfig.sh to install & configure BATMAN-adv.
See ./batmanRC.sh to initialize the network
See batmanBATSpace.service for a systemd service framework

return to index
Link- Scripting Octoprint || https://github.com/Jesssullivan/motioneye-mesh/tree/master/USBoN

USBoN

Serial over Network Plugin for Octoprint

remote setup:

# install depends:
pip3 install Requests
sudo apt-get install usbip
# nohup usbipd &  # starts daemon, this is run from the python script 

# move files to destinations:
cp Py3_client.py ~
sudo cp client.service /etc/systemd/client.service

# start systemd service:
sudo systemctl daemon-reload
sudo systemctl enable client.service

# reboot for changes to take effect:
sudo reboot

server setup: 

# install depends:
sudo apt-get install linux-tools-generic
sudo modprobe vhci-hcd
python3 Py3_server.py

Status

Intended for remote Pi Zeros (raspian + motioneye) installed around our facility.
A central server manages motioneye streams, OctoPrint, job slicing, etc.

Serial device (e.g. 3d printer) connected to a remote client is mounted on the server using the Debian usbip package.

As of 10/28/19, scripts / functions have yet to be merged into the OctoPrint plugin scaffold.

return to index
Link- Scripting for WiFi Captive Portal ||
|| https://www.transscendsurvival.org/2019/11/28/1568/
https://raw.githubusercontent.com/Jesssullivan/misc/master/sel/portal.py

wget https://raw.githubusercontent.com/Jesssullivan/misc/master/sel/portal.py

Some distros (Ubuntu and its derivatives on my Macbook for instance) struggle to find a route to the captive portal on public networks (read: Starbucks). Assuming you want to connect the way they intend (via the gateway through the captive portal) because you are a rule abiding patron, all you need to do is visit the address of the gateway. There is no need to fiddle with your network drivers, disable SSL stuff or engage in plebian network tomfoolery.

"""
find and open wifi captive portal (such as Starbucks)
written by Jess Sullivan
"""
from netifaces import gateways
from sys import argv
from time import sleep
import subprocess

help_str = str("\n " +
               'usage: \n ' +
               ' -h : print this message again \n' +
               ' -i : `pip3 install netifaces`  \n' +
               'You may specify a browser argument to complete the portal, such as \n' +
               'google-chrome')

def argtype():
    try:
        if len(argv) > 1:
            use = True
        elif len(argv) == 1:
            use = False
            print(help_str)
        else:
            print('command takes 0 or 1 args, use -h for help')
            raise SystemExit
    except:
        print('arg error... \n command takes 0 or 1 args, use -h for help')
        raise SystemExit
    return use

def main():

    all_gates = gateways()
    target = all_gates['default'][2][0]

    if argtype():

        if argv[1] == '-h':
            print(help_str)
            quit()

        if argv[1] == '-i':
            try:
                subprocess.Popen('pip3 install netifaces',
                                 shell=True,
                                 executable='/bin/bash')
                sleep(1)
                quit()
            except:
                print('err running pip3 install netifaces')
                quit()

        else:

            print(str('opening portal in ' + argv[1]))
            subprocess.Popen(str(argv[1] + ' ' + str(target)),
                             shell=True,
                             executable='/bin/bash')
            sleep(1)
            quit()

    else:

        print(str('\n please visit address ' + target +
                  ' in a browser to complete portal setup \n'))
        quit()

main()