Trans Scend Survival

Trans: Latin prefix implying "across" or "Beyond", often used in gender nonconforming situations – Scend: Archaic word describing a strong "surge" or "wave", originating with 15th century english sailors – Survival: 15th century english compound word describing an existence only worth transcending.

Author: Jess (Page 5 of 10)

Quick fix: 254 character limit in ESRI Story Map?

September 12, 2018 / / 0 Comments

https://gis.stackexchange.com/questions/75092/maximum-length-of-text-fields-in-shapefile-and-geodatabase-formats

https://en.wikipedia.org/wiki/GeoJSON

https://gis.stackexchange.com/questions/92885/ogr2ogr-converting-kml-to-geojson

If you happened to be working with….  KML data (or any data with large description strings) and transitioning it into the ESRI Story Map toolset, there is a very good chance  you hit the the dBase 254 character length limit with the ESRI Shapefile upload.  Shapefiles are always a terrible idea.

 

the solution:  with GDAL or QGIS (alright, even in ArcMap), one can use GeoJSON as an output format AND import into the story map system- with complete long description strings!

 

QGIS:

Merge vector layers -> save to file -> GeoJSON

arcpy:
import arcpy

import os

arcpy.env.workspace = “/desktop/arcmapstuff”

arcpy.FeaturesToJSON_conversion(os.path.join(“outgdb.gdb”, “myfeatures”), “output.json”)

GDAL:
<
ogr2ogr -f GeoJSON output.json input.kml

New App:  KML Search and Convert

July 27, 2018 / / 0 Comments

Written in R; using GDAL/EXPAT libraries on Ubuntu and hosted with AWS EC2.

New App:  KML Search and Convert

Here is an simple (beta) app of mine that converts KML files into Excel-friendly CSV documents.  It also has a search function, so you can download a subset of data that contains keywords.   🙂

The files will soon be available in Github.

I’m still working on a progress indicator; it currently lets you download before it is done processing.   Know a completely processed file is titled with “kml2csv_<yourfile>.csv”.

…YMMV.  xD

GDAL for R Server on Ubuntu – KML Spatial Libraries and More

July 23, 2018 / / 2 Comments

GDAL for R Server on Red Hat Xenial Ubuntu – KML Spatial Libraries and More

If you made the (possible mistake) of running with a barebones Red Hat Linux instance, you will find it is missing many things you may want in R.   I rely on GDAL (the definitive Geospatial Data Abstraction Library) on my local OSX R setup, and want it on my server too.  GDAL contains many libraries you need to work with KML, RGDAL, and other spatial packages.  It is massive and usually take a long time to sort out on any machine.

These notes assume you are already involved with a R server (usually port 8787 in a browser).  I am running mine from an EC2 instance with AWS.

! Note this is a fresh server install, using Ubuntu; I messed up my original ones while trying to configure GDAL against conflicting packages. If you are creating a new one, opt for at least a T2 medium (or go bigger) and find the latest Ubuntu server AMI.  For these instructions, you want an OS that is as generic as possible.

On Github:

https://github.com/Jesssullivan/rhel-bits

From Bash:

# SSH into the EC2 instance: (here is the syntax just in case)

#ssh -i “/Users/YourSSHKey.pem” ec2-user@yourAWSinstance.amazonaws.com

sudo su –

apt-get update

apt-get upgrade

nano /etc/apt/sources.list

#enter as a new line at the bottom of the doc:

deb https://cloud.r-project.org/bin/linux/ubuntu xenial/

#exit nano

wget https://raw.githubusercontent.com/Jesssullivan/rhel-bits/master/xen-conf.sh

chmod 777 xen-conf.sh

./xen-conf.sh

Or…

From SSH:

# SSH into the EC2 instance: (here is the syntax just in case)

ssh -i “/Users/YourSSHKey.pem” ec2-user@yourAWSinstance.amazonaws.com

# if you can, become root and make some global users- these will be your access to

# RStudio Server and shiny too!

sudo su –

adduser <Jess>

# Follow the following prompts carefully to create the user

apt-get update

nano /etc/apt/sources.list

# enter as a new line at the bottom of the doc:

deb https://cloud.r-project.org/bin/linux/ubuntu xenial/

# exit nano

# Start, or try bash:

apt-get install r-base

apt-get install r-base-dev

apt-get update

apt-get upgrade

wget http://download.osgeo.org/gdal/2.3.1/gdal-2.3.1.tar.gz

tar xvf gdal-2.3.1.tar.gz

cd  gdal-2.3.1

# begin making GDAL: this all takes a while

./configure  [if your need proper kml support (like me), search on configuring with expat or libkml.   There are many more options for configuration based on other packages that can go here, and this is the step to get them in order…]

sudo make

sudo make install

cd # Try entering R now and check the version!

# Start installing RStudio server and Shiny

apt-get update

apt-get upgrade
sudo apt-get install gdebi-core
wget https://download2.rstudio.org/rstudio-server-1.1.456-amd64.deb
sudo gdebi rstudio-server-1.1.456-amd64.deb

# Enter R or go to the graphical R Studio installation in your browser

R

# Authenticate if using the graphical interface using the usr:pwd you defined earlier

# this will take a long time

install.packages(“rgdal”)

# Note any errors carefully!

Then:

install.packages(“dplyr”)

install.packages(c(“data.table”, “tidyverse”, “shiny”)  # etc

Well, there you have it!

-Jess

Extras:

##Later, ONLY IF you NEED Anaconda, FYI:

# Get Anaconda: this is a large package manager, and is could be used for patching up missing # dependencies:

#Use  “ls” followed by rm -r <anaconda> (fill in with ls results) to remove conflicting conda

# installers if you have any issue there, I am starting fresh:

mkdir binconda

# *making a weak attempt at sandboxing the massive new package manager installation*

cd binconda
wget http://repo.continuum.io/archive/Anaconda2-4.3.0-Linux-x86_64.sh
# install and follow the prompts
bash Anaconda2-5.2.0-Linux-x86_64.sh

# Close the terminal window completely and start a new one, and ssh back to where you left

# off.  Conda install requires this.

# open and SSH back into your instance.  You should now have either additional flexibility in

# either patching holes in dependencies, or created some large holes in your server.  YMMV.

### Done

Red Hat stuff:

Follow these AWS instructions if you are doing something else:

https://aws.amazon.com/blogs/big-data/running-r-on-aws/

See my notes on this here:

https://www.transscendsurvival.org/2018/03/08/how-to-make-a-aws-r-server/

and notes on Shiny server:

https://www.transscendsurvival.org/2018/07/16/deploy-a-shiny-web-app-in-r-using-aws-ec2-red-hat/

GDAL on Red Hat:- Existing threads on this:

https://gis.stackexchange.com/questions/120101/building-gdal-with-libkml-support/120103#120103

This is a nice short thread about building from source:

https://gis.stackexchange.com/questions/263495/how-to-install-gdal-on-centos-7-4

neat RPM package finding tool, just in case:

https://rpmfind.net/linux/rpm2html/

Info on the LIBKML driver if you end up with issues there:

http://www.gdal.org/drv_libkml.html

 

I hope this is useful- GDAL is important and best to set it up early.  It will be a pain, but so is losing work while trying to patch it in later.  xD

 

-Jess

 

INFO: Deploy a Shiny web app in R using AWS (EC2 Red Hat)

July 16, 2018 / / 0 Comments

Info on deploying a Shiny web app in R using AWS (EC2 Redhat)

As a follow-up to my post on how to create an AWS RStudio server, the next logical step is to host some useful apps you created in R for people to use.  A common way to do this is the R-specific tool Shiny, which is built in to RStudio.  Learning the syntax to convert R code into a Shiny app is rather subtle, and can be hard.  I plan to do a more thorough demo on this- particularly the use of the $ symbol, as in “input$output”- later. 🙂

 

It turns out hosting a Shiny Web app provides a large number of opportunities for things to go wrong….  I will share what worked for me.  All of this info is accessed via SSH, to the server running Shiny and RStudio.

 

I am using the AWS “Linux 2” AMI, which is based on the Red Hat OS.  For reference, here is some extremely important Red Hat CLI language worth being familiar with and debugging:

 

“sudo yum install” and “wget” are for fetching and installing things like shiny.  Don’t bother with instructions that include “apt-get install”, as they are for a different Linux OS!

 

“sudo chmod -R 777” is how you change your directory permissions for read, write, and execute (all of those enabled).  This is handy if your server disconnecting when the app tries to run something- it is a simple fix to a problem not always evident in the logs.  The default root folder from which shiny apps are hosted and run is “/srv/shiny-server” (or just “/srv” to be safe).

 

“nano /var/log/shiny-server.log” is the location of current shiny logs.

 

“sudo stop shiny-server” followed by “sudo start shiny-server” is the best way to restart the server- “sudo restart shiny-server” is not a sure bet on any other process.  It is true, other tools like a node.js server or nginx could impact the success of Shiny- If you think nginx is a problem, “cd /ect/nginx” followed by “ls” will get you in the right direction.  Others have cited problems with Red Hat not including the directories and files at “/etc/nginx/sites-available”.  You do not need these directories.  (though they are probably important for other things).

 

“sudo rm -r” is a good way to destroy things, like a mangled R studio installation.  Remember, it is easy enough to start again fresh!  🙂

 

“sudo nano /etc/shiny-server/shiny-server.conf” is how to access the config file for Shiny.  The fresh install version I used did not work!  There will be lots of excess in that file, much of which can causes issues in a bare-bones setup like mine.  One important key is to ensure Shiny is using a root user- see my example file below.  I am the root user here (jess)- change that to mirror- at least for the beginning- the user defined as root in your AWS installation.  See my notes HERE on that- that is defined in the advanced settings of the EC2 instance.

 

BEGIN CONFIG FILE:   (or click to download) *Download is properly indented


# Define user: this should be the same user as the AWS root user!
#
run_as jess;
#
# Define port and where the home (/) directory is
# Define site_dir/log_dir - these are the defaults
#
server{
listen 3838;
location / {
site_dir /srv/shiny-server;
log_dir /var/log/shiny-server;
directory_index on;
}
}

END CONFIG FILE

Well, the proof is in the pudding.   At least for now, you can access a basic app I made that cleans csv field data files that where entered into excel by hand.  They start full of missing fields and have a weird two-column setup for distance- the app cleans all these issues and returns a 4 column (from 5 column) csv.

Download the test file here:   2012_dirt_PCD-git

And access the app here:  Basic Shiny app on AWS!

Below is an iFrame into the app, just to show how very basic it is.  Give it a go!

-Jess

Off-Grid File Sharing with SAMBA / GL.iNet

June 25, 2018 / / 0 Comments

Note:  SMB / SharePoint is surely better with a proper server/computer.  A Raspberry Pi running OpenMediaVault (Debian) is a more common and robust option (still 5v low power).

If you are actually in an “it must done in OpenWRT” scenario, Click Here for my Samba config file: OpenWRT_Samba-config and see below.  Also, please use a NTFS or EX4 format.  🙂

 

…While my sharing method wasn’t actually adopted by others, I still think it is good to know!

-Jess

How to Query KML point data as CSV using QGIS and R

June 10, 2018 / / 0 Comments

How to Query KML point data as CSV using QGIS and R

Here you can see more than 800 points, each describing an observation of an individual bird.  This data is in the form of KML, a sort of XML document from Google for spatial data.

 

I want to know which points have “pair” or “female” in the description text nodes using R.  This way, I can quickly make and update a .csv in Excel of only the paired birds (based on color bands).

 

 

Even if there was a description string search function in Google Earth Pro (or other organization-centric GIS/waypoint software), this method is more

robust, as I can work immediately with the output as a data frame in R, rather than a list of results.

 

First, open an instance of QGIS.  I am running ~2.8 on OSX.  Add a vector layer of your KML.

“Command-A” in the point dialog to select all before import!

Next, under “Vector”, select “Merge vector layers” via Data Management Tools.

 

Select CSV and elect to save the file instead of use a temporary/scratch file (this is a common error).

Open your csv in Excel for verification! 

 

 

 

 

 

 

The R bit:

# query for paired birds

#EDIT:  Libraries
library(data.table)
library(tidyverse)

data <- data.frame(fread("Bird_CSV.csv"))

pair_rows <- contains("pair", vars = data$description)

fem_rows <- contains("fem", vars = data$description)

result <- combine(pair_rows, fem_rows)

result <- data[result,]

write_csv(result, "Paired_Birds.csv")

Tada!

 

 

 

 

 

 

 

 

-Jess

Visual NH Research Update :)

June 8, 2018 / / 1 Comment

Olive-sided Flycatcher (Is Says, “Three Beers!! [please]”

Walking to work in the morning

Work

Song Sparrow

Common Yellowthroat Warbler

I’ll let the photos to the talking:  welcome to my world! 🙂 !!!!

 

-Jess

 

 

Solar upgrades!

May 31, 2018 / / 0 Comments

Solar upgrades!

Incredibly, the hut we are working from actually had another solar panel just laying around.  🙂
This 50w square panel had a junction box with MC4 connectors, the standard for small scale solar installations.  As I was unsure how to know when we are running low on electricity reserves, I decided to make some adjustments.

Additional 50w solar panel

(Everything is still solder, hot glue, alligator clips, and zip-ties I’m afraid…)
I traded my NEMA / USA two-prong connection with two MC4 splitters, such that both panels can run in parallel (into a standard USA 110v extension cord that goes into our hut).  This way we should make well over one of the two 35ah batteries-worth of electricity a day.

Dual MC4 splitters to extension cord

I also added a cheap 12v battery level indicator.  It is not very accurate (as it fluctuates with solar input) but it does give us some insight about how much “juice” we have available.  (I also wired and glued the remote-on switch to the back of the input for stability.)

Added battery indicator and button

🙂
-Jess

Research Year Two: Three Photos

May 31, 2018 / / 0 Comments

Male Common Yellowthroat Warbler

The field season has officially started in Northern NH!

Male Common Yellowthroat warbler (COYE):   This fellow is defending a small territory in a patch of open thicket.   These warblers rely on early succession forest- patches of substrate that haven’t  really grown in yet- to build cryptic, ground-level nests.  They develop complex systems to divert/confuse predators away from their nests.

 

Female Black-throated Blue Warbler (BTBW):  I was lucky to see this female.   She is paired with a male who defends a large mature forest territory.   They have quite a few BTBW neighbors, which makes for a lot of skirmishes among the males over land.  The females are often silent and move very fast…

Male Mourning Warbler (MOWA):  This is a rare bird here.   Even more amazing, it is defending a territory in our research site- and trying to chase out a male COYE while doing so.  The two species “share” resources, which means thy can’t stand each other.   🙂   Each time the male COYE sings near the MOWA, it gets berated and chased away- and vice versa.   It appears the COYE isn’t budging either, probably because it hasn’t had this domestic, neighborly problem before.

-Jess

Female Black-throated Blue Warbler

Male Mourning Warbler

Gathering point data using Compass 55 on Apple iOS

May 28, 2018 / / 0 Comments

Keeping track of birds is tricky!

Attached is our team’s workflow with Compass 55.    From the Kml, we go into Google Earth Pro – ArcGIS Desktop (arcmap).   QGIS is sometimes used too.

 

Cheers,

-Jess

 

840 Watts of Solar Power!

May 24, 2018 / / 0 Comments

Equipment used:

Inverter/PWM Controller:  http://a.co/fdl9YzI

2x 35ah Batteries: http://a.co/5JBIxTC

100w solar panel:  http://a.co/5JBIxTC

We need power!  While doing bird research in the wilds of northern NH, it became evident we needed electricity to power computers, big cameras, and phones/GPS units.

Below is a table of the system and our expected electricity needs:

System Solar 100w 35ah universal (x2)
Ah per day: 33.33333333 35 TOTAL Ah Reserve: 70
V 12 12 Parallel wiring: 12v
Wh in: 400 420 TOTAL Wh Reserve: 840
W 100
Cost $105.00 $64.00
ah/$ 2
Sun Hour / Multiplier 4 2
Need/Day Wh multiplier consump. in Wh = 259.36
Computer 100 2.5 250
iPhone 1.7 2 3.4
AAs 11.2 0.3 3.36
Camera 2.6 1 2.6

*The milk crate system below can charge a 100 watt MacBook Pro around 8-9 times from being completely empty.  

**Remember:  V*A=W,  W/V=A, and Watts over time is Wh.  

-Jess

+/- relates to size of standard prongs

Parallel maintains 12v but doubles Ah. (Series would go to 24v at 35ah)

 

« Older posts Newer posts »

© 2024 Trans Scend Survival

α wιρ Σ ♥ by Jess Sullivan — Up ↑