Trans: Latin prefix implying "across" or "Beyond", often used in gender nonconforming situations – Scend: Archaic word describing a strong "surge" or "wave", originating with 15th century english sailors – Survival: 15th century english compound word describing an existence only worth transcending.

Category: Featured (Page 4 of 5)

INFO: Deploy a Shiny web app in R using AWS (EC2 Red Hat)

Info on deploying a Shiny web app in R using AWS (EC2 Redhat)

As a follow-up to my post on how to create an AWS RStudio server, the next logical step is to host some useful apps you created in R for people to use.  A common way to do this is the R-specific tool Shiny, which is built in to RStudio.  Learning the syntax to convert R code into a Shiny app is rather subtle, and can be hard.  I plan to do a more thorough demo on this- particularly the use of the $ symbol, as in “input$output”- later. 🙂

 

It turns out hosting a Shiny Web app provides a large number of opportunities for things to go wrong….  I will share what worked for me.  All of this info is accessed via SSH, to the server running Shiny and RStudio.

 

I am using the AWS “Linux 2” AMI, which is based on the Red Hat OS.  For reference, here is some extremely important Red Hat CLI language worth being familiar with and debugging:

 

sudo yum install” and “wget” are for fetching and installing things like shiny.  Don’t bother with instructions that include “apt-get install”, as they are for a different Linux OS!

 

sudo chmod -R 777” is how you change your directory permissions for read, write, and execute (all of those enabled).  This is handy if your server disconnecting when the app tries to run something- it is a simple fix to a problem not always evident in the logs.  The default root folder from which shiny apps are hosted and run is “/srv/shiny-server” (or just “/srv” to be safe).

 

nano /var/log/shiny-server.log” is the location of current shiny logs.

 

sudo stop shiny-server” followed by “sudo start shiny-server” is the best way to restart the server- “sudo restart shiny-server” is not a sure bet on any other process.  It is true, other tools like a node.js server or nginx could impact the success of Shiny- If you think nginx is a problem, “cd /ect/nginx” followed by “ls” will get you in the right direction.  Others have cited problems with Red Hat not including the directories and files at “/etc/nginx/sites-available”.  You do not need these directories.  (though they are probably important for other things).

 

sudo rm -r” is a good way to destroy things, like a mangled R studio installation.  Remember, it is easy enough to start again fresh!  🙂

 

sudo nano /etc/shiny-server/shiny-server.conf” is how to access the config file for Shiny.  The fresh install version I used did not work!  There will be lots of excess in that file, much of which can causes issues in a bare-bones setup like mine.  One important key is to ensure Shiny is using a root user- see my example file below.  I am the root user here (jess)- change that to mirror- at least for the beginning- the user defined as root in your AWS installation.  See my notes HERE on that- that is defined in the advanced settings of the EC2 instance.

 

BEGIN CONFIG FILE:   (or click to download) *Download is properly indented


# Define user: this should be the same user as the AWS root user!
#
run_as jess;
#
# Define port and where the home (/) directory is
# Define site_dir/log_dir - these are the defaults
#
server{
listen 3838;
location / {
site_dir /srv/shiny-server;
log_dir /var/log/shiny-server;
directory_index on;
}
}

END CONFIG FILE

Well, the proof is in the pudding.   At least for now, you can access a basic app I made that cleans csv field data files that where entered into excel by hand.  They start full of missing fields and have a weird two-column setup for distance- the app cleans all these issues and returns a 4 column (from 5 column) csv.

Download the test file here:   2012_dirt_PCD-git

And access the app here:  Basic Shiny app on AWS!

Below is an iFrame into the app, just to show how very basic it is.  Give it a go!

-Jess

How to Query KML point data as CSV using QGIS and R

How to Query KML point data as CSV using QGIS and R

Here you can see more than 800 points, each describing an observation of an individual bird.  This data is in the form of KML, a sort of XML document from Google for spatial data.

 

I want to know which points have “pair” or “female” in the description text nodes using R.  This way, I can quickly make and update a .csv in Excel of only the paired birds (based on color bands).

 

 

Even if there was a description string search function in Google Earth Pro (or other organization-centric GIS/waypoint software), this method is more

robust, as I can work immediately with the output as a data frame in R, rather than a list of results.

 

First, open an instance of QGIS.  I am running ~2.8 on OSX.  Add a vector layer of your KML.

“Command-A” in the point dialog to select all before import!

Next, under “Vector”, select “Merge vector layers” via Data Management Tools.

 

Select CSV and elect to save the file instead of use a temporary/scratch file (this is a common error).

Open your csv in Excel for verification! 

 

 

 

 

 

 

The R bit:

# query for paired birds

#EDIT:  Libraries
library(data.table)
library(tidyverse)

data <- data.frame(fread("Bird_CSV.csv"))

pair_rows <- contains("pair", vars = data$description)

fem_rows <- contains("fem", vars = data$description)

result <- combine(pair_rows, fem_rows)

result <- data[result,]

write_csv(result, "Paired_Birds.csv")

Tada!

 

 

 

 

 

 

 

 

-Jess

Visual NH Research Update :)

Olive-sided Flycatcher (Is Says, “Three Beers!! [please]”

Walking to work in the morning

Work

Song Sparrow

Common Yellowthroat Warbler

I’ll let the photos to the talking:  welcome to my world! 🙂 !!!!

 

-Jess

 

 

Solar upgrades!

Solar upgrades!

Incredibly, the hut we are working from actually had another solar panel just laying around.  🙂
This 50w square panel had a junction box with MC4 connectors, the standard for small scale solar installations.  As I was unsure how to know when we are running low on electricity reserves, I decided to make some adjustments.

Additional 50w solar panel

(Everything is still solder, hot glue, alligator clips, and zip-ties I’m afraid…)
I traded my NEMA / USA two-prong connection with two MC4 splitters, such that both panels can run in parallel (into a standard USA 110v extension cord that goes into our hut).  This way we should make well over one of the two 35ah batteries-worth of electricity a day.

Dual MC4 splitters to extension cord

I also added a cheap 12v battery level indicator.  It is not very accurate (as it fluctuates with solar input) but it does give us some insight about how much “juice” we have available.  (I also wired and glued the remote-on switch to the back of the input for stability.)

Added battery indicator and button

🙂
-Jess

Research Year Two: Three Photos

Male Common Yellowthroat Warbler

The field season has officially started in Northern NH!

Male Common Yellowthroat warbler (COYE):   This fellow is defending a small territory in a patch of open thicket.   These warblers rely on early succession forest- patches of substrate that haven’t  really grown in yet- to build cryptic, ground-level nests.  They develop complex systems to divert/confuse predators away from their nests.

 

Female Black-throated Blue Warbler (BTBW):  I was lucky to see this female.   She is paired with a male who defends a large mature forest territory.   They have quite a few BTBW neighbors, which makes for a lot of skirmishes among the males over land.  The females are often silent and move very fast…

Male Mourning Warbler (MOWA):  This is a rare bird here.   Even more amazing, it is defending a territory in our research site- and trying to chase out a male COYE while doing so.  The two species “share” resources, which means thy can’t stand each other.   🙂   Each time the male COYE sings near the MOWA, it gets berated and chased away- and vice versa.   It appears the COYE isn’t budging either, probably because it hasn’t had this domestic, neighborly problem before.

-Jess

Female Black-throated Blue Warbler

Male Mourning Warbler

Gathering point data using Compass 55 on Apple iOS

Keeping track of birds is tricky!

Attached is our team’s workflow with Compass 55.    From the Kml, we go into Google Earth Pro – ArcGIS Desktop (arcmap).   QGIS is sometimes used too.

 

Cheers,

-Jess

 

Intro to the AWS Cloud 9 IDE

The Cloud 9 IDE is the fastest way I have come up with to develop web-based or otherwise “connected” programs.    Because it lives on a Linux-based EC2 server on AWS, running different node, html, etc programs that rely on a network system just work- it is all already on a network anyway.   🙂  There is no downtime trying to figure out your WAMP, MAMP, Apache, or localhost situation.

Similarly, other network programs work just as well-  I am running a MySQL server over here (RDS), storage over there (S3), and have various bits in Github and locally.   Instead of configuring local editors, permissions, and computer ports and whatnot, you are modifying the VPC security policies and IAM groups- though generally, it just works.

Getting going:   The only prerequisite is you have an AWS account.  Students:  get $40 EC2 dollars below:

https://aws.amazon.com/education/awseducate/
Open the cloud 9 tab under services.

 

 

Setup is very fast- just know if others are going to be editing to, understand the IAM policies and what VPC settings you actually want.

 

Know this ideally a browser-based service; I have tried to come up with a reason a SSH connection would be better and didn’t get any where.

For one person, micro is fine.   Know these virtual “RAMs” and “CPUs” are generous….

 

 

 

 

The default network settings are set up for you.   This follows good practice for one person; more than that (or if you are perhaps a far-travelling person) note these settings.  They are always editable under the VPC and EC2 instance tabs.

 

 

That’s it!   Other use things to know:

This is a linux machine maintained by Amazon.   Packages you think should work and be up to date (arguably like any other linux machine I guess…)  may not be.  Check your basics like the NPM installer and versions of what your going to be working on, it very well may be different than what you are used to.

In the editor:

You have two panels of workspace in the middle- shown is node and HTML.   Everything is managed by tabs- all windows can have as much stuff as you want this way.

Below there is a “runner” (shown with all the default options!) and a terminal window.  Off to the left is a generic file manager.

 

 

I hope this is useful, it sure is great for me.

-Jess

Using ESRI ArcGIS / ArcMap in the AWS Cloud

Selling AWS to… myself   🙂

Why struggle with underpowered local machines and VMs or watered-down web platforms for heavy lifting,  learning and work?

In addition to using ESRI software on mac computers, I am a big fan of the AWS WorkSpaces service (in addition to all their other developer tools, some of which are map-relevant: RDS for SQL and EC2 Redhat servers  for data management for example ).

Basically, for between ~$20 and ~$60 a month (Max, and not factoring in EDU discounts!), a user gets to use a well-oiled remote desktop.   You can download and license desktop apps like ArcMap and GIS products, file managers, and more from any computer connected to the internet.  This service is not very savvy; you make/receive a password and log right in.

A  big plus here of course is the Workspaces Application Manager (WAM); small sets of licenses can be administered in the same way desktops would, with extra easiness due to the “they are already really the same cloud thing any way”.

Another plus is any client- netbook, macbook, VM, etc- will work equally well.  In this regard it can be a very cheap way to get big data work done on otherwise insufficient machines.  Local storage and file systems work well with the client application, with the caveat being network speed.

🙂

https://docs.aws.amazon.com/workspaces/latest/adminguide/amazon-workspaces.html

 

 

 

 

 

 

Using ESRI ArcGIS / ArcMap on Mac OSX: 2 methods

Edit 07/26/2020:
Check out the expanded GIS notes page here!

Using ESRI ArcGIS / ArcMap on Macs: 2 methods

I need to run ESRI products on my MacBook Pro.   QGIS is always the prefered solution- open source, excellent free plugins, works on mac natively- but in a college / research environment, the only option that supports other people and school machines is ESRI.  Despite the annoying bureaucracy and expense of the software, some things are faster (but not better!) in ESRI, like dealing with raster / multiband data.

First, you need a license.

I went about this two ways;

My first solution was to buy an ESRI Press textbook on amazon.  A 180 day trial for $50- when taken as a college course, this isn’t to bad.  🙂   The book is slow and recursive, but a 180 days to play with all the plugins and whistles allows for way deeper learning via the internet.   🙂

Do know there is a little-documented limit to the number of license transfers you may perform before getting either lock in or out of your software.  I hit this limit, as I was also figuring out my virtual machine situation, which would occasionally need a re-installation.

My current solution is “just buy a student license”.   $100 per year is less than any adobe situation- so really not that bad.  

Now you need a windows ISO.  

https://www.microsoft.com/en-us/software-download/windows10ISO

Follow that link for the window 10, 64 bit ISO.  YOU DO NOT NEED TO BUY WINDOWS.  It will sometimes complain about not having an  authentication, but in the months of using windows via VMs, never have I been prohibited to do… anything.  When prompted for a license when configuring your VM, click the button that says "I don’t have a license".  Done.

 

Option one:  VirtualBox VM on a thumbdrive

https://www.virtualbox.org/wiki/Downloads – download for the VM software

http://a.co/4FEYMNY, http://a.co/hanHYl1 Suitable USBs.  the VM will take up most of a 128gb flash drive- ~70 gb just for windows and all the stuff you’ll want from a PC.  Add ESRI software and allocated space for a cache (where your GIS project works!), bigger is better.   Format all drives in disk utility as ExFat!  this is important, any other file system either won’t fly or could wreak havoc (other FAT based ones may have too small file allocations!

I used two drives, a 128 and a 64- this is great because I can store all my work on the 64, so I can easily plug it into other (school) machines running windows ArcMap and keep going, without causing issues with the massive VM in the 128.  

Installation is straightforward, just install EVERYTHING on the usb drive and it will be fine.   🙂

Problems:   Stability.   Crashes, and python / some other script modules do not work well.  This is a problem.  ArcAdministrator gets confused about all kinds of things- FWIW, if you are googling to delete the FLEXnet folder to solve authentication file issues, move to option 2 🙂

Speed is down, but actually the ~same speed as our school "super" PCs- (though I happened to know they are essentially glorified "hybrid" VMs too!) .

Option two: OSX Bootcamp 

https://support.apple.com/boot-camp

https://support.apple.com/en-us/HT201468

This way, you will hit "option/alt" each time you restart/boot your computer to choose from win/osx.   This is easy to install, as it is mac and mac = easy.

Big Caveat:  it is much harder to install windows externally  (on a usb, etc) from bootcamp.  I didn’t succeed in my efforts, but there could be a way….   The thing is, it really wants to run everything like a normal intel based PC, with all installations in the usual place.  This is good for the mac performance, but terrible for the tiny SSD hard drives we get as mac users.  I have a 256gb SSD.  I have an average of < 15 gb wiggle room here, and use every cloud service in the book.

If you need to manage your cloud storage because of a itsy mac SSD, my solution is still ODrive.   https://www.odrive.com/

I use Amazon cloud mostly with odrive, but I use a personal/school OneDrives, Dropboxes, Google,  etc.  with only the occasional hiccup.   Also, all of the AWS tools are great and cheap- EC2, S3, Cloud 9, lambda, RDS…. Great way to do your work outside of your mac via the internet.

Result:

ArcMap and GIS stuff is blazing fast on my modest 2015 i5/8gb macbook pro.  Comparing a huge, mega ATX+ school computer to my mac on boot camp, I am running large raster filtering operations significantly quicker than other folks doing the same type of work.   That is GOOD.

🙂

-Jess

How to make a AWS R server

When you need an R server and have lots of data to process, AWS is a great way to go.   Sign up of the free tier and poke around!

Creating an AWS Rstudio server:

https://aws.amazon.com/blogs/big-data/running-r-on-aws/ – using both the R snippet (works but the R core bits are NOT present and it will not work yet) and the JSON snippet provided  

https://www.rstudio.com/products/rstudio/download-server/ – the suite being installed

Follow most of the AWS blog AMI info, with the following items:

AMI:  Amazon Linux 2 (more packages and extras v. standard)  

  • t2.micro (free tier)
  • IAM policy follows AWS blog JSON snippet
  • Security Policy contains open inbound ports 22, 8787, 3838 (the latter two for R server specific communication)
  • Append user, username:password in the blog post’s initial r studio install text (pasted into the “advanced” text box when completing the AMI setup

 

SSH into the EC2 instance

sudo yum install –y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

sudo yum-config-manager –enable epel

sudo yum repolist

wget https://download2.rstudio.org/rstudio-server-rhel-1.1.423-x86_64.rpm

sudo yum update -y

sudo yum install -y R

sudo rstudio-server verify-installation

 

Access the graphical R server:

In a web browser, tack on “:8787” to the end of the Instance’s public “connect” link.  If it doesn’t load a login window (but seems to be trying to connect to something) the security policy is probably being overzealous……..

 

Notes on S3-hosted data:

  • S3 data is easiest to use if it is set to be public.
  • There are s3-specific tools for R, accessible as packages from CRAN directly from the R interface
  • Note data (delimited text at least) hosted in S3 will behave differently than it does locally, e.g. spaces, “na”, “null” need to be “cleaned” in R before use.  

 

There we have it!

 

-Jess

« Older posts Newer posts »

© 2024 Trans Scend Survival

α wιρ Σ ♥ by Jess SullivanUp ↑