Terrific birds today- on a pleasant bird walk here in Ithaca on May 19th I recorded 43 species! Here is my eBird checklist for those curious.
Category: Birding (Page 1 of 3)
….Even more demos @ ai.columbari.us
miscellaneous dregs, bits, bobs, demos in this playlist on youtube
- handling multiple annotator clients- Vue UI, 2/17/21
- handling multiple annotator clients- Leaflet UI, 2/7/21
- Jest x Puppeteer, testing annotator UI
- Bits from 1.21.21
- preformance updates to core leaflet annotator, 1/17/21
- mongodb, trust schema, drag & drop 12/7/20
- web tools, bbox classify, ridiculous CLI 12/1/20
- audio & photo annotators 11/23/20
- what this swiftui thing 10/26/20
get this script and other GIS bits here on github
The Ebird dataset is awesome. While directly handling data as a massive delimited file- as distributed by the eBird people- is cumbersome at best, the ebird api offers a fairly straightforward and efficient alternative for a few choice bits and batches of data.
-
The eBird
AWKtool for filtering the actual delimited data can be found over here:install.packages("auk")
It is worth noting R + auk (or frankly any R centered filtering method) will quickly become limited by the single-threaded approach of R, and how you’re managing memory as you iterate. Working and querying the data from a proper database quickly becomes necessary.
Most conveniently, the eBird API already exists- snag an key over here.
…The API package for R is over here:
install.packages("rebird")
…There is also a neat Python wrapper over here:
pip3 install ebird-api
Region Codes:
I’m not sure why, but some methods use normal latitude / longitude in decimal degrees while some others use "regionCode", which seems to be some kind of eBird special. Only ever seen this format in ebird data.
For example, recent observations uses regionCode:
# GET Recent observations in a region:
# https://api.ebird.org/v2/data/obs/{{regionCode}}/recent…But nearby recent observations uses latitude / longitude:
# GET Recent nearby observations:
# https://api.ebird.org/v2/data/obs/geo/recent?lat={{lat}}&lng={{lng}}Regardless, lets just write a function to convert decimal degrees to this regionCode thing. Here’s mine:
#!/usr/bin/env python3
"""
# provide latitude & longitude, return eBird "regionCode"
Written by Jess Sullivan
@ https://www.transscendsurvival.org/
available at:
https://raw.githubusercontent.com/Jesssullivan/GIS_Shortcuts/master/regioncodes.py
"""
import requests
import json
def get_regioncode(lat, lon):
# this municipal api is a publicly available, no keys needed afaict
census_url = str('https://geo.fcc.gov/api/census/area?lat=' +
str(lat) +
'&lon=' +
str(lon) +
'&format=json')
# send out a GET request:
payload = {}
get = requests.request("GET", census_url, data=payload)
# parse the response, all api values are contained in list 'results':
response = json.loads(get.content)['results'][0]
# use the last three digits from the in-state fips code as the "subnational 2" identifier:
fips = response['county_fips']
# assemble and return the "subnational type 2" code:
regioncode = 'US-' + response['state_code'] + '-' + fips[2] + fips[3] + fips[4]
print('formed region code: ' + regioncode)
return regioncode
The first ones to arrive in MA, brush up!
Palm warbler
This is usually the first one to arrive. Gold bird, medium sized warbler, rufus hat. When they arrive in MA they are often found lower than usual / on the ground looking for anything they can munch on. Song is a rapid trill. More “musical / pleasant” than a fast chipping sparrow, faster than many Junco trills.
Pine warbler
Slimmer than palm, no hat, very slim beak, has streaks on the breast usually. Also a triller. They remain higher in the trees on arrival.
Yellow-rumped warbler
Spectacular bird, if it has arrived you can’t miss it- also they will arrive by the dozen so worth waiting for a good visual. These also trill, which is another reason it is good to get a visual. The trill is slow, very “sing-song”, and has a downward inflection at the end. If there are a bunch sticking around for the summer, try to watch some sing- soon enough you will be able to pick out this trill from the others.
— Yellow warbler says “sweet sweet sweet, I’m so Sweet!” and can get a bit confusing with Yellow-rumped warbler
— Chestnut-sided warbler says “very very pleased to meet ya!” and can get a bit confusing with Yellow warbler
Black-and-white warbler
Looks like a zebra – always acts like a nuthatch (clings to trunk and branches). This one trills like a rusty wheel. It can easily be distinguished after a bit of birding with some around.
American Redstart
Adult males look like a late 50’s hot-rodded American muscle car: long, low, two tone paint job. Matte/luster black with flame accents. Can’t miss it. The females and young males are buff (chrome, to keep in style I guess) with yellow accents. Look for behavior- if a “female” is getting beaten up while trying to sing a song in the same area, it is actually a first year male failing to establish a territory due to obviously being a youth.
Cheers,
– Jess
How to Query KML point data as CSV using QGIS and R
Here you can see more than 800 points, each describing an observation of an individual bird. This data is in the form of KML, a sort of XML document from Google for spatial data.
I want to know which points have “pair” or “female” in the description text nodes using R. This way, I can quickly make and update a .csv in Excel of only the paired birds (based on color bands).
Even if there was a description string search function in Google Earth Pro (or other organization-centric GIS/waypoint software), this method is more
robust, as I can work immediately with the output as a data frame in R, rather than a list of results.
First, open an instance of QGIS. I am running ~2.8 on OSX. Add a vector layer of your KML.
“Command-A” in the point dialog to select all before import!
Next, under “Vector”, select “Merge vector layers” via Data Management Tools.
Select CSV and elect to save the file instead of use a temporary/scratch file (this is a common error).
Open your csv in Excel for verification! 
The R bit:
# query for paired birds
#EDIT: Libraries
library(data.table)
library(tidyverse)
data <- data.frame(fread("Bird_CSV.csv"))
pair_rows <- contains("pair", vars = data$description)
fem_rows <- contains("fem", vars = data$description)
result <- combine(pair_rows, fem_rows)
result <- data[result,]
write_csv(result, "Paired_Birds.csv")
Tada!
-Jess
Olive-sided Flycatcher (Is Says, “Three Beers!! [please]”
Walking to work in the morning
Work
Song Sparrow
Common Yellowthroat Warbler
I’ll let the photos to the talking: welcome to my world! 🙂 !!!!
-Jess
Solar upgrades!
Incredibly, the hut we are working from actually had another solar panel just laying around. 🙂
This 50w square panel had a junction box with MC4 connectors, the standard for small scale solar installations. As I was unsure how to know when we are running low on electricity reserves, I decided to make some adjustments.
Additional 50w solar panel
(Everything is still solder, hot glue, alligator clips, and zip-ties I’m afraid…)
I traded my NEMA / USA two-prong connection with two MC4 splitters, such that both panels can run in parallel (into a standard USA 110v extension cord that goes into our hut). This way we should make well over one of the two 35ah batteries-worth of electricity a day.
Dual MC4 splitters to extension cord
I also added a cheap 12v battery level indicator. It is not very accurate (as it fluctuates with solar input) but it does give us some insight about how much “juice” we have available. (I also wired and glued the remote-on switch to the back of the input for stability.)
Added battery indicator and button
🙂
-Jess
Male Common Yellowthroat Warbler
The field season has officially started in Northern NH!
Male Common Yellowthroat warbler (COYE): This fellow is defending a small territory in a patch of open thicket. These warblers rely on early succession forest- patches of substrate that haven’t really grown in yet- to build cryptic, ground-level nests. They develop complex systems to divert/confuse predators away from their nests.
Female Black-throated Blue Warbler (BTBW): I was lucky to see this female. She is paired with a male who defends a large mature forest territory. They have quite a few BTBW neighbors, which makes for a lot of skirmishes among the males over land. The females are often silent and move very fast…
Male Mourning Warbler (MOWA): This is a rare bird here. Even more amazing, it is defending a territory in our research site- and trying to chase out a male COYE while doing so. The two species “share” resources, which means thy can’t stand each other. 🙂 Each time the male COYE sings near the MOWA, it gets berated and chased away- and vice versa. It appears the COYE isn’t budging either, probably because it hasn’t had this domestic, neighborly problem before.
-Jess
Female Black-throated Blue Warbler
Male Mourning Warbler
Keeping track of birds is tricky!
Attached is our team’s workflow with Compass 55. From the Kml, we go into Google Earth Pro – ArcGIS Desktop (arcmap). QGIS is sometimes used too.
Cheers,
-Jess
Equipment used:
Inverter/PWM Controller: http://a.co/fdl9YzI
2x 35ah Batteries: http://a.co/5JBIxTC
100w solar panel: http://a.co/5JBIxTC
We need power! While doing bird research in the wilds of northern NH, it became evident we needed electricity to power computers, big cameras, and phones/GPS units.
Below is a table of the system and our expected electricity needs:
| System | Solar 100w | 35ah universal (x2) | ||
| Ah per day: | 33.33333333 | 35 | TOTAL Ah Reserve: | 70 |
| V | 12 | 12 | Parallel wiring: | 12v |
| Wh in: | 400 | 420 | TOTAL Wh Reserve: | 840 |
| W | 100 | |||
| Cost | $105.00 | $64.00 | ||
| ah/$ | 2 | |||
| Sun Hour / Multiplier | 4 | 2 | ||
| Need/Day | Wh | multiplier | consump. in Wh = | 259.36 |
| Computer | 100 | 2.5 | 250 | |
| iPhone | 1.7 | 2 | 3.4 | |
| AAs | 11.2 | 0.3 | 3.36 | |
| Camera | 2.6 | 1 | 2.6 |
*The milk crate system below can charge a 100 watt MacBook Pro around 8-9 times from being completely empty.
**Remember: V*A=W, W/V=A, and Watts over time is Wh.
-Jess
+/- relates to size of standard prongs
Parallel maintains 12v but doubles Ah. (Series would go to 24v at 35ah)
While somewhat of a historical entry:
Below are some of the few photos I took while birding on may 5, 2018- GLOBAL BIG DAY! Be sure to read about it here: https://ebird.org/news/global-big-day-5-may-2018
Myself and my father contributed 64 species, including the below Northern waterthrush, Female Rose-breasted Grosbeak, Northern Parula, and Common Yellowthroat warbler.
-Jess
Spring is (possibly) coming people! 🙂
Scituate, MA – Osprey
Scituate, MA – Killdeer
Plymouth, NH – (FOY photo for me) Hermit Thrush
Scituate, MA – Brant
Plymouth, NH – Savannah Sparrow
Edit 07/26/2020:
Check out the expanded GIS notes page here!
Using ESRI ArcGIS / ArcMap on Macs: 2 methods
I need to run ESRI products on my MacBook Pro. QGIS is always the prefered solution- open source, excellent free plugins, works on mac natively- but in a college / research environment, the only option that supports other people and school machines is ESRI. Despite the annoying bureaucracy and expense of the software, some things are faster (but not better!) in ESRI, like dealing with raster / multiband data.
First, you need a license.
I went about this two ways;
My first solution was to buy an ESRI Press textbook on amazon. A 180 day trial for $50- when taken as a college course, this isn’t to bad. 🙂 The book is slow and recursive, but a 180 days to play with all the plugins and whistles allows for way deeper learning via the internet. 🙂
Do know there is a little-documented limit to the number of license transfers you may perform before getting either lock in or out of your software. I hit this limit, as I was also figuring out my virtual machine situation, which would occasionally need a re-installation.
My current solution is “just buy a student license”. $100 per year is less than any adobe situation- so really not that bad.
Now you need a windows ISO.
https://www.microsoft.com/en-us/software-download/windows10ISO
Follow that link for the window 10, 64 bit ISO. YOU DO NOT NEED TO BUY WINDOWS. It will sometimes complain about not having an authentication, but in the months of using windows via VMs, never have I been prohibited to do… anything. When prompted for a license when configuring your VM, click the button that says "I don’t have a license". Done.
Option one: VirtualBox VM on a thumbdrive
https://www.virtualbox.org/wiki/Downloads – download for the VM software
http://a.co/4FEYMNY, http://a.co/hanHYl1 Suitable USBs. the VM will take up most of a 128gb flash drive- ~70 gb just for windows and all the stuff you’ll want from a PC. Add ESRI software and allocated space for a cache (where your GIS project works!), bigger is better. Format all drives in disk utility as ExFat! this is important, any other file system either won’t fly or could wreak havoc (other FAT based ones may have too small file allocations!
I used two drives, a 128 and a 64- this is great because I can store all my work on the 64, so I can easily plug it into other (school) machines running windows ArcMap and keep going, without causing issues with the massive VM in the 128.
Installation is straightforward, just install EVERYTHING on the usb drive and it will be fine. 🙂
Problems: Stability. Crashes, and python / some other script modules do not work well. This is a problem. ArcAdministrator gets confused about all kinds of things- FWIW, if you are googling to delete the FLEXnet folder to solve authentication file issues, move to option 2 🙂
Speed is down, but actually the ~same speed as our school "super" PCs- (though I happened to know they are essentially glorified "hybrid" VMs too!) .
Option two: OSX Bootcamp
https://support.apple.com/boot-camp
https://support.apple.com/en-us/HT201468
This way, you will hit "option/alt" each time you restart/boot your computer to choose from win/osx. This is easy to install, as it is mac and mac = easy.
Big Caveat: it is much harder to install windows externally (on a usb, etc) from bootcamp. I didn’t succeed in my efforts, but there could be a way…. The thing is, it really wants to run everything like a normal intel based PC, with all installations in the usual place. This is good for the mac performance, but terrible for the tiny SSD hard drives we get as mac users. I have a 256gb SSD. I have an average of < 15 gb wiggle room here, and use every cloud service in the book.
If you need to manage your cloud storage because of a itsy mac SSD, my solution is still ODrive. https://www.odrive.com/
I use Amazon cloud mostly with odrive, but I use a personal/school OneDrives, Dropboxes, Google, etc. with only the occasional hiccup. Also, all of the AWS tools are great and cheap- EC2, S3, Cloud 9, lambda, RDS…. Great way to do your work outside of your mac via the internet.
Result:
ArcMap and GIS stuff is blazing fast on my modest 2015 i5/8gb macbook pro. Comparing a huge, mega ATX+ school computer to my mac on boot camp, I am running large raster filtering operations significantly quicker than other folks doing the same type of work. That is GOOD.
🙂
-Jess
GitHub Link: https://github.com/Jesssullivan/EBPP
…Epic Birding Predicting Project: Predicting good birding near you…
…Currently under development in R.
🙂
-Jess
