Trans Scend Survival

Trans: Latin prefix implying "across" or "Beyond", often used in gender nonconforming situations – Scend: Archaic word describing a strong "surge" or "wave", originating with 15th century english sailors – Survival: 15th century english compound word describing an existence only worth transcending.

Merlin AI Demos!

post updated 11/23/20

Demos, interpreter implementations & data ingress tools for annotating, interpreting, and deploying trained models.

Web:

demos
Visit audio demos on Heroku
demos
Visit Leaflet.annotation demos on Heroku


Watch environment setup here


Watch Leaflet.annotation demo here

Hack upon these demos:

# Clone:
git clone --branch=master --depth=1 https://github.com/Jesssullivan/tmpUI && cd tmpUI

## Follow the prompts: ##
npm run-script develop-web-demos
Demo Description
deploy/demos/spec_record_crop_dl
deploy/demos/spec_record_crop_post
Experiments with record --> crop --> classify --> download; both client-side & server-side classifications methods
deploy/demos/load_audio
deploy/demos/spec_display
Experiment with Macaulay audio sources --> spectrogram
deploy/demos/spec_record_crop_v3
deploy/demos/spec_record_v2
deploy/demos/spec_record_v2
Single page feature experiments
deploy/demos/webgl_init
deploy/demos/deploy/demos/webgl_float_test
deploy/demos/spec_record_v2
Evaluate web client's capability for classification

Notes:

Configure Flask in config.py:

# config.py

# `True` serves demos @ 127.0.0.1:5000 via node proxy (set `False` for production @ 0.0.0.0:80)
devel = True

# rebuild header + demo + footer html renders before serving anything (set `False` for production):
prerender = True
  • / runs webgl_init, which figures out if the browser can or cannot make classifications and routes the client accordingly.
    • classification options:
    • if browser cannot do classification (i.e. safari on mobile, webgl mediump not supported) recording is beamed up to /uploader_standard for processing
    • both POST destinations /uploader_select & /uploader_standard can also be operated from within browser as a multipart form

Leaflet.annotation @ tmpUI:

demos

Visit Leaflet.annotation Audio demo
demos

Visit Leaflet.annotation Photo demo
  • (Jess is still typifying Annotator source, hang tight)

Hack on Annotator:

#### develop-anno-demos:

# packs annotator demos
# generates unique openssl cert & key
# serves annotator demos on node http-server

npm run-script develop-anno-demos
# pack only tool definitions @ `./src/annotator_tool.js:
npm run-script build-anno-tool

# pack only implementations of audio annotator @ `./demos/annotator_audio.ts:
npm run-script build-anno-audio

# pack only implementations of photo annotator @ `./demos/annotator_photo.ts:
npm run-script build-anno-photo
Demo Description
deploy/demos/annotator_audio Leaflet.annotator tool implementations for generating, labeling, exporting mel spectrogams as annotation data
deploy/demos/annotator_photo Leaflet.annotator tool implementations for labeling & exporting photo annotations
deploy/src/annotator_tool epic Annotator_tool entrypoint, contains class handler functions

Swift Native:

# Hack on Swift stuff:
npm run-script develop-swift-demos
  • focusing on codepaths for:

    • tflite interpreter
    • generating mel spectrograms
  • make sure info.plist has permissions for microphone access

  • The entrypoint for Swift tests is ./swift/swift-pkgs-tmpui/swift-pkgs-tmpui/swift_pkgs_tmpuiApp.swift

  • Toggle various interpreter experiments from entrypoint

Other Linker Libraries:

Project:

$(inherited)
-force_load Pods/TensorFlowLiteSelectTfOps/Frameworks/TensorFlowLiteSelectTfOps.framework/TensorFlowLiteSelectTfOps
-force_load Pods/TensorFlowLiteC/Frameworks/TensorFlowLiteC.framework/TensorFlowLiteC
-force_load Pods/TensorFlowLiteC/Frameworks/TensorFlowLiteCCoreML.framework/TensorFlowLiteCCoreML
-force_load Pods/TensorFlowLiteC/Frameworks/TensorFlowLiteCMetal.framework/TensorFlowLiteCMetal
-ObjC
-l"c++"

Target:

-force_load Pods/TensorFlowLiteSelectTfOps/Frameworks/TensorFlowLiteSelectTfOps.framework/TensorFlowLiteSelectTfOps
-force_load Pods/TensorFlowLiteC/Frameworks/TensorFlowLiteC.framework/TensorFlowLiteC
-force_load Pods/TensorFlowLiteC/Frameworks/TensorFlowLiteCCoreML.framework/TensorFlowLiteCCoreML
-force_load Pods/TensorFlowLiteC/Frameworks/TensorFlowLiteCMetal.framework/TensorFlowLiteCMetal
-ObjC
-l"c++"
# niftily switch between xcode versions:
sudo xcode-select --switch ~/Downloads/Xcode-beta.app

Interpreter Operations:

Hack on fft functions:
[./etc/tone.py:](https://github.com/Jesssullivan/tmpUI/blob/master/etc/tone.py)

### copy from here:
cp etc/tone.py .

### generate some .wav files for testing fft things:
python3 tone.py

### ...you can also specify duration in seconds & frequency in Hz like so:
python3 tone.py 5 440

### ...or just duration:
python3 tone.py 2

some fft-related links

Scripts:

# See ./package.json & ./scripts/ for additional scripts

main scripts links:

local ssl:

# Generates local ssl certs for testing w/ node http-server:
npm run-script sslgen

# you can also provide a $DOMAIN argument like so:
npm run-script sslgen hiyori
# ...returns key `hiyori_key.pem` & cert `hiyori.pem`

# ...or:
sudo chmod +x scripts/sslgen.sh && ./scripts/sslgen.sh
# osx is a bit more finicky

tone generator:

### available from here:
cp etc/tone.py .

### generate some .wav files for testing fft things:
python3 tone.py

### ...you can also specify duration in seconds & frequency in Hz like so:
python3 tone.py 5 440

### ...or just duration:
python3 tone.py 2

removing stuff:

# clean up with:
npm run-script clean all
# ...and follow the instruction prompt

# ...demo bundles:
npm run-script clean-web-bundles

# ...demo renders:
npm run-script clean-web-renders

etc, etc etc

xD

-Jess

Client-side, asynchronous HTTP methods- TypeScript

Despite the ubiquitousness of needing to make a POST request from a browser (or, perhaps for this very reason) there seems to be just as many ways, methods, libraries, and standards of implementing http functions in JavaScript as there are people doing said implementing. Between the adoption of the fetch api in browsers and the prevalence and power of Promises in JS, asynchronous http needn't be a hassle!

/*
...happily processing some data in a browser, when suddenly...
....panik!
you need to complete a portion of this processing elsewhere on some server...:
*/

Continue reading

Obliterate non-removable MDM profiles enforced by Apple’s Device Enrollment Program

Or, when life gives you apples, use Linux

Seemingly harder to remove with every eye-glazing gist and thread... A mac plagued with an is_mdm_removable=false Mobile Device Management profile: the worst! 🙂

First, boot into recovery mode by rebooting while holding down the Command & R keys.

At this stage, you'll need to connect to the internet briefly to download the recovery OS. This provides a few tools including like disk utility, support, an osx reinstaller- at the top menu, you'll find an option to access a terminal.

Once in there, you'll want to:

Disable SIP:

csrutil disable

Then reboot:

reboot now

While holding down Command + Option + P + R to start afresh with cleared NVRAM.

Reboot once again while holding down the Command & R keys to return to the recovery OS. Reinstall whatever version of OSX it offers- instead of trying to deal with the slippery, network connected DEP plists & binaries contained within the various LaunchAgents and LaunchDaemons found in the /System/Library directories directly, we'll let Apple finish with the ConfigurationProfiles first, then sneak in and remove them.

While this stuff is cooking, get yourself a usb stick and a penguin, such as Budgie:

wget -nd http://cdimage.ubuntu.com/ubuntu-budgie/releases/20.04.1/release/ubuntu-budgie-20.04.1-desktop-amd64.iso
umount /dev/sdc 2>/dev/null || true
sudo dd if=ubuntu-budgie-20.04.1-desktop-amd64.iso of=/dev/sdc bs=1048576 && sync

Boot up again, this time holding the Option key for the bootloader menu. Once in the live usb system, make sure you can read Apples HFS filesystem:

sudo apt-get install hfsprogs

For me at least, I needed to run a quick fsck to fix up the headers before I could mount the osx filesystem living at /dev/sda2 (sda1 is the efi partition):

sudo fsck.hfsplus /dev/sda2

Now, lets go in there and remove those ConfigurationProfiles:

mkdir badapple
sudo mount -o force /dev/sda2 badapple
cd badapple
sudo rm -rf private/var/db/ConfigurationProfiles/*

🙂

KVM does Fruit: Xcode from QEMU

The digression starts over here:

https://github.com/kholia/OSX-KVM

Hmmmm.....

?

...In the mean time...
./OpenCore-Boot.sh:

#!/usr/bin/env bash

# setup tap0 if haven't already for $session:
sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up promisc on 
sudo ip link set dev virbr0 up 
sudo ip link set dev tap0 master virbr0

REPO_PATH="./"
OVMF_DIR="."

args=(
  -enable-kvm -m 24000 -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check
  -machine q35
  -smp 4,cores=2,sockets=1
  -device usb-ehci,id=ehci
  -device usb-kbd,bus=ehci.0
  -device usb-mouse,bus=ehci.0
  -device nec-usb-xhci,id=xhci
  -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
  -drive if=pflash,format=raw,readonly,file="$REPO_PATH/$OVMF_DIR/OVMF_CODE.fd"
  -drive if=pflash,format=raw,file="$REPO_PATH/$OVMF_DIR/OVMF_VARS-1024x768.fd"
  -smbios type=2
  -device ich9-intel-hda -device hda-duplex
  -device ich9-ahci,id=sata
  -drive id=OpenCoreBoot,if=none,snapshot=on,format=qcow2,file="$REPO_PATH/OpenCore-Catalina/OpenCore-nopicker.qcow2"
  -device ide-hd,bus=sata.2,drive=OpenCoreBoot
  -device ide-hd,bus=sata.3,drive=InstallMedia
  -drive id=InstallMedia,if=none,file="$REPO_PATH/BaseSystem.img",format=raw
  -drive id=MacHDD,if=none,file="$REPO_PATH/mac_hdd_ng.img",format=qcow2
  -device ide-hd,bus=sata.4,drive=MacHDD
  -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device vmxnet3,netdev=net0,id=net0,mac=52:54:00:c9:18:27
  -vga vmware
)

qemu-system-x86_64 "${args[@]}"

…Ever tried to Chrome Remote ➡️ Ubuntu Budgie?

Check out this project on my Github over here 🙂

Fully automated patching for Chrome Remote Desktop on Ubuntu Budgie.

Chrome remote desktop is fantastic, but often clashes with Xorg nuances from a variety of desktop environments in Ubuntu. This chrome-remote-desktop script extends and replaces the version automatically installed by Google in /opt/google/chrome-remote-desktop/chrome-remote-desktop. This stuff is only relevant for accessing your Ubuntu machine from elsewhere (e.g. the "server", the client machine should not be installing anything, all it needs is a web browser).

Set up the server:

Before patching anything or pursuing other forms of delightful tomfoolery, follow the installation instructions provided by Google. Set up everything normally- install Google's .deb download with dpkg, set up a PIN, etc.
The trouble comes when you are trying to remote in- some problems you may encounter include:

  • none of the X sessions work, each immediately closing the connection to the client
  • the remote desktop environment crashes or becomes mangled
  • odd scaling issues or flaky resolution changes

Patch it up:

# get this script:
# wget https://raw.githubusercontent.com/Jesssullivan/chrome-remote-desktop-budgie/master/chrome-remote-desktop

# or:
git clone https://github.com/Jesssullivan/chrome-remote-desktop-budgie/ 
cd chrome-remote-desktop-budgie 

# behold:
python3 chrome-remote-desktop

# ...perhaps, if you are keen (optional):
sudo chmod u+x addsystemd.sh
sudo ./addsystemd.sh

What does this do?

We are primarily just enforcing the use of existing instances of X and correct display values as reported by your system.

  • This version keeps a persistent version itself in /usr/local/bin/ in addition updating the one executed by Chrome in /opt/google/chrome-remote-desktop/.
  • A mirror of this script is also maintained at /usr/local/bin/chrome-remote-desktop.github, and will let the user know if there are updates.
  • The version distributed by google is retained in /opt/ too as chrome-remote-desktop.verbatim.
  • Each of these versions are compared by md5 hash- this way our patched version of chrome-remote-desktop will always make sure it is where it should be, even after Google pushes updates and overwrites everything in /opt/.

This, That, etc

....Updated 07/19/2020

Bits & bobs, this & that of late:

...In effort to thwart the recent heat and humidity here in the White Mountains (or, perhaps just to follow the philosophy of circuitous overcomplication... 🙂 ) here are some sketches of quick-release exhaust fittings of mine for a large, wheeled AC & dehumidifier unit (these have been installed throughout my home via window panels).

...Sketching out a severely overcomplicated "computer shelf", rapid-fab style:
(plasma cut / 3d printed four-post server rack == RepRapRack?? xD) 🙂

...Also, Ryan @ V1Engineering recently released his new MPCNC Primo here, should anyone be keen. Long Live the MPCNC! 🙂

...Oodles of fun everyday over in the clipi project- check it out!

...A really short morning metal:

...All Morning Metal tracks here, a couple videos here

xD

clipi CLI!

Find this project on my github here!

...post updated 07/19/2020

An efficient toolset for Pi devices

Emulate, organize, burn, manage a variety of distributions for Raspberry Pi.


Choose your own adventure....

Emulate:
clipi virtualizes many common sbc operating systems with QEMU, and can play with both 32 bit and 64 bit operating systems.

  • Select from any of the included distributions (or add your own to /sources.toml!) and clipi will handle the rest.

Organize:
clipi builds and maintains organized directories for each OS as well a persistent & convenient .qcow2 QEMU disk image.

  • Too many huge source .img files and archives? clipi cleans up after itself under the Utilities... menu.
  • additional organizational & gcc compilation methods are available in /kernel.py

Write:
clipi burns emulations to external disks! Just insert a sd card or disk and follow the friendly prompts. All files, /home, guest directories are written out.

  • Need to pre-configure (or double check) wifi? Add your ssid and password to /wpa_supplicant.conf and copy the file to /boot in the freshly burned disk.
  • Need pre-enabled ssh? copy /ssh to /boot too.
  • clipi provides options for writing from an emulation's .qcow2 file via qemu...
  • ...as well as from the source's raw image file with the verbatim argument

Manage:
clipi can find the addresses of all the Raspberry Pi devices on your local network.

  • Need to do this a lot? clipi can install itself as a Bash alias (option under the Utilities... menu, fire it up whenever you want.

Shortcuts:

Shortcuts & configuration arguments can be passed to clipi as a .toml (or yaml) file.

  • Shortcut files access clipi's tools in a similar fashion to the interactive menu:
# <shortcut>.toml
# you can access the same tools and functions visible in the interactive menu like so:
'Burn a bootable disk image' = true  
# same as selecting in the interactive cli
'image' = 'octoprint'
'target_disk' = 'sdc'  
  • clipi exposes many features only accessible via configuration file arguments, such as distribution options and emulation settings.
# <shortcut>.toml

# important qemu arguments can be provided via a shortcut file like so:
'kernel' = "bin/ddebian/vmlinuz-4.19.0-9-arm64"
'initrd' = "bin/ddebian/initrd.img-4.19.0-9-arm64"

# qemu arguments like these use familiar qemu lexicon:
'M' = "virt" 
'm' = "2048"

# default values are be edited the same way:
'cpu' = "cortex-a53"
'qcow_size' = "+8G"
'append' = '"rw root=/dev/vda2 console=ttyAMA0 rootwait fsck.repair=yes memtest=1"'

# extra arguments can be passed too:
'**args' = """
-device virtio-blk-device,drive=hd-root \
-no-reboot -monitor stdio
"""

# additional network arguments can be passed like so:
# (clipi may automatically modify network arguments depending on bridge / SLiRP settings)
'network' = """
-netdev bridge,br=br0,id=net0 \
-device virtio-net-pci,netdev=net0
"""
  • Supply a shortcut file like so:
    python3 clipi.py etc/find_pi.toml

  • take a look in /etc for some shortcut examples and default values

TODOs & WIPs:

bridge networking things:

  • working on guest --> guest, bridge --> host, host only mode networking options.
    as of 7/17/20 only SLiRP user mode networking works,
    see branch broken_bridge-networking
    to see what is currently cooking here

kernel stuff:

  • automate ramdisk & kernel extraction-
    most functions to do so are all ready to go in /kernel.py

  • other random kernel todos-

    • working on better options for building via qemu-debootstrap from chroot instead of debian netboot or native gcc
    • add git specific methods to sources.py for mainline Pi linux kernel
      • verify absolute binutils version
      • need to get cracking on documentation for all this stuff

gcp-io stuff:

  • formalize ddns.py & dockerfile

  • make sure all ports (22, 80, 8765, etc) can up/down as reverse proxy

# clone:
git clone https://github.com/Jesssullivan/clipi
cd clipi

# preheat:
pip3 install -r requirements.txt
# (or pip install -r requirements.txt)

# begin cooking some Pi:
python3 clipi.py

Parse fdisk -l in Python

fdisk -l has got to be one of the more common disk-related commands one might use while fussing about with raw disk images. The fdisk utility is ubiquitous across linux distributions (also brew install gptfdisk and brew cask install gdisk, supposedly). The -l argument provides a quick look raw sector & file system info. Figuring out the Start, End, Sectors, Size, Id, Format of a disk image's contents without needing to mount it and start lurking around is handy, just the sort of thing one might want to do with Python. Lets write a function to get these attributes into a dictionary- here's mine:

import subprocess
import re

def fdisk(image):

    #  `image`, a .img disk image:
    cmd = str('fdisk -l ' + image)

    # read fdisk output- everything `cmd` would otherwise print to your console on stdout
    # is instead piped into `proc`.
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)

    # the raw stuff from stdout is not parseable as is, so we read into a string:
    result = proc.stdout.read().__str__()

    # figure out what type we should iterate with when looking via file / part contained within image.  I have no idea if anything besides .img will work- YMMV, but YOLO xD
    if '.iso' in result:
        iter = '.iso'
    if '.qcow2' in result:
        iter = '.qcow2'
    else:  
        iter = '.img'

    # chop up fdisk results by file / partition-
    # the resulting `parts` are equivalent to fdisk "rows" in the shell
    parts = re.findall(r'' + iter + r'\d', result)

    # dictionary `disk` contains each "row" from `parts`:
    disk = {}
    for p in parts:
        # sub dictionary 'part' contains the handy fdisk output values:
        part = {}
        # get just the number words with regex sauce:
        line = result.split(p)[1]
        words = re.split(r'\s+', line)
        # place each word into 'part':
        part['Start'] = words[1]
        part['End'] = words[2]
        part['Sectors'] = words[3]
        part['Size'] = words[4]
        part['Id'] = words[5]
        part['Format'] = words[6].split('\\n')[0]
        # stick this part into 'disk', move onto next disk part:
        disk[p] = part
    return disk

Dover’s Enclosure

A stylish demo enclosure for the Xilinx / Digilent Genesys 2 with a display panel.

Check out what Dover Microsystems is up to here:
https://www.dovermicrosystems.com/

Prototyping & production @ the D&M Makerspace- see what else we're up to:
https://makerspace.plymouthcreate.net/

Electronics:

FPGA- Digilent Xilinx Genesys 2 FPGA Reference

The display and HDMI driver board from pimoroni-
The sketch for panel dimensions are shared over here too

BOM for version 6:

You can find the V6 interactive Fusion 360 model over here

...additional V6 svg, stl layouts on tinkercad

Materials:

Size Type
12"x12" 1/4" (6.35mm) clear acrylic sheet
12"x12" 3mm clear acrylic sheet
12"x12" 3mm colored acrylic sheet
~45 grams printer plastic (filament or resin)

Hardware:

Qty Size
3 m3x8
3 m3x18
1 m3x20
7 m3 nut
2 m2x14
2 m2x10
4 m2 nut

What is this thing? Some words from founding scientist @gregsgit:

*"We use the FPGA to prototype / emulate a "Soft Core" CPU with and without Dover's IP (logic) called CoreGuard.
An FPGA can simulate (sometimes called "emulate") logical circuits, and is reprogrammable. So you can design circuitry that eventually will be fabricated in silicon, but you can work out bugs and try different designs using the FPGA "fabric".*

*For demos, we synthesize to the Xilinx FPGA: a design for a RISC-V CPU, a simple UART (serial interface), an interface to the on-board DDR memory and flash memory, and a simple video output. We put some software in the on-board flash, then boot a working RISC-V system. We'll show how the software can be attacked, using I/O over the serial port to mimic what would typically take place over a network connection. Next, we show the same SoC (CPU + UART + memory) with CoreGuard logic added in. We run the same software and then show that the same attack is blocked by CoreGuard. We also use the FPGA to emulate the Arm CPU that we are interfacing with for our NXP customer."*

The eBird API & regionCode

get this script and other GIS bits here on github

The Ebird dataset is awesome. While directly handling data as a massive delimited file- as distributed by the eBird people- is cumbersome at best, the ebird api offers a fairly straightforward and efficient alternative for a few choice bits and batches of data.

  • The eBird AWK tool for filtering the actual delimited data can be found over here:

    install.packages("auk")

It is worth noting R + auk (or frankly any R centered filtering method) will quickly become limited by the single-threaded approach of R, and how you're managing memory as you iterate. Working and querying the data from a proper database quickly becomes necessary.

Most conveniently, the eBird API already exists- snag an key over here.

...The API package for R is over here:
install.packages("rebird")

...There is also a neat Python wrapper over here:
pip3 install ebird-api

Region Codes:

I'm not sure why, but some methods use normal latitude / longitude in decimal degrees while some others use "regionCode", which seems to be some kind of eBird special. Only ever seen this format in ebird data.

For example, recent observations uses regionCode:

# GET Recent observations in a region:
# https://api.ebird.org/v2/data/obs/{{regionCode}}/recent

...But nearby recent observations uses latitude / longitude:

# GET Recent nearby observations:
# https://api.ebird.org/v2/data/obs/geo/recent?lat={{lat}}&lng={{lng}}

Regardless, lets just write a function to convert decimal degrees to this regionCode thing. Here's mine:

#!/usr/bin/env python3
"""
# provide latitude & longitude, return eBird "regionCode"
Written by Jess Sullivan
@ https://transscendsurvival.org/
available at: 
https://raw.githubusercontent.com/Jesssullivan/GIS_Shortcuts/master/regioncodes.py
"""
import requests
import json

def get_regioncode(lat, lon):

    # this municipal api is a publicly available, no keys needed afaict
    census_url = str('https://geo.fcc.gov/api/census/area?lat=' +
                     str(lat) +
                     '&lon=' +
                     str(lon) +
                     '&format=json')

    # send out a GET request:
    payload = {}
    get = requests.request("GET", census_url, data=payload)

    # parse the response, all api values are contained in list 'results':
    response = json.loads(get.content)['results'][0]

    # use the last three digits from the in-state fips code as the "subnational 2" identifier:
    fips = response['county_fips']

    # assemble and return the "subnational type 2" code:
    regioncode = 'US-' + response['state_code'] + '-' + fips[2] + fips[3] + fips[4]
    print('formed region code: ' + regioncode)
    return regioncode

Prius, Printers

Add EV only mode button for 2009 Prius-

Pinouts and wiring reference here:
http://www.calcars.org/prius-evbutton-install.pdf

Big shiny EV mode button in the Prius!

Fusion 360 files here: https://a360.co/2zOACJJ

Files uploaded to thingiverse here: https://www.thingiverse.com/thing:4422091

xposted to prius chat too:
https://priuschat.com/threads/3d-printed-ev-mode-button-xd.216774/

PLA & Carbon Polycarbonate button housings:


While we're at it....

See more notes on D&M 3d Printer stuff on github here:
https://github.com/Jesssullivan/Funmat-HT-Notes
...and here:
https://github.com/Jesssullivan/AeroTaz5_hotfix

While at a safe distance…

...Playing with Bandlab's Sonar reboot --> morning metal ....Frankly the whole suite (yes, Melodyne, the whole nine yards) is way better than when it was with the late Cakewalk, and its all free now. PSA!

...Unexpected success with
Nylon 680 FDA {3mm @ .8} for some rather delicate parts:


...Yet another improved pi monitoring sketch, currently in production w/ polycarbonate & 1/4"... ...or to quote Mad-eye Moody, "CONSTANT VIGILANCE!" 🙂

xD

Install Adobe Applications on AWS WorkSpaces

By default, the browser based authentication used by Adobe’s Creative Cloud installers will fail on AWS WorkSpace instances. Neither the installer nor Windows provide much in the way of useful error messages- here is how to do it!

Open Server Manager. Under “Local Server”, open the “Internet Explorer Enhanced Security Configuration”- *(mercy!)* - and turn it off.

Good Lord

##### Tada! The sign on handoff from the installer→Browser→ back to installer will now work fine. xD

Convert .heic –> .png

on github here, or just get this script:

wget https://raw.githubusercontent.com/Jesssullivan/misc/master/etc/heic_png.sh

Well, following the current course of Apple’s corporate brilliance, iOS now defaults to .heic compression for photos.

Hmmm.

Without further delay, let's convert these to png, here from the sanctuary of Bash in ♡Ubuntu Budgie♡.

Libheif is well documented here on Github BTW

#!/bin/bash
# recursively convert .heic to png
# by Jess Sullivan
#
# permiss:
# sudo chmod u+x heic_png.sh
#
# installs heif-convert via ppa:
# sudo ./heic_png.sh
#
# run as $USER:
# ./heic_png.sh

command -v heif-convert >/dev/null || {

  echo >&2 -e "heif-convert not intalled! \nattempting to add ppa....";

  if [[ $EUID -ne 0 ]]; then
     echo "sudo is required to install, aborting."
     exit 1
  fi

  add-apt-repository ppa:strukturag/libheif
  apt-get install libheif-examples -y
  apt-get update -y

  exit 0

  }

# default behavior:

for fi in *.heic; do

  echo "converting file: $fi"

  heif-convert $fi $fi.png

 # FWIW, convert to .jpg is faster if png is not required 
 # heif-convert $fi $fi.jpg

  done

D&M Shields – Fusion 360

As of 4/4/20, we are busy 3d printing our rigid shield design, efficiently hacked into its current form by Bret here at D&M. click here to visit or download the Fusion files!

The flat, snap-fit nature of this design can easily be lasercut as well- the varied depths of the printed model are just an effort to minimize excess plastic and print time.

More to come on the laser side of things- in addition to the massive time savings- like <20 seconds vs. >3 hours per shield- we can use far cheaper and varied materials with the addition of our sterilizable and durable UV resins and coatings. Similarly, lasercut stock + resin offers the possibility quick adaptation and derivative design, such as [flexible](https://a360.co/2UFKRHM) UV cured forms.

ppe & whatnot

Yep, we too are busy cooking up protective medical devices.......

¯_(ツ)_/¯

& whatnot:

Prototyping bits, bobs for an ada motorsports startup-

ADA auto prototyping

Fast Pi camera stand sketch:

Quick pass at a low friction filament spool holder for some very fragile materials:

Some GDAL shell macros from R instead of rgdal

also here on github

it's not R sacrilege if nobody knows

Even the little stuff benefits from some organizational scripting, even if it’s just to catalog one’s actions. Here are some examples for common tasks.

Get all the source data into a R-friendly format like csv. ogr2ogr has a nifty option -lco GEOMETRY=AS_WKT (Well-Known-Text) to keep track of spatial data throughout abstractions- we can add the WKT as a cell until it is time to write the data out again.

# define a shapefile conversion to csv from system's shell:
sys_SHP2CSV <- function(shp) {
  csvfile <- paste0(shp, '.csv')
  shpfile <-paste0(shp, '.shp')
  if (!file.exists(csvfile)) {
    # use -lco GEOMETRY to maintain location
    # for reference, shp --> geojson would look like:
    # system('ogr2ogr -f geojson output.geojson input.shp')
    # keeps geometry as WKT:
    cmd <- paste('ogr2ogr -f CSV', csvfile, shpfile, '-lco GEOMETRY=AS_WKT')
    system(cmd)  # executes command
  } else {
    print(paste('output file already exists, please delete', csvfile, 'before converting again'))
  }
  return(csvfile)
}

Read the new csv into R:

# for file 'foo.shp':
foo_raw <- read.csv(sys_SHP2CSV(shp='foo'), sep = ',')

One might do any number of things now, some here lets snag some columns and rename them:

# rename the subset of data "foo" we want in a data.frame:
foo <- data.frame(foo_raw[1:5])
colnames(foo) <- c('bar', 'eggs', 'ham', 'hello', 'world')

We could do some more careful parsing too, here a semicolon in cell strings can be converted to a comma:

# replace ` ; ` to ` , ` in col "bar":
foo$bar <- gsub(pattern=";", replacement=",", foo$bar)

Do whatever you do for an output directory:

# make a output file directory if you're into that
# my preference is to only keep one set of output files per run
# here, we'd reset the directory before adding any new output files
redir <- function(outdir) {
  if (dir.exists(outdir)) {
    system(paste('rm -rf', outdir))
  }
  dir.create(outdir)
}

Of course, once your data is in R there are countless "R things" one could do...

# iterate to fill empty cells with preceding values
for (i in 1:length(foo[,1])) {
  if (nchar(foo$bar[i]) < 1) {
    foo$bar[i] <- foo$bar[i-1]
  }
  # fill incomplete rows with NA values:
  if (nchar(foo$bar[i]) < 1) {
    foo[i,] <- NA  
  }
}

# remove NA rows if there is nothing better to do:
newfoo <- na.omit(foo)

Even though this is totally adding a level of complexity to what could be a single ogr2ogr command, I've decided it is still worth it- I'd definitely rather keep track of everything I do over forget what I did.... xD

# make some methods to write out various kinds of files via gdal:
to_GEO <- function(target) {
  print(paste('converting', target, 'to geojson .... '))
  system(paste('ogr2ogr -f', " geojson ",  paste0(target, '.geojson'), paste0(target, '.csv')))
}

to_SHP <- function(target) {
  print(paste('converting ', target, ' to ESRI Shapefile .... '))
  system(paste('ogr2ogr -f', " 'ESRI Shapefile' ",  paste0(target, '.shp'), paste0(target, '.csv')))
}

# name files:
foo_name <- 'output_foo'

# for table data 'foo', first:
write.csv(foo, paste0(foo_name, '.csv'))

# convert with the above csv:
to_SHP(foo_name)

Cheers!
-Jess

When it must be Windows….

Edit 07/26/2020:
Check out the expanded GIS notes page here!

Regarding Windows-specific software, such as ArcMap:

Remote Desktop:
The greatest solution I've settled on for ArcMap use continues to be Chrome Remote Desktop, coupled with an IT Surplus desktop purchased for ~$50. Once Chrome is good to go on the remote Windows computer, one can operate everything from a web browser from anywhere else (even reboot and share files to and from the remote computer). While adding an additional, dedicated computer like this may not be possible for many students, it is certainly the simplest and most dependable solution.

VirtualBox, Bootcamp, etc:
Oracle's VirtualBox is a longstanding (and free!) virtualization software. A Windows virtual machine is vastly preferable over Bootcamp or further partition tomfoolery.
One can start / stop the VM only when its needed, store it on a usb stick, avoid insane pmbr issues, etc.

  • Bootcamp will consume at least 40gb of space at all times before even attempting to function, whereas even a fully configured Windows VirtualBox VDI will only consume ~22gb, and can be moved elsewhere if not in use.
  • There are better (not free) virtualization tools such as Parallels, though any way you slice it a dedicated machine will almost always be a better solution.

Setup & Configure VirtualBox:

There are numerous sites with VirtualBox guides, so I will not go into detail here.

Extra bits on setup-

  • Guest Additions are not necessary, despite what some folks may suggest.

  • Dynamically Allocated VDI is the way to go as a virtual disk. There is no reason not to set the allocated disk size to the biggest value allowed, as it will never consume any more space than the virtual machine actually needs.

  • Best to click through all the other machine settings just to see what is available, it is easy enough to make changes over time.

  • There are many more levels of convoluted not worth stooping to, ranging from ArcMap via AWS EC2 or openstack to KVM/QEMU to WINE. Take it from me

xD

Ubuntu on Captive Portal WiFi

wget https://raw.githubusercontent.com/Jesssullivan/misc/master/sel/portal.py

Some distros (Ubuntu and its derivatives on my Macbook for instance) struggle to find a route to the captive portal on public networks (read: Starbucks). Assuming you want to connect the way they intend (via the gateway through the captive portal) because you are a rule abiding patron, all you need to do is visit the address of the gateway. There is no need to fiddle with your network drivers, disable SSL stuff or engage in plebian network tomfoolery.

"""
find and open wifi captive portal (such as Starbucks)
written by Jess Sullivan
"""
from netifaces import gateways
from sys import argv
from time import sleep
import subprocess

help_str = str("\n " +
               'usage: \n ' +
               ' -h : print this message again \n' +
               ' -i : `pip3 install netifaces`  \n' +
               'You may specify a browser argument to complete the portal, such as \n' +
               'google-chrome')

def argtype():
    try:
        if len(argv) > 1:
            use = True
        elif len(argv) == 1:
            use = False
            print(help_str)
        else:
            print('command takes 0 or 1 args, use -h for help')
            raise SystemExit
    except:
        print('arg error... \n command takes 0 or 1 args, use -h for help')
        raise SystemExit
    return use

def main():

    all_gates = gateways()
    target = all_gates['default'][2][0]

    if argtype():

        if argv[1] == '-h':
            print(help_str)
            quit()

        if argv[1] == '-i':
            try:
                subprocess.Popen('pip3 install netifaces',
                                 shell=True,
                                 executable='/bin/bash')
                sleep(1)
                quit()
            except:
                print('err running pip3 install netifaces')
                quit()

        else:

            print(str('opening portal in ' + argv[1]))
            subprocess.Popen(str(argv[1] + ' ' + str(target)),
                             shell=True,
                             executable='/bin/bash')
            sleep(1)
            quit()

    else:

        print(str('\n please visit address ' + target +
                  ' in a browser to complete portal setup \n'))
        quit()

main()

JDK Management in R

Quickly & forcefully manage extra JDKs in base R
Simplify rJava woes

# get this script:
wget https://raw.githubusercontent.com/Jesssullivan/rJDKmanager/master/JDKmanager.R

rJava is depended upon by lots of libraries- XLConnect, OpenStreetMap, many db connectors and is often needed while scripting with GDAL.

library(XLConnect)   # YMMV

Errors while importing a library with depending on a JDK are many, but can (usually) be resolved by reconfiguring the version listed somewhere in the error.

On mac OSX (on Mojave at least), check what you have installed here- (as admin, this is a system path) :

sudo ls  "/Library/Java/JavaVirtualMachines/ 

I seem to usually have at least half a dozen or more versions in there, between Oracle and openJDK. Being Java, these are basically sandboxed as JVMs and are will not get in each others way.

However...

Unlike JDK configuration for just about everything else, aliasing or exporting a specific release to $PATH will not cut it in R. The shell command to reconfigure for R-

sudo R CMD javareconf

...seems to always choose the wrong JDK. Renaming, hiding, otherwise trying to explain to R the one I want (lib XLConnect currently wants none other than Oracle 11.0.1) is futile.
The end-all solution for me is usually to temporarily move other JDKs elsewhere.
This is not difficult to do now and again, but keeping a CLI totally in R for moving / replacing JDKs makes for organized scripting.

 JDKmanager help: 
 (args are not case sensitive) 
 (usage: `sudo rscript JDKmanager.R help`) 

 list    :: prints contents of default JDK path and removed JDK path 
 reset   :: move all JDKs in removed JDK path back to default JDK path 
 config ::  configure rJava.  equivalent to `R CMD javareconf` in shell 

 specific JDK, such as 11.0.1, 1.8,openjdk-12.0.2, etc: 
    searches through both default and removed pathes for the specific JDK.  
    if found in the default path, any other JDKs will be moved to the `removed JDKs` directory. 
    the specified JDK will be configured for rJava.

QEMU for Raspian ARM!

Updated 11/6/19

Visit the repo on Github here-
https://github.com/Jesssullivan/QEMU-Raspian

tested on Mac OSX 10.14.6

Emulates a variety of Raspian releases on proper ARM hardware with QEMU.

Prerequisites:

QEMU and wget (OSX homebrew)

brew install qemu wget

Get the Python3 CLI in this repo:

wget https://raw.githubusercontent.com/Jesssullivan/USBoN/master/QEMU_Raspian.py

Usage:

After the first launch, it will launch from the persistent .qcow2 image.

With no arguments & in a new folder, Raspian "stretch-lite" (no desktop environment) will be:

  • downloaded as a zip archive with a release kernel
  • unarchived --> to img
  • converted to a Qcow2 with 8gb allocated as disk
  • launched from Qcow2 as administrator
sudo python3 QEMU_Raspian.py 

Optional Arguments:

  • -h prints CLI usage help
  • -rm removes ALL files added in dir with QEMU_Raspian.py
  • stretch uses standard graphical stretch release with GUI
  • stretchlite for stretchlite release [default!]
  • buster for standard graphical buster release [YMMV]
  • busterlite for busterlite release [YMMV]
# examples:
sudo python3 QEMU_Raspian.py busterlite
python3 QEMU_Raspian -h  # print help

Burn as .img:

qemu-img convert -f qcow2 -O raw file.qcow2 file.img

Four Pi Emulations ala QEMU

Small Updates to GDAL Notes

GIS Shortcuts


GDAL on OSX, Ubuntu & Windows WSL

Bash is great: It is a shell scripting language that is baked into every new Apple computer and many, many Linux distributions too. It is fun to learn and easy to find your “insert repetitive task here” to Bash translation on Stack Exchange, just a search away. As you uses the command prompt more and more for increasingly cool GIS tasks- be it for Python QGIS OR ESRI, R language for data cleaning and analysis, or just because you noticed you can get “mad amounts of work done” at increasingly faster rates- your vocabulary in the UNIX shell and bash will naturally grow.

I wanted to make a GIS post for Mac OS because it is both under-represented (for great reasons) in GIS and arguably the number 1 choice for any discerning consumers of computer hardware.

Many Linux OS options are faster for much of this “UNIX for GIS”, as quite a few of the things we need are already included with many Debian / Ubuntu distros, and come forms that have been stable for a long, long time.

If you are looking to setup a system primarily for GIS / data science (disregarding ESRI of course), See my initial notes on the Ubuntu variant Pop_OS by System76. If you like the ChromeOS vibe for multitasking and simplicity and the familiarity of Mac OS, it is a keeper (and also a sleeper, seeing how many folks are still on Windows for GIS…….).

Open Terminal
(The less you have in your way on your screen, the faster you can go!) xD

Note: in my opinion, homebrew and macPorts are good ideas- try them! If you don’t have it, get it now:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

(….However, port or brew installing QGIS and GDAL (primarily surrounding the delicate links between QGIS / GDAL / Python 2 & 3 / OSX local paths) can cause baffling issues. If possible, don’t do that. Use QGIS installers from the official site and build from source!)

if you need to resolve issues with your GDAL packages via removal:
on MacPorts, try similar:

sudo port uninstall qgis py37-gdal

# on homebrew, list then remove (follow its instructions):

brew list
brew uninstall gdal geos gdal2  

!!! NOTE: I am investigating more reliable built-from-source solutions for gdal on mac.

Really!

There are numerous issues with brew-installed gdal. Those I have run into include:

  • linking issues with the crucial directory “gdal-data” (libraries)
  • linking issues Python bindings and python 2 vs. 3 getting confused
  • internal raster library conflicts against the gdal requirements
  • Proj.4 inconsistencies (see source notes below)
  • OSX Framework conflicts with source / brew / port (http://www.kyngchaos.com/software/frameworks/)
  • Linking conflicts with old, qgis default / LTR libraries against new ones
  • Major KML discrepancies: expat standard vs libkml.
    brew install gdal
    #
    # brew install qgis can work well too.  At least you can unbrew it!
    #

Next, assuming your GDAL is not broken (on Mac OS this is rare and considered a miracle):

# double check CLI is working:
gdalinfo --version
# “GDAL 2.4.0, released 2018/12/14”
gdal_merge.py
# list of args

Using Ubuntu GDAL on Windows w/ WSL

LINK: get the WSL shell from Microsoft

# In the WSL shell:

sudo apt-get install python3.6-dev -y
sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get install libgdal-dev -y
sudo apt-get install gdal-bin -y

# See here for more notes including Python bindings:
# https://mothergeo-py.readthedocs.io/en/latest/development/how-to/gdal-ubuntu-pkg.html

In a new Shell:

# Double check the shell does indeed have GDAL in $PATH:
gdalinfo --version

To begin- try a recent GIS assignment that relies on the ESRI mosaic system and lots of GUI and clicking but use GDAL instead.

Data source: ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/d-10m/

!! Warning! These files are not projected in a way ESRI or GDAL understands. They WILL NOT HAVE A LOCATION IN QGIS. They will, however, satisfy the needs of the assignment.

# wget on mac is great.  This tool (default on linux) lets us grab GIS data from
# most providers, via FTP and similar protocols.

brew install wget

make some folders

mkdir GIS_Projects && cd GIS_Projects

use wget to download every .dem file (-A .dem) from the specified folder and sub-folders (-r)

wget -r -A .dem ftp://ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem/

cd ftp.granit.sr.unh.edu/pub/GRANIT_Data/Vector_Data/Elevation_and_Derived_Products/d-elevationdem

make an index file of only .dem files.
(If we needed to download other files and keep them from our wget (more common)
this way we can still sort the various files for .dem)

ls -1 *.dem > dem_list.txt

use gdal to make state-plane referenced “Output_merged.tif” from the list of files
in the index we made.
it will use a single generic "0 0 255" band to show gradient.

gdal_merge.py -init "0 0 255" -o Output_Merged.tif --optfile dem_list.txt

copy the resulting file to desktop, then return home

cp Output_Merged.tif ~/desktop && cd

if you want (recommended):

rm -rf GIS_Projects  # remove .dem files.  Some are huge!

In Finder at in ~/desktop, open the new file with QGIS. A normal photo viewer will NOT show any detail.

Need to make something like this a reusable script? In Terminal, just a few extra steps:

mkdir GIS_Scripts && cd GIS_Scripts

open an editor + filename. Nano is generally pre-installed on OSX.

nano GDAL_LiveMerge.sh

COPY + PASTE THE SCRIPT FROM ABOVE INTO THE WINDOW

  • ctrl+X , then Y for yes

make your file runnable:

chmod u+x GDAL_LiveMerge.sh

run with ./

./GDAL_LiveMerge.sh

You can now copy + paste your script anywhere you want and run it there. scripts like this should not be exported to your global path / bashrc and will only work if they are in the directory you are calling them: If you need a global script, there are plenty of ways to do that too.

See /Notes_GDAL/README.md for notes on building GDAL from source on OSX

Click here to visit this repo

🙂

Decentralized Pi Video Monitoring w/ motioneye & BATMAN

Visit the me here on Github
Added parabolic musings 10/16/19, see below

...On using motioneye video clients on Pi Zeros & Raspbian over a BATMAN-adv Ad-Hoc network

link: motioneyeos
link: motioneye Daemon
link: Pi Zero W Tx/Rx data sheet:
link: BATMAN Open Mesh

This implementation of motioneye is running on Raspbian Buster (opposed to motioneyeos).

Calculating Mesh Effectiveness w/ Python:
Please take a look at dBmLoss.py- the idea here is one should be able to estimate the maximum plausible distance between mesh nodes before setting anything up. It can be run with no arguments-

python3 dBmLoss.py

...with no arguments, it should use default values (Tx = 20 dBm, Rx = |-40| dBm) to print this:

you can add (default) Rx Tx arguments using the following syntax:
                 python3 dBmLoss.py 20 40
                 python3 dBmLoss.py <Rx> <Tx>                 

 57.74559999999994 ft = max. mesh node spacing, @
 Rx = 40
 Tx = 20

Regarding the Pi:
The Pi Zero uses an onboard BCM43143 wifi module. See above for the data sheet. We can expect around a ~19 dBm Tx signal from a BCM43143 if we are optimistic. Unfortunately, "usable" Rx gain is unclear in the context of the Pi.

Added 10/16/19:
Notes on generating an accurate parabolic antenna shape with FreeCAD’s Python CLI:

For whatever reason, (likely my own ignorance) I have been having trouble generating an accurate parabolic dish shape in Fusion 360 (AFAICT, Autodesk is literally drenching Fusion 360 in funds right now, I feel obligated to at least try). Bezier, spline, etc curves are not suitable!
If you are not familiar with FreeCAD, the general approach- geometry is formed through fully constraining sketches and objects- is quite different from Sketchup / Tinkercad / Inventor / etc, as most proprietary 3d software does the “constraining” of your drawings behind the scenes. From this perspective, you can see how the following script never actually defines or changes the curve / depth of the parabola; all we need to do is change how much curve to include. A wide, shallow dish can be made by only using the very bottom of the curve, or a deep / narrow dish by including more of the ever steepening parabolic shape.

import Part, math

# musings derived from:
# https://forum.freecadweb.org/viewtopic.php?t=4430

# thinking about units here:
tu = FreeCAD.Units.parseQuantity

def mm(value):
    return tu('{} mm'.format(value))

rs = mm(1.9)
thicken = -(rs / mm(15)) 

# defer to scale during fitting / fillet elsewhere 
m=App.Matrix()
m.rotateY(math.radians(-90))
# create a parabola with the symmetry axis (0,0,1)
parabola=Part.Parabola()
parabola.transform(m)

# get only the right part of the curve
edge=parabola.toShape(0,rs)
pt=parabola.value(rs)
line=Part.makeLine(pt,App.Vector(0,0,pt.z))
wire=Part.Wire([edge,line])
shell=wire.revolve(App.Vector(0,0,0),App.Vector(0,0,1),360)

# make a solid
solid=Part.Solid(shell)

# apply a thickness
thick=solid.makeThickness([solid.Faces[1]],thicken,0.001)
Part.show(thick)

Gui.SendMsgToActiveView("ViewFit")

"""
# Fill screen:
Gui.SendMsgToActiveView("ViewFit")
# Remove Part in default env:
App.getDocument("Unnamed1").removeObject("Shape")
"""

FWIW, here is my Python implimentation of a Tx/Rx "Free Space" distance calulator-
dBmLoss.py:

from math import log10
from sys import argv
'''
# estimate free space dBm attenuation:
# ...using wfi module BCM43143:

Tx = 19~20 dBm
Rx = not clear how low we can go here

d = distance Tx --> Rx
f = frequency
c = attenuation constant: meters / MHz = -27.55; see here for more info:
https://en.wikipedia.org/wiki/Free-space_path_loss
'''

f = 2400  # MHz
c = 27.55 # RF attenuation constant (in meters / MHz)

def_Tx = 20  # expected dBm transmit
def_Rx = 40  # (absolute value) of negative dBm thesh

def logdBm(num):
    return 20 * log10(num)

def maxDist(Rx, Tx):
    dBm = 0
    d = .1  # meters!
    while dBm < Tx + Rx:
        dBm = logdBm(d) + logdBm(f) - Tx - Rx + c
        d += .1  # meters!
    return d

# Why not use this with arguments Tx + Rx from shell if we want:
def useargs():
    use = bool
    try:
        if len(argv) == 3:
            use = True
        elif len(argv) == 1:
            print('\n\nyou can add (default) Rx Tx arguments using the following syntax: \n \
                python3 dBmLoss.py 20 40 \n \
                python3 dBmLoss.py <Rx> <Tx> \
                \n')
            use = False
        else:
            print('you must use both Rx & Tx arguments or no arguments')
            raise SystemExit
    except:
        print('you must use both Rx & Tx arguments or no arguments')
        raise SystemExit
    return use

def main():

    if useargs() == True:
        arg = [int(argv[1]), int(argv[2])]
    else:
        arg = [def_Rx, def_Tx]

    print(str('\n ' + str(maxDist(arg[0], arg[1])*3.281) + \
        ' ft = max. mesh node spacing, @ \n' + \
        ' Rx = ' + str(arg[0]) + '\n' + \
        ' Tx = ' + str(arg[1])))

main()

Google Calendar API with Chapel & Python

Mirroring the repo here:

Google Calendar API with Chapel & Python

Despite Chapel's many quirks and annoyances surrounding string handling, its efficiency and ease of running in parallel are always welcome. The idea here is a Chapel script that may need to weed through enormous numbers of files while looking for a date tag ($D + other tags currently) is probably a better choice overall than a pure Python version. (the intent is to test this properly later.)

As of 9/19/19, there is still a laundry list of things to add- control flow (for instance, “don't add the event over and over”) less brittle syntax, annotations, actual error handling, etc. It does find and upload calendar entries though!

I am using Python with the Google Calendar API (see here: https://developers.google.com/calendar/v3/reference/) in a looping Daemon thread. All the sifting for tags is managed with the Chapel binary, which dumps anything it finds into a csv from which the daemon will push calendar entries with proper formatting. FWIW, Google’s dates (datetime.datetime) adhere to RFC3339 (https://tools.ietf.org/html/rfc3339) which conveniently is the default of the datetime.isoformat() method.

Some pesky things to keep in mind:

This script uses a sync$ variable to lock other threads out of an evaluation during concurrency. So far I think the easiest way to manage the resulting domain is from within a module like so:

module charMatches {
  var dates : domain(string);
}

Here, domain charMatches.dates will need to accessed as a reference variable from any procedures that need it.

proc dateCheck(aFile, ref choice) {
    ...
}
... 
coforall folder in walkdirs('check/') {
    for file in findfiles(folder) {
        dateCheck(file, charMatches.dates);
    }
}

errors like:

error: unresolved call '_ir_split__ref_string.size'

unresolved call 'norm(promoted expression)'

...or other variants of:
string.split().size  (length, etc)

...Tie into a Chapel Specification issue.

https://github.com/chapel-lang/chapel/issues/7982

The short solution is do not use .split; instead, I have been chopping strings with .partition().

// like so:
...
if choice.contains(line.partition(hSep)[3].partition(hTerminate)[1]) == false {
    ...process string...
    ...
}
« Older posts