…Despite being a chilly & wintery March up here in the White Mountains, there is no shortage of fun birds and exciting projects!

Merlin AI pipeline for Mushroom identification!

It’s happening, and it is going to be awesome YMMV, but YOLO:

Image-based mushroom identification

Artifacts:

Dataset: images.tgz train.tgz test.tgz
Annotator json: images.json categories.json config.json
@ai.columbari.us: Web Annotator! mo_example_task.tar.gz mo_example_task.zip

Local Processing:

Remote & Notebook Processing:

Setup (Locally)

# venv:
python3 -m venv mushroomobserver_venv
source mushroomobserver_venv/bin/activate
pip3 install -r requirements.txt

preprocess: (Locally)

python3 preprocess
  • Fetches & saves off gbif archive to ./static/
    • Checks the archive, tries loading it into memory etc
  • Fetches Leaflet Annotator binary & licenses from JessSullivan/MerlinAI-Interpreters
  • Generates an images.json annotation file from the assets selected by Joe & Nathan
  • Generates an categories.json file from the annotatable classes selected by Joe & Nathan
  • Downloads, organizes the selected assets from images.mushroomoberver.org at ./static/images/<category>/<id>.jpg
    • writes out images archive
  • More or less randomly divvies up testing & training image sets
    • writes out example testing/training archives; (while training it’ll probably be easier to resample directly from images.tgz from keras)

Structures:

  • Leaflet Annotator images.json Structure:
    • id: filename The MO image filename
    • category_id: The binomen defined in the ./static/sample_select_assets.csv; for directories and URIs this is converted to snake case.
    • taxon_id: the MO taxon id integer in the case of *_v1.csv; otherwise duplicate of category_id for now (eg *_v2.csv)
    • url: Temporary elastic ip address this asset will be available from, just to reduce any excessive / redundant traffic to images.mushroomobserver.org
    • src: imageURL The asset’s source URL form Mushroom Observer 
      [{
"id": "290214.jpg",
"taxon_id": "12326",
"category_id": "Peltula euploca",
"url": "https://mo.columbari.us/static/images/peltula_euploca/290214.jpg",
"src": "https://mushroomobserver.org/images/640/290214.jpg"
}]
  • ./static/ directory structure following python3 preprocess: 
    ├── static
├── categories.json
├── gbif.zip
├── images
|   ...
│   └── peltula_euploca
│       ├── <123456>.jpg
│       ...
│       └── <654321>.jpg
│   ...
├── images.tgz
├── testing
|   ...
│   └── peltula_euploca
│       ├── <234567>.jpg
│       ...
│       └── <765432>.jpg
│   ...
├── testing.tgz
├── training
|   ...
│   └── peltula_euploca
│       ├── <345678>.jpg
│       ...
│       └── <876543>.jpg
│   ...
├── images.json
├── training.tgz
├── js
│   ├── leaflet.annotation.js
│   └── leaflet.annotation.js.LICENSE.txt
├── md
│   ├── output_10_0.png
│   ├── output_11_0.png
│   ├── output_17_0.png
│   └── output_27_0.png
├── mo_example_task
│   ├── categories.json
│   ├── config.json
│   └── images.json
├── sample_select_assets_v1.csv
├── sample_select_assets_v2.csv
├── test.tgz
└── train.tgz
...

Train (Locally)

  python3 train

  • Fetches, divvies & shuffles train / validation sets from within Keras using archive available at mo.columbari.us/static/images.tgz

  • More or less running Google’s demo transfer learning training script in train/training_v1.py as of 03/17/21, still need to bring in training operations and whatnot from merlin_ai/ repo –> experiment with Danish Mycology Society’s ImageNet v4 notes

    Local Jupyter:

  • One may also open and run notebooks locally like this:

    • rename ipython notebook: 
      cp train/notebook/training_v1.ipynb.bak train/notebook/training_v1.ipynb
    • launch jupyter: 
      jupyter notebook
    • or without authentication: 
      jupyter notebook --ip='*' --NotebookApp.token='' --NotebookApp.password=''

Google Colab:

Notes:

Preprocess: (Jupyter)

  from load_dwca import MODwca
  from preprocess import Preprocess
  from common import *
  """MODwca():
  fetch & save off the gbif export
  make sure we can load the dwca archive into memory:
  """
  dwca = MODwca()

  """BuildImages():
  functions to construct image dataset and annotator artefacts
  """
  buildData = Preprocess()

  buildData.write_images_json()

  buildData.fetch_leaflet_tool()

  buildData.fetch_online_images(_json=STATIC_PATH + "images.json")

  buildData.export_tgz()

  buildData.split_training_testing()

  buildData.write_categories_json()

Train: (Jupyter)

  import matplotlib.pyplot as plt
  import numpy as np
  import os
  import PIL
  import tensorflow as tf
  import pathlib
  from tensorflow import keras
  from tensorflow.keras import layers
  from tensorflow.keras.models import Sequential
  dataset_url = "https://mo.columbari.us/static/images.tgz"
  data_dir = tf.keras.utils.get_file('', origin=dataset_url, untar=True)
  data_dir = pathlib.Path(data_dir)

  image_count = len(list(data_dir.glob('*/*.jpg')))
  print("Read " + str(image_count) + " images into Keras!")
  armillaria_tabescens = list(data_dir.glob('armillaria_tabescens/*'))
  PIL.Image.open(str(armillaria_tabescens[0]))

png

  PIL.Image.open(str(armillaria_tabescens[1]))

png

load parameters:

  batch_size = 32
  img_height = 500
  img_width = 375

  train_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  validation_split=0.1,
  subset="training",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size)
  val_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  validation_split=0.1,
  subset="validation",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size)
  class_names = train_ds.class_names
  num_classes = len(class_names)

  print("Read " + str(num_classes) + " classification classes:")

  x=1
  for _name in class_names:
    print(str(x) + ": " + _name)
    x += 1

Visualize:

  import matplotlib.pyplot as plt

  plt.figure(figsize=(10, 10))
  for images, labels in train_ds.take(1):
  for i in range(9):
    ax = plt.subplot(3, 3, i + 1)
    plt.imshow(images[i].numpy().astype("uint8"))
    plt.title(class_names[labels[i]])
    plt.axis("off")

png

  for image_batch, labels_batch in train_ds:
    print("Processed image batch shape: " + image_batch.shape.__str__())
    print("Processed labels batch shape: " + labels_batch.shape.__str__())
    break
  AUTOTUNE = tf.data.AUTOTUNE

  train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
  val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
  normalization_layer = layers.experimental.preprocessing.Rescaling(1./255)
  normalized_ds = train_ds.map(lambda x, y: (normalization_layer(x), y))
  image_batch, labels_batch = next(iter(normalized_ds))
  first_image = image_batch[0]
  # Notice the pixels values are now in [0,1].
  print(np.min(first_image), np.max(first_image))
  data_augmentation = keras.Sequential(
  [
    layers.experimental.preprocessing.RandomFlip("horizontal",
                                                 input_shape=(img_height,
                                                              img_width,
                                                              3)),
    layers.experimental.preprocessing.RandomRotation(0.1),
    layers.experimental.preprocessing.RandomZoom(0.1),
  ]
  )
  model = Sequential([
  data_augmentation,
  layers.experimental.preprocessing.Rescaling(1./255),
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Dropout(0.2),
  layers.Flatten(),
  layers.Dense(128, activation='relu'),
  layers.Dense(num_classes)
  ])
  model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])
  model.summary()
  epochs = 15
  history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs
  )
  acc = history.history['accuracy']
  val_acc = history.history['val_accuracy']

  loss = history.history['loss']
  val_loss = history.history['val_loss']

  epochs_range = range(epochs)

  plt.figure(figsize=(8, 8))
  plt.subplot(1, 2, 1)
  plt.plot(epochs_range, acc, label='Training Accuracy')
  plt.plot(epochs_range, val_acc, label='Validation Accuracy')
  plt.legend(loc='lower right')
  plt.title('Training and Validation Accuracy')

  plt.subplot(1, 2, 2)
  plt.plot(epochs_range, loss, label='Training Loss')
  plt.plot(epochs_range, val_loss, label='Validation Loss')
  plt.legend(loc='upper right')
  plt.title('Training and Validation Loss')
  plt.show()

png

Predict:

  tabescens_url = "https://www.mushroomexpert.com/images/kuo6/armillaria_tabescens_06.jpg"
  tabescens_path = tf.keras.utils.get_file('armillaria_tabescens_06', origin=tabescens_url)

  img = keras.preprocessing.image.load_img(
    tabescens_path, target_size=(img_height, img_width)
  )
  img_array = keras.preprocessing.image.img_to_array(img)
  img_array = tf.expand_dims(img_array, 0)

  predictions = model.predict(img_array)
  score = tf.nn.softmax(predictions[0])

  print(
    "This image most likely belongs to {} with a {:.2f} percent confidence."
    .format(class_names[np.argmax(score)], 100 * np.max(score))
  )
This image most likely belongs to armillaria_tabescens with a 45.49 percent confidence.