(LEFT) Kishenehnoasilus bhl gen. sp. nov. Dikow 2019 (Diptera: Asilidae) & (RIGHT) Sylvicola silibrarius sp. nov. Greenwalt 2019 (Diptera: Anisopodidae). These two extinct species of flies were collected from the Kishenehn Formation in Northwestern Montana as part of the National Museum of Natural History’s research in paleobiology and the evolution of insect diversity. The fossils were exposed by erosion of oil shale, derived from sediments of a 46-million-year-old lake, by the Middle Fork of the Flathead River along the southwestern boundary of Glacier National Park. The robber fly (Kishenehnoasilus bhl; USNM 624491) was collected at the Dakin site in 2014; the wood gnat (Sylvicola silibrarius; USNM 626077) was collected from the Deep Ford site in 2015. The scale bars = 5 mm. The specimens are named in honor of the staff of the Biodiversity Heritage Library and the library staff of the National Museum of Natural History, respectively. Photo by: Dale Greenwalt.

Two new fossil fly species have been named in honor of the Smithsonian Libraries and the Biodiversity Heritage Library. Sylvicola silibrarius Greenwalt, 2019 and Kishenehnoasilus bhl Dikow, 2019 have been described from specimens collected from the 46-million-year-old Kishenehn Formation of northwestern Montana. The species were published last week in the open access journal Palaeontologia Electronica.

The species are two of 15 new species identified in a new study led by scientists at the Smithsonian’s National Museum of Natural History and colleagues on the diversity of fossil flies (order Diptera) in the Coal Creek Member of the Kishenehn Formation, a 34-56 million year old fossil site. The study also records 15 new families, dramatically expanding scientists’ knowledge of fly diversity during the Eocene in North America. With this publication, a total of 21 different families of flies have now been described from the region. The study provides researchers with a snapshot of flies during their evolution and helps establish the Kishenehn Formation as one of the most important fossil sites in North America for understanding the evolution of one of the most successful groups of life on Earth.

Sylvicola silibrarius Greenwalt, 2019 is a wood gnat in the widely-distributed Anisopodidae family containing nearly 200 species and boasting a rich fossil record. The specific epithet is a combination of the abbreviation SI (for the Smithsonian Institution) and the Latin term librarius (pertaining to books). Dale Greenwalt, lead study author and a resident research associate at the National Museum of Natural History, named the species in appreciation for the essential services that the Smithsonian Libraries perform.

“My research here at the National Museum of Natural History would be impossible without the Libraries,” said Greenwalt. “In my ten years here at the museum, I have never had an interaction with our world class library staff that was not positive, friendly and helpful.”

Kishenehnoasilus bhl Dikow, 2019, a robber fly in the family Asilidae, represents the type species for this new genus. The specific epithet is an abbreviation referring to the Biodiversity Heritage Library (BHL). Torsten Dikow, study co-author and curator of Diptera at the National Museum of Natural History, drew a comparison between the fly and the Library when naming the species.

“Taxonomic research involves not only the study of specimens, but also the review of often old literature,” said Dikow. “Having digital access to publications enhanced with tools such as scientific name recognition makes my research so much more efficient. The Biodiversity Heritage Library provides a lens into the literature published a long time ago. The same is true for this new species of robber fly, as it helps us to understand and document the past of biodiversity.”

The type specimens for both Sylvicola silibrarius and Kishenehnoasilus bhl have been deposited within the paleobiology collections of the Smithsonian’s National Museum of Natural History, where they will be available to researchers to help inform future studies on dipteran evolution and diversification.

“For centuries, libraries have been essential centers of knowledge with collections serving as windows into the universe of human scholarship, history and creativity,” said Martin R. Kalfatovic, associate director of the Smithsonian Libraries and program director for the Biodiversity Heritage Library. “As libraries have evolved to embrace the digital world — from online access for collections to data mining those same collections for new discoveries — they have empowered new research efficiencies and connections and facilitated the democratization of knowledge through open access principles like those embraced by the Smithsonian and Biodiversity Heritage Library. That these researchers have honored our libraries through these species is a testament to the essential services that libraries provide today. We are privileged to be recognized in such a meaningful way for our impact on the global research community.”

Citation: Greenwalt, D.E., and Bickel, D.J., Kerr, P.H., Curler, G.R., Brown, B.V., de Jong, H., Scott J. Fitzgerald, S.J., Dikow, T., Tkoč, M. and Kehlmaier, C. 2019. Diptera of the Kishenehn Formation. I. Documentation of diversity at the family level. Palaeontologia Electronica. 22(2):1-57. https://doi.org/10.26879/891.

Insects trapped in amber.

For anyone who has seen Jurassic Park, this description immediately conjures up familiar imagery. In the movie, such a fortuitously-preserved mosquito provided the means to bring dinosaurs back to life. While that may be the realm of science fiction, in the realm of science, such amber time capsules are still a valuable window into the past, allowing scientists today to examine ancient specimens and, sometimes, discover new species.

Take, for example, Cyrtinella flavinigra.

Cyrtinella flavinigra gen. et sp. nov. A, male (right) and female (left) paratypes (SMF Be 1267a and 1267b), dorsal view, photo: John Martin; B, male (bottom) and female (top) paratypes (SMF Be 1267a and 1267b), ventral view, photo: John Martin. Images from: Jessica P. Gillung & Shaun L. Winterton (2018) A review of fossil spider flies (Diptera: Acroceridae) with descriptions of new genera and species from Baltic Amber, Journal of Systematic Palaeontology, 16:4, 325-350, DOI: 10.1080/14772019.2017.1289566.

This species of spider fly lived during the Eocene Epoch, a major division of the geological timescale that lasted from 56 to 33.9 million years ago. Several specimens of this species were recovered from amber deposits in the Prussian Formation of the Baltic region.

These specimens were used by Drs. Jessica P. Gillung and Shaun L. Winterton to describe a new genus and species with the designation Cyrtinella flavinigra. This was one of three new species described by the duo from specimens preserved in Baltic amber within their review of fossil spider flies (Diptera: Acroceridae) published within the Journal of Systematic Palaeontology in 2017.

Cyrtinella flavinigra is also just one of 30 new insect species described by Gillung, a Postdoctoral Associate at the Danforth Lab at Cornell University. As an evolutionary biologist with a specialization in entomology, Gillung’s research focuses on biodiversity discovery and cataloging and the processes that originate and maintain biodiversity. A major aspect of her work is describing new species.

The Biodiversity Heritage Library (BHL) provides critical resources for this taxonomic work.

“For species discovery and cataloging, I need to be able to track down and understand the species that have already been described,” explains Gillung. “If it wasn’t for BHL, I would not have been able to access many original species descriptions and compare them to the specimens I had at hand. Through those comparisons, mainly, I was able to recognize a new species or identify a previously described one.”

Jessica Gillung holding Atlas Moths at the Bohart Museum of Entomology. Photo courtesy: Kathy Garvey.

Gillung has been studying evolutionary biology since 2006. As a Master’s student in Brazil, her research focused on the family Acroceridae, or spider flies, which develop almost exclusively as endoparasitoids of spiders, their maggots living within a host spider while feeding on its internal fluids until they metamorphosize into adults, leaving a dead, dry host behind.

A spider fly with a very hunchbacked appearance (species: Megalybus pictus). Photo courtesy: Vicente Valdés Guzmán.

It was during her time as a Master’s student ten years ago that Gillung first discovered BHL.

“I was looking for some ancient, specialized papers on spider flies, and a colleague sent me the link to the website. It changed my life! I was able to find pretty much EVERY single research paper I ever wanted, and I was able to study that literature very easily.”

Following the completion of her Master’s, Gillung moved to California to begin her PhD in the Department of Entomology & Nematology at University of California – Davis, where she also worked at the Bohart Museum of Entomology, one of the largest and most important natural history collections in the United States. She used BHL almost daily during this period, downloading whole and later custom PDFs of relevant publications and pages to her external hard drive, building a personal library of invaluable reference materials.

As she confided, “I downloaded pretty much EVERY work that has ever been published about the insect group I study.”

Jessica Gillung collecting insects in the field. Photo courtesy: Laura Breitkreuz.

The ability to download PDFs from BHL has allowed Gillung to continue her research anytime, anywhere — even when Internet access is unavailable.

“I specifically remember one time I was stuck at the airport on my way to visit my family in Brazil,” recalls Gillung. “My flights were delayed, and I had long layovers. So, I started working on one of my scientific publications, and I was able to get plenty of work done because I had saved the PDFs beforehand. Had BHL not existed, that would not have been possible.”

A specimen of Philopota tuberculata. Photo courtesy: Ricardo Kawada.

Of the over 240,000 volumes available in BHL, Gillung’s favorite is Wiedemann’s 1830 monograph on the “Flies from Outside of Europe” (Aussereuropäische Zweiflügelige Insekten). A German historian, physician, naturalist, and entomologist, Wiedemann described many new insect species, including the fly genus Philopota, which Gillung studied for her Master’s, during which she completed a revision of the genus. As part of the revision, Gillung re-described the known species, provided photographs and drawings for their identification, and described 13 new species from throughout Latin America.

“The library at my university did not have a copy of Wiedemann’s monograph,” recalls Gillung. “My progress was hindered until I was able to download the whole PDF from BHL. I fell in love with the website!”

Illustration from Wiedemann’s Aussereuropäische Zweiflügelige Insekten v. 2 (1830). Contributed in BHL from Smithsonian Libraries.

By providing free and open access to critical biodiversity literature, the Biodiversity Heritage Library is empowering global research. From enabling the description of new species to providing access to beautiful scientific illustrations (Gillung confides that one of her favorite pastimes is exploring the plates in the PDFs she’s downloaded), BHL has become an invaluable resource for researchers around the world.

To quote Dr. Gillung: “BHL has made my research possible.”

Dr. Robert Mesibov in millipede collecting hat.

Like many taxonomists, I like to group things together and sort them: specimens into species, species into genera, references into bibliographies, images into galleries. The Biodiversity Heritage Library has been a powerful enabler for me as a grouper-sorter.

Fifteen or so years ago, a literature search still required at least one long and expensive trip from my home town in regional Tasmania to an academic library in Hobart (Tasmania’s capital city), or in Melbourne or Canberra on the Australian mainland. Reams of paper were used to make photocopies of key references that I could take home with me. Weeks were spent waiting for additional photocopies to arrive through inter-library loans. When I finally had all the relevant older references, I could do my revisionary taxonomic work.

Then BHL appeared.

For the past 10 years, almost all the older literature I need to see has been accessed through BHL — quickly, cheaply and paperlessly.

In 2002, I compiled the first Australian millipedes checklist in the old way for the Australian government’s Fauna of Australia publication series. The successor to that checklist is the Millipedes of Australia website (2006 – 2019), an online catalogue I maintain. In 2009 I began adding links to open-access references in the website’s bibliography. That bibliography currently covers the period 1833-2018, and a remarkable 66 of the 72 references from the first 90 years of Australian millipede taxonomy are in BHL. What a savings in time and money for website users!

Screenshot of a section of the bibliography on the Millipedes of Australia website. Links to BHL can be seen for many of the included references.

In preparation for my upcoming retirement this year, I will be depositing the Millipedes of Australia in a public data repository and publishing a data paper to describe the archived website. The Tasmanian locality records have also been given to the Atlas of Living Australia, allowing continued access to this data.

BHL has also made it easy for me to gather up published taxonomic illustrations in digital form, and then to manipulate the images into scaled, side-by-side comparisons of different species. Although I mainly do this for my own use, I produced a “gallery” document (in PDF format) of diagnostic images in 2017 to accompany an annotated catalogue of southern South American millipedes in the family Dalodesmidae (https://doi.org/10.11646/zootaxa.4338.3.6). This document has since been provided to South American and other millipede workers. It’s a big time-saver. Millipede species are mainly distinguished through differences in the male genitalia, and having illustrated genitalia of almost all the 52 dalodesmids in one place makes for instant enlightenment: “Ah ha! So that’s why that species was put in that genus!”

Many of the illustrations are from publications digitally archived in BHL.

Views of the male genitalia of the South American millipede Anaulacodesmus levissimus (Attems, 1898), as illustrated by C. Attems in 1898 (left; Fig. 72) and F. Silvestri in 1905 (right; Figs 10-13).

I’ve also been compiling taxonomic treatments (descriptions) in this way using BHL’s OCR output, edited where necessary. No more flipping through numerous old journals and photocopies, and scanning or typing blocks of text! I can quickly build a document containing multiple descriptions of the same species, or of genus-diagnostic characters across multiple species, all in one place for handy reference, and with BHL page URLs if I need to revisit the sources.

OK, maybe “enabled” isn’t the best word to describe what BHL has done for me as a taxonomist. “Spoiled” would be more accurate!

Title page. Vroeg, Adrian. Beredeneerde catalogus van eene, by uitstek fraaye en weergaalooze verzameling, zoo van inlandsche als uitheemsche vogelen, viervoetige en gekorvene dieren. 1764. Contributed in BHL from Smithsonian Libraries.

As an early work in the history of Linnaean taxonomy, Beredeneerde catalogus van eene, by uitstek fraaye en weergaalooze verzameling, zoo van inlandsche als uitheemsche vogelen, viervoetige en gekorvene dieren (i.e. Vroeg’s Catalogue, 1764) by Adrian Vroeg is the source of dozens of new species of birds [1]. Published just six years after the 10th edition of Linnaeus’ Systema Naturae — considered the starting point of zoological nomenclature — the work is extremely rare, with only a handful of copies known to exist worldwide.

Because of the work’s age and rarity, first-hand access to the title has been difficult, and many researchers have had to rely on secondary sources for Vroeg’s names, which may have introduced errors or even overlooked the priority of a name established by Vroeg.

“It is absolutely rare that the scientific community gets access to such an early work in which new names were established after 1758,” explains Dr. Francisco Welter-Schultes of the Zoological Institute of Göttingen University in Germany. “Imagine being able to finally view an original spelling of a name that might not have been verified for more than 100 years. Generations of scientists never reliably saw the correct spellings of these names.”

Dr. Francisco Welter-Schultes of the Zoological Institute of Göttingen University in Germany.

Between 2008-2011, during his time on the BHL Europe project, Welter-Schultes served as the Lead of the Project Group for AnimalBase, which “was originally established to link the early literature with the names of the animals described therein” [2]. The database provides original and corrected spellings of names, type localities, citations for the original descriptions, and, where, possible, links to digitized copies of these descriptions in the literature — many from the Biodiversity Heritage Library.

A major reference for the Project Group’s work was Sherborn’s Index Animalium (1902), “a compendium of zoological taxonomic species nomenclature from 1758 to 1850” [3]. Where possible, the Project Group compared the Index with the original works to verify the accuracy of the names and make any necessary corrections. If it was not possible to consult the original work, the Group had to rely on Sherborn’s Index.

The question, of course, is whether Sherborn himself was able to verify every name against the original source or whether he, too, had to rely on secondary sources in some instances.

“I knew immediately after realizing the date of Vroeg’s publication that new names which were cited in secondary sources like Sherborn’s Index Animalium were not very reliably cited,” explains Welter-Schultes. “If a work is that rare, then the likelihood is greater that Sherborn may never have seen the original. Did Sherborn just copy errors from other lists, and we just copied Sherborn’s forwarded errors to AnimalBase?”

In the hopes of being able to verify the names published by Vroeg in 1764, Welter-Schultes submitted a scan request to the Biodiversity Heritage Library for Vroeg’s Catalogue in 2010. At the time of the request, no BHL partner library appeared to hold the title. So, the request remained in the queue in the chance that, with the addition of new partners in the future or the growth of existing partners’ collections, it might be possible to digitize the work at a later date.

A few years later, whilst reviewing outstanding requests in the queue, Smithsonian Libraries staff discovered that its Joseph F. Cullman 3rd Library of Natural History had since cataloged the item after it was transferred from the Libraries’ general collections. The Smithsonian Libraries is not only the sole BHL partner to hold this work, but the Libraries’ copy is also one of only a few known copies worldwide, with the others being in Denmark (2), the Netherlands (2), and Germany (1) [4].

The catalog record provides insight into the interesting provenance of the Smithsonian Libraries’ copy of this very rare book, which was previously owned (and donated) by Charles Wallace Richmond, a Smithsonian ornithologist and bibliographer who served as Assistant Curator, Associate Curator, and Curator in the Division of Birds of the United States National Museum (USNM) (now the Smithsonian National Museum of Natural History) between 1894 and his death in 1932 [5]. Richmond’s signature can be seen on the verso of the title page, dated Aug. 31, 1905, and a book dealer’s mailing label to Richmond from Frederik Muller & Co. of Amsterdam is bound into the volume.

Richmond’s signature on the verso of the title page of the Smithsonian Libraries’ copy of Vroeg’s Catalogue.

Book dealer’s mailing label to Richmond from Frederik Muller & Co. of Amsterdam in the Smithsonian Libraries’ copy of Vroeg’s Catalogue.

An avid book collector, Richmond’s personal library consisted of over 200 publications and auction catalogs in German, Italian, French, Spanish, Latin, and English on various natural history topics — many quite rare as Richmond specifically sought out obscure books, particularly those with new species names. Richmond donated the contents of his library to several institutions, including not only the Smithsonian but also the Academy of Natural Sciences of Philadelphia, the Library of Congress, and the California Academy of Sciences — all of which are BHL partner institutions [6]. His donation of taxonomic books to the Smithsonian became the foundation of the Bird Library at the National Museum of Natural History [7].

Unfortunately, the Smithsonian’s copy of Vroeg’s Catalogue was in need of conservation before it could be scanned. The item was added to the Smithsonian Libraries’ preservation queue, and once the necessary treatments were completed, it was added to the Libraries’ queue for folio digitization.

Finally, in 2018 — eight years after the initial request, Vroeg’s Catalogue was digitized by the Smithsonian Libraries’ digitization team and made freely available online through the Biodiversity Heritage Library.

Several Smithsonian Libraries staff members were involved in completing this book’s journey from shelf to screen, including Digital Library staff Jacqueline Chapman, Adriana Marroquin, and Stefaan Hurts; Special Collections staff Leslie Overstreet, Daria Wingreen-Mason, and Alexandra Newman; Cataloging staff Julia Blakely; Preservation staff Katie Wagner; and BHL Secretariat staff Bianca Crowley and Grace Costantino. We extend our thanks to all involved for making this extremely rare book freely available to the world.

And because this work is now easily accessible to researchers, it can be used to empower global research, such as that of Welter-Schultes. His response to Jacqueline Chapman’s email notification that the work had been digitized was swift: “This was really very, very useful. I immediately spent two hours exploring this interesting work.”

With the digital copy of the Smithsonian Libraries’ Catalogue, Welter-Schultes was able to uncover taxonomic errors that had persisted for at least a century and improve the accuracy of the names data available through AnimalBase.

“It was easy for me to verify our data set of names, and in fact it turned out that Sherborn must have relied on a third source,” explains Welter-Schultes. “There were so many inaccuracies in the spellings of the names that we must assume that Sherborn, who in his work (published in 1902) had a very low average error rate, did very probably not extract those names from an original copy of the 1764 work.”

New bird species introduced in Vroeg’s Catalogue. Vroeg, Adrian. Beredeneerde catalogus van eene, by uitstek fraaye en weergaalooze verzameling, zoo van inlandsche als uitheemsche vogelen, viervoetige en gekorvene dieren. 1764. Contributed in BHL from Smithsonian Libraries.

Knowing that their work is so valued and impactful is also very encouraging for the staff who dedicate so much time and energy to addressing the needs of BHL’s user community.

“It is rare that we hear from users that the items we digitize in response to these older requests are so immediately relevant and impactful on current research,” shares Chapman. “And it is so fulfilling!”

Thanks to the efforts of the staff at Smithsonian Libraries and other BHL partners around the world, researchers can now easily access original species descriptions and other rare literature that underpins the work of taxonomists today.

“Today, thousands of names in online data resources are equipped with links to their original descriptions in BHL,” lauds Welter-Schultes. “What we did 10 years ago with AnimalBase was pioneering work — many others followed and did the same. Today, providing a link to a BHL source is something absolutely usual and taxonomists can probably not imagine life anymore without BHL.”

We are grateful to Dr. Welter-Schultes for sharing this example of how BHL is impacting real research today. We also extend our sincere thanks to all of our partners for making BHL the essential resource that it is today. The dedication and diligence of the staff at our partner libraries to support the scientific and research community — whether that involves more straightforward digitization projects or complex, multi-year undertakings —  ensures that the Biodiversity Heritage Library will continue to empower global research for decades to come.

References

[1] AnimalBase. 2018. “Taxa described as new in Vroeg 1764.” Accessed on 29 November 2018. http://www.animalbase.uni-goettingen.de/zooweb/servlet/AnimalBase/list/taxa?from_reference=1908.

[2] AnimalBase. 2018. “About.” Accessed on 29 November 2018. http://www.animalbase.org/.

[3] Smithsonian Libraries. 2018. “About this Project: Index Animalium.” Accessed on 29 November 2018. http://www.sil.si.edu/DigitalCollections/indexanimalium/about/.

[4] Leslie Overstreet, email message to author, 24 September 2018.

[5] Stone, Witmer. 1933. “In Memoriam: Charles Wallace Richmond.” The Auk, 50(1): 1-22.

[6] Smithsonian Institution Archives. 2011. “Record Unit 7382,  Richmond, Charles Wallace, 1868-1932, Charles Wallace Richmond Library Catalog and Lists, 1906-1908 and undated.” Collections, 19 August. Accessed on 29 November 2018. https://siarchives.si.edu/collections/siris_arc_217538.

[7] Smithsonian National Museum of Natural History. 2018. “Charles W. Richmond (1868-1932).” Department of Vertebrate Zoology: Division of Birds. Accessed on 29 November 2018. http://vertebrates.si.edu/birds/Hall_of_fame/CharlesRichmond.html.

This post was originally published on the rOpenSci blog on 28 August 2018 and is republished with permission of the author, Dr. Maëlle Salmon, and rOpenSci.

Armed with rOpenSci’s packages binding powerful C++ libraries and open taxonomy data, how much information can we automatically extract from images? Maybe not much, but, experimenting with gorgeous drawings from a natural history collection, we can least explore image manipulation, optical character recognition (OCR), language detection, and taxonomic name resolution with rOpenSci’s packages.

Free natural history images and appropriate R tooling!

A long time ago I had bookmarked the Flickr account of the Biodiversity Heritage Library (BHL). So many beautiful images of biodiversity, moreover free to use! In particular, I downloaded all pictures from one of the Birds of Australia albums (contributed in BHL by Museums Victoria).

I wanted to try to extract the bird names from images using packages of Jeroen Ooms’, rOpenSci’s post-doc hacker & C(++)-bindings wizard. For that I worked with magick for image manipulation, tesseract for optical character recognition (OCR), cld2/cld3 for language detection… Quite the armory! We’ll also sprinkle some taxonomy magic by Scott Chamberlain, one of rOpenSci’s co-founders, to resolve the names extracted.

OCR bird naming workflow, piece by piece

In this section, we explain the different elements of our R workflow: preparing images, extracting text, resolving taxonomic names.

Image preparation

I saved the pictures locally in a “birds” folder. Yes, I click-buttoned instead of using the Flickr API for which e.g. Jim Hester wrote a minimal R wrapper… I don’t do everything with R scripts (yet).

library("magrittr") 
filenames <- fs::dir_ls("birds")

Each image shows a bird and its name. Images are either landscape- or portrait-oriented, but for the sake of simplicity, I’ll act as if they were all portrait-oriented. A possible easy and lazy fix here would be to duplicate images rotated (magick::image_rotate) in all possible directions and then to apply the workflow to all 4 versions of each image, hoping to get good data from one of the rotated versions.

magick::image_read(filenames[1])

Mathews, Gregory M. The Birds of Australia. v. 11 (1923-24). Art by Henrik Grönvold. Contributed in BHL from Museums Victoria.

From that image, I wanted to extract the name indicated below the bird. To maximize the efficiency of OCR, I shall first prepare the image, since the accuracy of OCR depends on the quality of the input image which can be influenced a bit. This part could be tweaked even more, and in real life examples it’ll be worth spending time trying different magick functions and parameter values. Since I have in mind the case of a bunch of images to be batch-processed, nothing is done by hand.

crop_bird <- function(filename){
  image <- magick::image_read(filename)

  height <- magick::image_info(image)$height

  # crop the top of the image
  image <- magick::image_crop(image, 
                     paste0("+0+",round(0.75*height))) %>%
    # convert the image to black and white
    magick::image_convert(type = "grayscale") %>%
    # increase brightness
    magick::image_modulate(brightness = 120) %>%
    magick::image_enhance() %>%
    magick::image_median() %>%
    magick::image_contrast() 

  # we'll need the filename later
  attr(image, "filename") <- filename

  return(image)
}

crop_bird(filenames[1])

It does look cleaner now!

Text extraction

Now is the actual OCR step! The tesseract package provides bindings to the Tesseract OCR engine, free software currently sponsored by Google. It is a powerful engine, with a ton of parameters. Here again, tweaking a lot is warranted. Particularly useful reads are hocr vignette and this Wiki page of Tesseract about improving the quality of the output. The hocr package might be of interest for post-processing of OCR results.

Below, the only option changed from default is the page segmentation mode choosing 1 for “Automatic page segmentation with Orientation and script detection (OSD)”. When using Latin instead of English training data the results were not as good.

One can use either tesseract::ocr for a file path, url, or raw vector to image, or magick::image_ocr for a magick object which is quite handy in our pipeline.

The function below also filters results of the OCR using language detection. By only keeping text recognized as either Latin or English by one of the two language detection packages cld2 and cld3 that are interfaces to Google compact language detectors 2 and 3, one gets a first quality filter. If not doing that, the output would contain more unusable text.

get_names <- function(image){
  filename <- attr(image, "filename")
  ocr_options <- list(tessedit_pageseg_mode = 1)

  text <- magick::image_ocr(image, options = ocr_options)
  text <- stringr::str_split(text, "\n", simplify = TRUE)
  text <- stringr::str_remove_all(text, "[0-9]")
  text <- stringr::str_remove_all(text, "[:punct:]")
  text <- trimws(text)
  text <- stringr::str_remove_all(text, "~")
  text <- text[text != ""]
  text <- tolower(text)

  # remove one letter words
  # https://stackoverflow.com/questions/31203843/r-find-and-remove-all-one-to-two-letter-words
  text <- stringr::str_remove_all(text, " *\\b[[:alpha:]]{1,2}\\b *")
  text <- text[text != ""]

  # keep only the words that are recognized as either Latin
  # or English by cld2 or cld3
  if(length(text) > 0){
    results <- tibble::tibble(text = text,
                 cld2 = cld2::detect_language(text),
                 cld3 = cld3::detect_language(text),
                 filename = filename)

  results[results $cld2 %in% c("la", "en") |
          results$cld3 %in% c("la", "en"),]
  }else{
    return(NULL)
  }


}

(results1 <- filenames[1] %>%
  magick::image_read() %>%
  get_names())

## NULL

Nothing! Now if we replace magick::image_read with the previously defined crop_bird function that crops and cleans the image…

(results2 <- filenames[1] %>%
  crop_bird() %>%
  get_names())

## # A tibble: 2 x 4
##   text                 cld2  cld3  filename                          
##   <chr>                <chr> <chr> <chr>                             
## 1 climacteris picumnus <NA>  la    birds/n115_w1150_42399797481_o.jpg
## 2 brown tree creeper   en    <NA>  birds/n115_w1150_42399797481_o.jpg

We get a result! So we see that the image transformation was quite useful.

Now, these names look fine, but how to be sure they’re actually taxonomic names?

Taxonomic name resolution

The taxize package by Scott Chamberlain, is a taxonomic toolbelt for R, providing access to many fantastic data sources and tools for taxonomy. One of them, the Global Name Resolver, provides, well, resolution of taxonomic names, sadly not common names. taxize::gnr_resolve has many options, of which only one is used below: best_match_only = TRUE means it’ll only return the best match from the different data sources.

latin <- results2$text[results2$cld2 == "la"|
                         results2$cld3 == "la"]
taxize::gnr_resolve(latin,
  best_match_only = TRUE)

## # A tibble: 1 x 5
##   user_supplied_name submitted_name  matched_name   data_source_tit~ score
## * <chr>              <chr>           <chr>          <chr>            <dbl>
## 1 climacteris picum~ Climacteris pi~ Climacteris p~ NCBI             0.988

English names could be cleaned up a bit using the spelling package, also an rOpenSci package, since spelling::spell_check_text would output possible typos.

OCR bird naming workflow in action!

First the two steps image processing and OCR are applied to all images.

bird_names <- purrr::map(filenames, crop_bird) %>%
  purrr::map_df(get_names)

Out of 51 images only 17 are present in the final table with possible names which is a bit disheartening, but one could surely do better in the image processing and OCR tweaking steps! Maybe one could frame the parameter search as a machine learning problem. Please also keep in mind that such natural history images are quite hard to parse.

The name resolution information can be added to this table.

safe_resolve <- function(text){

  results <- taxize::gnr_resolve(text,
                                 best_match_only = TRUE)

  if(nrow(results) == 0){
    list(NULL)
  }else{
    list(results)
  }
}

bird_names <- dplyr::group_by(bird_names, text) %>%
  dplyr::mutate(gnr = ifelse(cld2 == "la" | cld3 == "la",
                             safe_resolve(text),
                             list(NULL)))

We do not get much resolution, but we knew the names weren’t very good to start with. A better (untested here!) idea here might be to get a full list of names of Australian birds, potentially leveraging the taxizedb package by Scott Chamberlain, and to then fuzzy-match them with the names we have.

unique(bird_names$gnr)

## [[1]]
## # A tibble: 1 x 5
##   user_supplied_name submitted_name  matched_name   data_source_tit~ score
## * <chr>              <chr>           <chr>          <chr>            <dbl>
## 1 climacteris picum~ Climacteris pi~ Climacteris p~ NCBI             0.988
## 
## [[2]]
## [1] NA
## 
## [[3]]
## NULL
## 
## [[4]]
## # A tibble: 1 x 5
##   user_supplied_na~ submitted_name  matched_name  data_source_title  score
## * <chr>             <chr>           <chr>         <chr>              <dbl>
## 1 austrodicaeum ii~ Austrodicaeum ~ Austrodicaeu~ The Interim Regis~  0.75
## 
## [[5]]
## # A tibble: 1 x 5
##   user_supplied_name submitted_name  matched_name   data_source_tit~ score
## * <chr>              <chr>           <chr>          <chr>            <dbl>
## 1 melithreptus laet~ Melithreptus l~ Melithreptus ~ CU*STAR          0.988
## 
## [[6]]
## # A tibble: 1 x 5
##   user_supplied_na~ submitted_name matched_name  data_source_title   score
## * <chr>             <chr>          <chr>         <chr>               <dbl>
## 1 rad isdlvorniode  Rad isdlvorni~ Rad Baker & ~ The Interim Regist~  0.75

Conclusion

rOpenSci packages supporting this (and your) workflow

In this post, we made use of R packages quite useful to wrangle information from diverse formats:

• magick for image manipulation,
• tesseract for optical character recognition (OCR),
• cld2/cld3 for language detection.

We also used a function from taxize allowing us to use the Global Name Resolver. Discover more packages from our suite here.

Applicability of this OCR bird naming workflow

Actually, the BHL itself provides OCR output for its collection, see this example. I wasn’t able to find information about the software powering this OCR. What I was able to find out is that the BHL uses purposeful gaming in its OCR workflow. The raw OCR results aren’t much better than what we got in this post which is comforting.

More data from the Biodiversity Heritage Library

If you’re interested in other types of data from the BHL, in addition to the images, have a look at the rbhl package, part of rOpenSci’s suite, that interacts with the BHL API. One can e.g. search all books by the same author as the one we used images from.

author <- rbhl::bhl_authorsearch("Gregory M Mathews")
books <- rbhl::bhl_getauthortitles(creatorid = author$CreatorID)
head(books$FullTitle)

## [1] "A manual of the birds of Australia,"                                                                                                                                                                                                                                          
## [2] "A list of the birds of the Phillipian sub-region : which do not occur in Australia. "                                                                                                                                                                                         
## [3] "A manual of the birds of Australia /"                                                                                                                                                                                                                                         
## [4] "A list of the birds of Australia : containing the names and synonyms connected with each genus, species, and subspecies of birds found in Australia, at present known to the author /"                                                                                        
## [5] "Austral avian record; a scientific journal devoted primarily to the study of the Australian avifauna."                                                                                                                                                                        
## [6] "Arcana, or, The museum of natural history : containing the most recent discovered objects : embellished with coloured plates, and corresponding descriptions : with extracts relating to animals, and remarks of celebrated travellers; combining a general survey of nature."

Or we could get all books whose title contains the words “birds” and “australia”.

australia_birds <- rbhl::bhl_booksearch(title = "birds Australia")
head(australia_birds$FullTitle)

## [1] "Handbook to the birds of Australia. : [Supplementary material in Charles Darwin's copy]."
## [2] "An introduction to The birds of Australia /"                                             
## [3] "The useful birds of southern Australia : with notes on other birds /"                    
## [4] "The Birds of Australia"                                                                  
## [5] "The birds of Australia,"                                                                 
## [6] "The birds of Australia,"

And to get the OCR results of the pages of the book we used, we could write:

library("magrittr")

# ocr=TRUE to extract OCR for all pages
rbhl::bhl_getitempages("250938", ocr = TRUE) %>%
  # for each page transform the type into a string
  dplyr::group_by(PageUrl) %>%
  dplyr::mutate(page_type = toString(PageTypes[[1]])) %>%
  # keep only the illustration pages
  # that are like the ones we used 
  dplyr::filter(page_type == "Illustration") %>%
  # from the data.frame extract the OCR
  dplyr::pull(OcrText) %>%
  head()

## [1] "491 \nFAL CUNCULUS LEUCOGASTER. \n( WHITE -BELLIED £ If BIKE - TIT) \nFALCUNCULUS FRONTATUS. \nSHRIKE - TIT). \n"                                                                                                                   
## [2] "492 \nA** \nOREOICA GUTTURALIS. \n(CRESTED BELL-BIRD). \n"                                                                                                                                                                          
## [3] "APHELOCEPHALA LEUCOPSIS \n( WHITE FACE ). \n"                                                                                                                                                                                       
## [4] "* \nAPHELOCEPHALA PE CTORALIS. \n(CHE <3 TNUT -BREASTED WHITEFA CEj. \nAPHELOCEPHALA NIGRICINCTA. \n(BE A CK-BAH.DED WHITE FA CEj. \n"                                                                                              
## [5] "H . Gronvold. del. \nWitherLy & C° \nSPHENOSTOMA CRIS TATUM \n(WEDGE BIEL). \n"                                                                                                                                                     
## [6] "49 6 \nH \n(jronvolcl. del. \nN E O SIT TA LE CJ C O CE PHAI.A. \n( WHITE ¦ HE AID EE THE EH UN HE FL). \nNEOSHTA ALBATA \n(F IE E> T Ft E EE UNNEFlj. \nNEOSITTA CHRYSOPTERA \nf OFi. A. NGE - wing-e d tree runner). \nWitWLjA \n"

So there’s quite a lot to explore!

More birding soon!

Follow this series on the rOpenSci blog to explore this topic further! In the meantime, happy birding!