Last month, Smithsonian Libraries and Archives (SLA), Smithsonian Transcription Center (STC), and the Biodiversity Heritage Library (BHL) celebrated a significant milestone – technical staff worked collaboratively to integrate over 43,000 pages of transcription materials from STC into BHL. An additional 151,362 scientific name access points have now been added to the BHL search index for SLA archival field notes. These transcriptions enhance BHL’s full-text search, enable taxonomic name recognition, improve accessibility for vision-impaired users, and support climate research.

The Smithsonian Field Books Collection

Dall, William Healey. (1895). Alaska Coal Fields, Bogosloff Volcano, Corral Hollow, California, 1895 (Vol. 1, pp. 81 and 82).

The Smithsonian Field Books Collection is a set of primary source archival records selected from the Smithsonian Libraries and Archives (SLA). This material dates from the late nineteenth century and the first comprehensive biological survey of the continental United States to the most recently accessioned materials at the Smithsonian Institution Archives.

The collection includes personal records of naturalists and scientists such as William Healey Dall (1845 – 1927) and Cleofe Calderon (1929 – 2007) at work around the world and expedition records such as the Western Union Telegraph Expedition (1865 – 1867) of Russian America and the United States Exploring Expedition (1838 – 1842) of the Pacific Ocean.

Naturalists and scientists in the field recorded their firsthand observations and data in a wide variety of forms including diaries and journals, hand-drawn maps and tables of data, observation logs and specimen catalogs, correspondence and reports, manuscripts, sketches, photographs, even audio recordings.

Recognizing the Value of Accurate Transcription

Mining the rich information embedded in these field notes depends on accurate transcription. The majority of the field notes are handwritten – even the most recent ones. Digital surrogates provide high resolution images but fail to afford anything beyond visual accessibility. Transcription, however, opens up this material to full-text searching, pattern recognition, visual accessibility aids, and more.

Beyond the handwriting itself, the field notes also contain other non-textual information that can only be captured through transcription. Examples include ornithologist Martin Moynihan’s notations of bird song in remote regions of Central America or Charles Dolittle Walcott’s sketches and diagrams of sedimentary stratification where his fossil specimens were found in the canyons of the American Southwest.

Moynihan, M. (1963). 1962-1965 Andean Birds Mixed Flocks, Colombia, (4 of 4) (p. 78).

Chief naturalist with the U.S. Department of Agriculture, Vernon Orlando Bailey’s “Journal kept by Bailey on field trip to Wyoming and New Mexico, March 15-June 1906” focused on extermination techniques of gray wolves that would bring them to near extinction in the continental United States not long afterwards. The transcript includes descriptions of the sketches he included in his notes.

Smithsonian Transcription Center image with corresponding transcription, including a text description of an ink drawing of a cow. https://transcription.si.edu/view/6656/EBOtg

The Smithsonian Field Book Project’s initial goal was to catalog these hidden biodiversity research material, improving discoverability in keeping with FAIR (Findable Accessible Interoperable Reusable) Data Principles. Doing so quickly resulted in additional researcher demand for more access and usability – first to view the field notes online (digitization) and then to examine them more closely (transcription). Grant funding assisted in a major rapid-capture digitization effort. However when it came to transcribing the digitized field notes, the Archives simply lacked the capacity to meet the level of researchers’ demand.

Turning to digital volunteers and the just-launched Smithsonian Transcription Center in 2013 changed the transcription equation beyond our best expectations both in volume and in accuracy. We quickly came to recognize these community scientists as collaborators, “volunpeers”, in the effort to advance and disseminate knowledge.

We are far from the end of this journey. More than half the collection remains to be digitized, and over two thousand digitized field notes still need transcription.

Inspired to Transcribe

These archival field notes contain vital historic biodiversity information. By transcribing the handwriting into machine readable text, volunpeers can help inform current day scientists and assist with their research on a multitude of topics such as climate change, the extinction crisis, or the spread of invasive species. Transcribing field notes can also take volunpeers on an adventure across time and distance, accompanying the writer on their journey. Having these adventures transcribed in machine readable text can also help inform science historians assisting them by making these historic documents more easily findable, searchable, and reusable.

Volunpeers collaborate to transcribe as accurately as possible the pages of the field journals provided. Multiple volunpeers will work on each project page transcribing the hard-to-read handwriting into machine readable text. Once satisfied the transcription is as complete as possible, a volunpeer will mark the transcribed page as complete. The volunpeers will then move onto the next page of the field notes until the project is finished.

Review is the second step in the transcription process.

Having many volunpeers working on one project helps ensure the quality of the transcription. What one volunpeer finds illegible, others may be able to read, especially as all the volunpeers working on a particular project become more familiar with the handwriting.

Volunteers have transcribed hundreds of field journals over the years. Two favorite examples have been the field journals of Vernon Orlando Bailey, a field naturalist who journeyed throughout the U.S. midwest studying and collecting mammals. Another volunpeer favorite were the papers of Arctic explorer and naturalist Robert Kennicott. Read more about these volunpeer experiences on the STC blog.

The Power of the Smithsonian Transcription Center

When the first of the field books were added to Smithsonian Transcription Center (STC), the program was still available only as a beta version. Approximately 450 volunpeers were transcribing on the site (co-author Siobhan Leachman among them), with the first 6,000 completed pages under their belt. Even during this moment of immense energy and fresh connections, it must have been difficult to imagine what STC would become. Today, a little over a decade later, more than 91,000 individual volunpeers have worked together to transcribe and review over 1.4 MILLION (!) pages of historic and scientific collections.

STC is the largest digital volunteering and crowdsourcing program at the Smithsonian Institution, and provides opportunities to engage with and contribute to digitized materials from across the full breadth of content areas represented by its museums, archives, and libraries. Through collaborative transcription and review, Smithsonian staff and digital volunteers work together to ensure that this content is more readable, accessible, and text-searchable across Smithsonian data systems and beyond.

Currently, the most active and popular projects are the Freedmen’s Bureau Transcription Project, a collaboration with the National Museum of African American History and Culture that deepens insight into the Reconstruction period and empowers African American genealogical research, and Project PHaEDRA, a collaboration with the Harvard-Smithsonian Center for Astrophysics that illuminates the work and discoveries of early women computers at the Harvard College Observatory.

If you feel inspired by this data access success story, consider joining the digital volunteer community, or sign up for our newsletter to stay up-to-date on upcoming projects.

Reusing the Liberated Data

The successful integration of transcription materials into BHL makes historical scientific data more accessible and useful. Through the collaborative efforts of technical staff from the Biodiversity Heritage Library, Smithsonian Libraries and Archives, and the Smithsonian Transcription Center and dedicated volunpeers, over 151,362 scientific name access points were added to the BHL search index, greatly enhancing search capabilities over the digitized corpus of Smithsonian field notes and archives. Out of the 556 eligible items reviewed, 522 were uploaded, contributing 43,460 pages of improved OCR text. These contributions not only improve BHL’s full-text search and taxonomic name recognition services but also provide better accessibility for vision-impaired users, and support ongoing biodiversity and climate change research.

A special thanks goes to Mike Lichtenberg, BHL’s Lead Developer and Systems Architect and Paul Day, Lead Developer at Smithsonian Transcription Center. As with many data improvement and platform enhancement projects, the requisite technical expertise is pivotal in ensuring the success of our collective efforts!

References and Resources

Dearborn, J., Lichtenberg, M., Richard, J. M., deVeer, J., Trizna, M., & Mika, K. 2023. [Presentation] Unearthing the Past for a Sustainable Future: Extracting and Transforming Data in the Biodiversity Heritage Library for Climate Action. Presented virtually at TDWG, Tasmania, Australia 2023. https://www.youtube.com/watch?v=8sGssyrpuJw

Trizna, M., & Dearborn, J. June 2023. [Poster] AI Models Are Getting Better at Reading Handwriting, but How Can We Find Handwritten Text to Begin With?. 7th Annual Digital Data Conference, Leveraging Digital Data for Conservation, Ecology, Systematics, and Novel Biodiversity Research, Tempe, Arizona, United States of America. https://doi.org/10.25573/data.23523495.v1

Dearborn, J., & Mika, K. June 5, 2022. [Poster] Extracting Expedition Log Data Found in the Biodiversity Heritage Library. Through the Door and Through the Web: Releasing the Power of Natural History Collections Onsite and Online, Edinburgh, Scotland, United Kingdom: Society for the Preservation of Natural History Collections (SPNHC). https://doi.org/10.5281/zenodo.6593457

The Biodiversity Heritage Library (BHL) has added functionality to allow BHL Partners to upload transcriptions in place of the automatically-generated OCR (Optical Character Recognition) for archival materials digitized in BHL. This functionality supports transcriptions generated as part of Partner crowdsourcing projects on Smithsonian Transcription Center, DigiVol, and From the Page.

Optical Character Recognition (OCR), also called text recognition, translates text characters in scanned documents into code that can be used for data processing and enables searching of document text. Handwritten archival materials like correspondence and field notes are notoriously problematic for OCR software. Full-text searching of these materials is significantly hampered by poor OCR output.

Example of the poor automatically-generated OCR output for handwritten correspondence. Spencer Fullerton Baird and John Torrey correspondence, 1851-1860. Contributed in BHL from the LuEsther T. Mertz Library of The New York Botanical Garden.

Crowdsourcing the transcription of archival materials has become a popular way to generate machine-readable text that enables searching and discoverability. Several BHL Partners are using crowdsourcing platforms (e.g. Smithsonian Transcription Center, DigiVol, and From the Page) to transcribe field notes, correspondence, and other archival materials that they have digitized in BHL.

Example of a field notebook being transcribed in the Smithsonian Transcription Center. This notebook, Brasil 1979, Amazonia #3, from Cleofé Calderón is also available in BHL from the Smithsonian Institution Archives.

With this new functionality, these transcriptions can now be uploaded in place of the automatically-generated OCR for these items, allowing them to be full-text searchable and enabling our taxonomic name recognition software to index scientific names within their pages. Since the transcribed text can be viewed alongside the digitized page image, users can also more easily read materials with difficult-to-decipher handwriting. Thus, this new functionality makes it easier for researchers and the public to explore these valuable primary source materials and access specific information from their pages.

Above example with the OCR replaced with a crowdsourced transcription generated as part of The John Torrey Papers project from The New York Botanical Garden on From the Page. Spencer Fullerton Baird and John Torrey correspondence, 1851-1860. Contributed in BHL from the LuEsther T. Mertz Library of The New York Botanical Garden.

Since the transcribed text can be viewed alongside the digitized page image in the BHL book viewer, users can also more easily read archival materials. William Healey Dall’s Field Notes, 1871. Transcription generated on the Smithsonian Transcription Center. Contributed in BHL from the Smithsonian Institution Archives.

Crowdsourced transcriptions allow digitized archival materials in BHL to be full-text searchable, as shown in this example searching for “Yellow Palm Warbler” within William Brewster’s 1903 journal. Transcription generated as part of the Ernst Mayr Library of Harvard University project on DigiVol. Contributed in BHL from the Ernst Mayr Library of the Museum of Comparative Zoology at Harvard University.

Crowdsourced transcriptions allow BHL’s taxonomic name recognition software to index scientific names within the pages of digitized archival materials, as seen in this example in which Catharacta antarctica is indexed on a page within the first volume of the Ornithological Field Diaries of A. Graham Brown. Transcription generated as part of a project from BHL Australia and Museums Victoria on DigiVol. Contributed in BHL from Museums Victoria.

Participating Partners have begun uploading transcriptions to BHL. To date, transcriptions have been uploaded from Partner crowdsourcing projects with BHL Australia, Ernst Mayr Library of Harvard University, The New York Botanical Garden, and Smithsonian Institution Archives. This is an ongoing process, and more transcriptions will be uploaded to the Library over time.

Interested in transcribing archival materials? Several BHL Partners have active transcription projects on various crowdsourcing platforms. Follow the links below to explore the opportunities and get involved:

• Ernst Mayr Library of Harvard University on DigiVol
• The John Torrey Papers from The New York Botanical Garden on From the Page
• Smithsonian Institution Archives on the Smithsonian Transcription Center

L to R: Photo of Devra Kleiman by Hirshfeld, Max, 1978, Smithsonian Archives – History Div, 94-8340; Page from Kleiman field book (Brazil notebook, 1975-1982); Photo of Cleofé E. Calderón, Smithsonian Institution Archives, SIA Acc. 11-008, Box 02, Image No. OPA-90; Page from Calderón field book (Brasil 1972, 1).

Looking for a challenge? The Smithsonian Institution Archives and the Smithsonian Transcription Center are teaming up for a #HerNaturalHistory-themed transcription challenge. Starting today (8 March), help a team of #volunpeers transcribe field notes from conservation biologist Devra Kleiman and botanist Cleofé Calderón. Dive into observations of golden lion tamarins with Kleiman, who worked at Smithsonian’s National Zoological Park from 1972 to 2001. And head to South America with Calderón to collect grasses for Smithsonian’s Department of Botany.

When? Starting TODAY, 8 March 2019
Where? Smithsonian Transcription Center. The field notes are available on the Smithsonian Institution Archives project page.

The Smithsonian Institution Archives has launched 11 field books on the Smithsonian Transcription Center. Throughout the month, additional field books will be added as others are completed. Our goal is to fully transcribe 18 field books in total (9 from each woman) by the end of March.

These transcriptions will also make the corresponding digitized field books in the Biodiversity Heritage Library more discoverable, as they can be uploaded in place of the automatically-generated OCR text in BHL to allow the field books to be full-text searchable. This will make it easier for researchers and the public to explore these valuable primary source materials and access specific information from their pages.

We hope you’ll join our #HerNaturalHistory transcription challenge to help improve access to and discoverability of the primary research of two female naturalists. Continue reading to learn more about Kleiman and Calderón.

This challenge is presented as part of the #HerNaturalHistory campaign — an international social media campaign produced in collaboration with BHL partners to celebrate women in natural history. Visit the Her Natural History webpage to learn more about this and other events and programming occurring all month long.

Devra Kleiman

Giant Panda researcher Devra Kleiman records Hsing-Hsing’s vocalizations during breeding season, 1991, by Jessie Cohen, C-type print, National Zoo, Image ID: 614-95JC.

Conservation biologist Devra G. Kleiman (1942-2010) was the first female scientist at the National Zoological Park (NZP). She was hired by NZP in 1972 to manage the zoo’s captive breeding program. Much of her research focused on the captive breeding of giant pandas and the reintroduction of golden lion tamarins to the wild. Over her career at the zoo, she served as Assistant Director for Animal Programs, Assistant Director of Zoological Research and Education, Assistant Director of Research, Assistant Director of Zoological Research, and Senior Scientist. She authored approximately 150 manuscripts and two books.[1]

Smithsonian Institution Archives holds a large collection of archival materials documenting Kleiman’s research and professional activities. Many of her field books have been digitized in the Biodiversity Heritage Library as part of the Smithsonian Field Book Project.

Cleofé Calderón

Visiting Argentine botanist Cleo Calderón with plant specimens and microscope. 1965.
From 120mm black and white negative. Smithsonian Institution Archives, SIA Acc. 11-008, Box 02, Image No. OPA-90.

Botanist Cleofé Calderón (1929-2007), born in Buenos Aires, Argentina, studied with renowned agrostologist Prof. Ing. Lorenzo Parodi at the University of Buenos Aires. She was introduced to Dr. Thomas R. Soderstrom, Curator of Grasses in the Department of Botany at the U.S. National Herbarium at the Smithsonian, during a trip to Washington, D.C. in 1961-62. The two began a close collaboration, and Calderón became a prominent botanist specializing in bamboo. With support from the Smithsonian, the Office of Scientific Affairs of the Organization of American States, and the National Geographic Society, Calderón conducted extensive field work in Central and South America, collecting plant specimens in Panama, Costa Rica, Ecuador, Colombia, and Brazil between 1966 and 1982. One of her most significant finds was the “re-discovery” of Anomochloa marantoidea, a small species of bamboo last seen in the late 1840s.[2]

Smithsonian Institution Archives holds a collection of field books documenting Calderón’s botanical research in Brazil, Panama, and Colombia between 1967 and 1981. Many of these have been digitized in the Biodiversity Heritage Library as part of the Smithsonian Field Book Project.

References