Sixty-two million pages of scientific text, images, and metadata, representing 500 years of biodiversity data will be openly accessible via the Registry of Open Data on AWS

Washington, DC, November 27, 2024 — The BHL Technical Team is thrilled to announce that Biodiversity Heritage Library (BHL) datasets will be openly accessible on the Amazon Web Services (AWS) cloud, thanks to the AWS Open Data Sponsorship Program. Moving BHL data to the cloud allows researchers globally to explore and analyze over 500 years of biodiversity data, enhancing their ability to derive scientific insights from our shared past to inform future global environmental policy.

BHL data is now hosted on AWS, and comprises over 62 million pages of scientific text from the 15th to the 21st centuries. BHL’s vast collection represents an unparalleled biodiversity resource with enormous potential to be used for longitudinal studies and conservation efforts.

Key Use Cases for BHL’s Data:

• Historical Baseline Data: Spanning centuries, BHL provides a unique record of biodiversity literature, allowing researchers to establish long-term species baselines and track changes in species distributions and abundances over time.
• Longitudinal Studies: The dataset supports analysis of biodiversity trends and ecosystem responses to environmental changes, offering valuable insights into historical and contemporary biodiversity dynamics.
• Rare and Endangered Species: BHL’s historical records include species that may no longer exist or have become rare, aiding in the understanding of past biodiversity and informing current conservation efforts.
• Taxonomic Stability: The collection includes taxonomic descriptions and classifications from various time periods, essential for understanding species relationships.
• Cultural and Scientific Heritage: Beyond scientific data, BHL preserves historical texts, scientific illustrations, and annotations, enriching our understanding of past scientific practices and societal attitudes toward nature.

Transitioning to AWS services represents a transformative opportunity for BHL. As BHL currently operates outside a cloud environment, this move will optimize scalability, accessibility, performance, and cost-efficiency. AWS’s advanced computing capabilities will enable faster data processing and analysis, while AWS’s cost management tools will in the long-term help reduce infrastructure costs associated with on-premise storage and upgrades.

As a vital component of the global biodata infrastructure, the demand for BHL’s content and services continues to rise. By leveraging AWS services, we can address several critical needs, including:

• Complimentary storage and access to a comprehensive suite of AWS cloud computing tools.
• Ensured page persistence and unique page identifiers essential for biodiversity informatics applications.
• Enhanced page-serving speeds in low-bandwidth regions where biodiversity research is critical.

BHL’s Data Manager, JJ Dearborn remarked, “Collaborating with AWS unlocks the full potential of BHL data. Cloud hosting not only provides an additional data safe harbor but also makes it accessible to researchers worldwide. We’re excited to democratize access to BHL data and see it used to spark innovative cross-disciplinary research in biodiversity informatics.”

The AWS Open Data Sponsorship Program is committed to making high-value datasets freely available to encourage innovation and collaboration. By supporting the free access to a diverse array of big datasets, AWS helps advance research and development in numerous fields.

To explore the BHL dataset and learn more about its content, visit the AWS Data Exchange.

About the Biodiversity Heritage Library

The Biodiversity Heritage Library (BHL) is the world’s largest open access digital library for biodiversity literature and archives. Headquartered at the Smithsonian Libraries and Archives in Washington, D.C., BHL is a global consortium that digitizes and freely shares natural history literature and archives. By providing open access to critical biodiversity knowledge, BHL addresses major research challenges and supports the global scientific community in understanding and conserving Earth’s species amid climate change and extinction crises. BHL also collaborates with rights holders, volunteers, and international experts to enhance content accessibility and interoperability. Since its inception in 2006, BHL has been working towards a shared future where biodiversity knowledge is globally accessible and supports universal bioliteracy.

Download Press Release

Ten years ago in June 2010, the Atlas of Living Australia and Museums Victoria signed an agreement with the Biodiversity Heritage Library – and BHL Australia was born.

Dr. Mark Lonsdale (left), the then Chief of the Commonwealth Scientific and Industrial Research Organisation’s (CSIRO’s) Entomology Division, and Martin Kalfatovic (right), BHL Program Director, signing the Relationship Agreement between the Atlas of Living Australia, Museums Victoria and BHL. Image sourced from Martin Kalfatovic.

BHL Australia’s mission is to make Australia’s biodiversity literature freely accessible and discoverable. Ten years ago, we started with a single contributing organisation, Museums Victoria, and a team of five incredibly dedicated volunteers.

Over the past 10 years, BHL Australia has grown considerably. Our operation is still hosted by Museums Victoria (at the Melbourne Museum), but we now digitise literature (and ingest born-digital material) on behalf of 27 organisations across the country.

We are now a truly national project, representing Australia’s state and territory museums, herbaria, royal societies and field naturalists clubs, as well as government agencies and natural history publishers. Together these organisations have contributed more than 350,000 pages from over 2,400 volumes.

Dr. Elycia Wallis, Project Lead for BHL Australia at the Atlas of Living Australia, with our Zeutschel 16000 Book Scanner, purchased in 2017. Photo Credit: Nicole Kearney.

These volumes include treasures such as George Shaw’s The Naturalist’s Miscellany (1789-1813), Helena Forde and Harriet Scott’s Australian lepidoptera and their transformations, drawn from the life (1890-1898) and John Gould’s A synopsis of the birds of Australia, and the adjacent Islands (1837).

BHL Australia has also uploaded an extensive list of journals onto BHL (they may not be as pretty, but they’re just as important). To peruse all 2,400+ volumes, see our full BHL Australia Collection.

The Naturalist’s Miscellany includes the first published scientific description and illustration of the Duck-billed Platypus (Ornithorhynchus anatinus). Shaw, George. The Naturalist’s Miscellany. Volume 10. 1799. Contributed to  BHL by Museums Victoria. DOI: https://doi.org/10.5962/p.304567. 

Our volunteer team has also grown since 2010; BHL Australia now has 15 amazing volunteers who do the majority of our scanning, cropping, image processing and metadata addition work, as well as three science communication volunteers. (You may have seen their hugely successful takeover of the BHL Instagram account during #BirdWeek last year.)

BHL Australia volunteer, Tiziana Tizian, cropping page images in preparation for upload onto BHL. Photo Credit: Nicole Kearney.

Of course, like so many other digitisation operations around the globe, BHL Australia is now in lockdown (and has been since mid-March). But out of adversity comes opportunity. While in lockdown, we’ve switched our focus from physical to born-digital material. We’ve welcomed new contributors and have uploaded journal volumes published as recently as 2020. We’ve also started a major project (in collaboration with BHL superuser Rod Page) to upload article metadata for every Australian journal on BHL (we’re pretty obsessed with discoverability).

The Records of the Western Australian Museum is now complete on BHL from 1910 to 2019 as a result of our efforts (during our COVID-19 lockdown) to upload born-digital material from Australia’s journals.

And, like so many others, we’ve also had to postpone our celebrations. We had planned to invite all our Australian contributors and volunteers to a big BHL birthday bash. That’s on hold for now, but in the meantime, here are our BHL staff — Cerise, Chris, Veronica and myself — waving our thanks to all those who support BHL Australia. To our volunteers, our contributors, the Atlas of Living Australia and our BHL community around the world — thank you for a wonderful 10 years!

The BHL Australia team: Manager Nicole Kearney (top right); Digitisation Coordinators Veronica Scholes (top left) and Cerise Howard (bottom right); and Technician Chris Healey (bottom left).

Illustration of a Cactus Dahlia from Revue horticole (1908). Contributed in BHL from the New York Botanical Garden, LuEsther T. Mertz Library. This image is one of over 160,000 that are freely available for download and reuse through the BHL Flickr.

BHL’s digital library provides free and open online access to over 250,000 volumes from the 15th-21st centuries on a wide range of biodiversity subjects. These collections offer great resources to support distance learning.

All of our content can be accessed online for free and in full through our digital library portal at biodiversitylibrary.org. No login, account, or membership is required. You can also freely download anything in our library in a variety of formats. Our FAQ provides information on how to search the library and download content.

In addition to our general collections, we’ve identified a selection of materials that may be of particular interest to educators and students. Explore these resources below.

Thematic Collections

Looking for curated content? Our Collections offer a pre-selected set of materials on a variety of subjects, from Antarctic Exploration & Discovery to Seed & Nursery Catalogs and Latino Natural History.

Explore over 50 thematic collections on a range of subjects.

Image Resources

If you need image resources for your distance learning activities, we have a rich collection of over 160,000 free nature images in our Flickr. Here’s how to search and download those collections.

Instructions on how to search for specific subjects and species in the BHL Flickr.

Looking for an educational activity to do online around nature art? Tagging images in the BHL Flickr with species and artist names is a great option. It’s not only a fun way to learn about taxonomy, investigate individual species, and explore artists and artistic practices, but it also makes these images more accessible and discoverable for others to support their research and educational activities.

Learn more about tagging and get started with our instructional guide.

Coloring Books

Looking for more hands-on educational resources? Explore and download our free coloring books created from images in our collection.

Our “Female Illustrators in Natural History” coloring book is one of several freely available from BHL. Explore and download our coloring books.

Women’s History Month Educational Resources

Looking for resources to support educational activities around Women’s History Month? Our #HerNaturalHistory collections offer a range of content related to women’s contributions to natural history. Explore our book and image collections, coloring book, Facebook videos, and blog series.

Our Her Natural History collections on women in science include book and image collections, a coloring book, Facebook videos, and informative articles.

Earlier this week, Rod Page and I received an email from Richard Orr, the Lead Developer at Unpaywall, telling us that he had created a work-around that would finally enable the Unpaywall extension to discover content in BHL. And I (Nicole) literally spent the rest of the day jumping for joy. 

Let us explain: 

Firstly, what’s a DOI?

DOIs (Digital Object Identifiers) are used throughout the scholarly research community to uniquely identify academic articles. They help readers locate the definitive version of a published article, and they make linking together the academic literature much easier – look at any recent paper and you’ll see that most of the references cited have DOIs.

DOIs do two things: 1) they uniquely identify an article, and 2) they point to the online location of the definitive version of that article (typically hosted by the article’s publisher). 

Many articles are not free to read: a significant proportion of both recently-published and historic articles are locked behind paywalls. However, with the rise of open access, it’s increasingly the case that there may be a free version of an article available somewhere online. BHL, for example, has scanned and made available tens of thousands of articles that also exist on commercial publishers’ websites. DOIs always direct users to the definitive version of an article. But if the definitive version is behind a paywall (as they so often are), how do we tell users that BHL has a free version available? Enter Unpaywall.

What’s Unpaywall?

Unpaywall finds (legally) open access versions of paywalled literature. Since its launch in 2016, Unpaywall has become an indispensable tool for scientists  (see “How Unpaywall is transforming open science”). Unpaywall’s free browser extension (downloadable via their website) displays a discrete padlock symbol on the side of your browser whenever you are on a paywalled paper. If Unpaywall is able to locate a freely-accessible copy of the article elsewhere, the padlock symbol appears green and clicking on it will take you directly to the open access version. To discover whether an article is free, Unpaywall scans a database of millions of articles compiled from over 50,000 sources. Until this week, BHL wasn’t one of them. 

Why couldn’t Unpaywall link to BHL?

BHL contains hundreds of thousands of journal articles. More than a quarter of a million of these articles have been indexed (the vast majority by Rod Page via Biostor), which means that they now have article-level landing pages containing their article-level metadata. Tens of thousands of these article landing pages now have DOIs. This should have made them discoverable, but Unpaywall still couldn’t find them. 

In June, we contacted Unpaywall to find out why. It turns out that the reason BHL content has never been picked up by Unpaywall is because of the way BHL uploads and presents its journal content. Most providers of online journals present each article neatly packaged as an individual PDF. BHL, however, is first and foremost a virtual library. We upload complete volumes of journals made up of individual page images. Our article landing pages don’t link to PDFs; they link to the page in the volume upon which the article starts. Unpaywall looks for a PDF link to confirm that the document is actually available. Thus, the 57 million pages of open access content on BHL was excluded from Unpaywall’s database.  

After we explained to Unpaywall how significant BHL’s content was, Richard Orr, Unpaywall’s Lead Developer, very kindly agreed to create a work-around, specifically for BHL, that would enable Unpaywall to link to BHL article landing pages. The result of this work-around is that (as of this week) 43,000 journal articles on the BHL website are now discoverable via Unpaywall. 

To demonstrate how useful this is, here are two examples:

The first description of the iconic (sadly now-extinct) Thylacine (or Tasmanian Tiger) was published in the Transactions of the Linnean Society of London in 1808. This article is well and truly out of copyright and yet the definitive version of this article is behind a paywall on the Wiley Online website: https://doi.org/10.1111/j.1096-3642.1818.tb00336.x. Downloading the PDF of this out-of-copyright article from the Wiley website will cost you $42 USD.

The definitive DOI version of this article is behind a paywall on the Wiley Online website, but it is freely available on the BHL website. With the Unpaywall extension, you can now easily navigate to the free version on BHL.

Now that BHL’s content is discoverable via Unpaywall, anyone directed to the Wiley version via the article’s DOI can discover the free version on BHL (via the Unpaywall extension).

Article freely available via the Biodiversity Heritage Library.

It’s important to note that making BHL discoverable via Unpaywall doesn’t just enhance access to legacy literature such as the Thylacine paper; it also applies to much more recent research. For example, “The generic relationships of the new endemic Australian ant spider genus Notasteron (Araneae, Zodariidae)” was published in The Journal of Arachnology in 2005. This article has the DOI (https://doi.org/10.1636/04-56.1) and is behind a paywall on BioOne. With Unpaywall’s extension in your browser you can now discover the free version hosted by BHL.

How to make even more BHL content discoverable via Unpaywall

43,000 may seem like a large number, but it’s actually only a tiny fraction of the articles freely available on BHL. For the Unpaywall extension to be able to locate all the journal content on BHL, we need to unlock the rest of the journal articles on BHL by 1) adding more article-level metadata, and 2) ensuring that, for every article on BHL that has an existing DOI, we include that DOI in the article-level metadata. That’s our next task…

We would like to thank Unpaywall for providing access to an ever-increasing number of open access scholarly articles (23,943,966 at 14/8/19) and to particularly thank their Lead Developer, Richard Orr, for making it possible for BHL (a square peg) to fit into their open database (a round hole).