Sixty-two million pages of scientific text, images, and metadata, representing 500 years of biodiversity data will be openly accessible via the Registry of Open Data on AWS

Washington, DC, November 27, 2024 — The BHL Technical Team is thrilled to announce that Biodiversity Heritage Library (BHL) datasets will be openly accessible on the Amazon Web Services (AWS) cloud, thanks to the AWS Open Data Sponsorship Program. Moving BHL data to the cloud allows researchers globally to explore and analyze over 500 years of biodiversity data, enhancing their ability to derive scientific insights from our shared past to inform future global environmental policy.

BHL data is now hosted on AWS, and comprises over 62 million pages of scientific text from the 15th to the 21st centuries. BHL’s vast collection represents an unparalleled biodiversity resource with enormous potential to be used for longitudinal studies and conservation efforts.

Key Use Cases for BHL’s Data:

• Historical Baseline Data: Spanning centuries, BHL provides a unique record of biodiversity literature, allowing researchers to establish long-term species baselines and track changes in species distributions and abundances over time.
• Longitudinal Studies: The dataset supports analysis of biodiversity trends and ecosystem responses to environmental changes, offering valuable insights into historical and contemporary biodiversity dynamics.
• Rare and Endangered Species: BHL’s historical records include species that may no longer exist or have become rare, aiding in the understanding of past biodiversity and informing current conservation efforts.
• Taxonomic Stability: The collection includes taxonomic descriptions and classifications from various time periods, essential for understanding species relationships.
• Cultural and Scientific Heritage: Beyond scientific data, BHL preserves historical texts, scientific illustrations, and annotations, enriching our understanding of past scientific practices and societal attitudes toward nature.

Transitioning to AWS services represents a transformative opportunity for BHL. As BHL currently operates outside a cloud environment, this move will optimize scalability, accessibility, performance, and cost-efficiency. AWS’s advanced computing capabilities will enable faster data processing and analysis, while AWS’s cost management tools will in the long-term help reduce infrastructure costs associated with on-premise storage and upgrades.

As a vital component of the global biodata infrastructure, the demand for BHL’s content and services continues to rise. By leveraging AWS services, we can address several critical needs, including:

• Complimentary storage and access to a comprehensive suite of AWS cloud computing tools.
• Ensured page persistence and unique page identifiers essential for biodiversity informatics applications.
• Enhanced page-serving speeds in low-bandwidth regions where biodiversity research is critical.

BHL’s Data Manager, JJ Dearborn remarked, “Collaborating with AWS unlocks the full potential of BHL data. Cloud hosting not only provides an additional data safe harbor but also makes it accessible to researchers worldwide. We’re excited to democratize access to BHL data and see it used to spark innovative cross-disciplinary research in biodiversity informatics.”

The AWS Open Data Sponsorship Program is committed to making high-value datasets freely available to encourage innovation and collaboration. By supporting the free access to a diverse array of big datasets, AWS helps advance research and development in numerous fields.

To explore the BHL dataset and learn more about its content, visit the AWS Data Exchange.

About the Biodiversity Heritage Library

The Biodiversity Heritage Library (BHL) is the world’s largest open access digital library for biodiversity literature and archives. Headquartered at the Smithsonian Libraries and Archives in Washington, D.C., BHL is a global consortium that digitizes and freely shares natural history literature and archives. By providing open access to critical biodiversity knowledge, BHL addresses major research challenges and supports the global scientific community in understanding and conserving Earth’s species amid climate change and extinction crises. BHL also collaborates with rights holders, volunteers, and international experts to enhance content accessibility and interoperability. Since its inception in 2006, BHL has been working towards a shared future where biodiversity knowledge is globally accessible and supports universal bioliteracy.

Download Press Release

The BHL API will be updated on 25 July 2016 to support changes to the BHL site. These changes will accommodate identifying additional Contributors for Items and Parts of items.

First are changes to the API that may affect your existing processes.

The Contributor and ContributorID elements in the result sets of API methods that return “Part” information will move. ContributorID will be included as a PartIdentifier in the Identifiers list. Contributor will be included in a new Contributors list.

These changes are being made to accommodate more than one contributor per part.  For example, if one institution researches/compiles the data and a second institution facilitates the inclusion of that data in BHL, both institutions may be listed as a contributor.  Initially, no more than two contributors per part will be allowed, but by adopting these changes to API responses we allow for additional (unlimited, actually) contributors in the future.

Here is an example that shows how the API responses are changing.  The examples shown here are an output of the GetPartMetadata method, and have been abbreviated for clarity.

Original API Results – highlighted elements are being moved:

<Response>
 <Status>okStatus>
 <Result>
 <PartUrl>
        http://www.biodiversitylibrary.org/part/1
 PartUrl>
 <PartID>1PartID>
 <ItemID>22498ItemID>
 <StartPageID>3190776StartPageID>
 <SequenceOrder>1SequenceOrder>
 <Contributor>BioStorContributor>
 <ContributorID>4443ContributorID>
 <GenreName>ArticleGenreName>
 <Title>
      Notes on certain species of Tetragnatha 
      (Araneae, Argiopidae) in Central America 
      and Mexico
 Title>
 <ContainerTitle>BrevioraContainerTitle>
 <Volume>67Volume>
 <Date>1957Date>
 <PageRange>1--4PageRange>
 <StartPageNumber>1StartPageNumber>
 <EndPageNumber>4EndPageNumber>
 <Authors> [...] Authors>
 <Subjects />
 <Identifiers>
 <PartIdentifier>
 <IdentifierName>ISSNIdentifierName>
 <IdentifierValue>0006-9698IdentifierValue>
 PartIdentifier>
 Identifiers>
 <Pages> [...] Pages>
 <RelatedParts />
 Result>
Response>

Updated API Results – Highlighted elements are the new locations of the moved data:

<Response>
 <Status>okStatus>
 <Result>
 <PartUrl>
        http://www.biodiversitylibrary.org/part/969
 PartUrl>
 <PartID>1PartID>
 <ItemID>22498ItemID>
 <StartPageID>3190776StartPageID>
 <SequenceOrder>1SequenceOrder>
 <GenreName>ArticleGenreName>
 <Title>
        Notes on certain species of Tetragnatha
        (Araneae, Argiopidae) in Central America
        and Mexico
 Title>
 <ContainerTitle>BrevioraContainerTitle>
 <Volume>67Volume>
 <Date>1957Date>
 <PageRange>1--4PageRange>
 <StartPageNumber>1StartPageNumber>
 <EndPageNumber>4EndPageNumber>
 <Authors> [...] Authors>

 <Contributors>
 <Contributor>
 <ContributorName>BioStorContributorName>
 Contributor>
 Contributors>
 <Subjects />
 <Identifiers>
 <PartIdentifier>
 <IdentifierName>BioStorIdentifierName>
 <IdentifierValue>4443IdentifierValue>
 PartIdentifier>
 <PartIdentifier>
 <IdentifierName>ISSNIdentifierName>
 <IdentifierValue>0006-9698IdentifierValue>
 PartIdentifier>
 Identifiers>
 <Pages> [...] Pages>

 <RelatedParts />
 Result>
Response>

Additionally, there will be two additions to the Item metadata.

The new data elements are: RightsHolder and ScanningInstitution.
These will optionally be displayed if there is data for the relevant organizations.

<Response>
 <Status>okStatus>
 <Result>
 <ItemID>59382ItemID>
 <PrimaryTitleID>20770PrimaryTitleID>
 <ThumbnailPageID>17605914ThumbnailPageID>
 <Source>Internet ArchiveSource>
 <SourceIdentifier>bulletin5160hatcSourceIdentifier>
 <Volume>v.51-60 1898-99Volume>
 <Year/>
 <Contributor>
      UMass Amherst Libraries (archive.org)
 Contributor>
 <RightsHolder>
 Biodiversity Heritage Library
 RightsHolder>
 <ScanningInstitution>
 Smithsonian Institution Libraries
 ScanningInstitution>
 <Sponsor>UMass Amherst LibrariesSponsor>
    [...]
 Result>
Response>

Detailed documentation for the BHL APIs is available at http://www.biodiversitylibrary.org/api2/docs/docs.html.  It will be updated to reflect these changes after they are moved into production on 25 July 2016.

Go to http://www.biodiversitylibrary.org/getapikey.aspx to get an API key for BHL.

Information about other BHL developer tools can be found at http://biodivlib.wikispaces.com/Developer+Tools+and+API.

If you have questions, please feel free to submit feedback via this form.

The Biodiversity Heritage Library has joined the Global Biodiversity Information Facility (GBIF) as an Associate Participant. GBIF operates through a network of global nodes to develop and maintain an open data infrastructure for sharing digital biodiversity data. As an Associate Participant, BHL will encourage open access and use of biodiversity data among its stakeholders and actively participate in the implementation of the GBIF Work Programme.

“BHL provides open and free access to over 250 years of biodiversity information via web services and open APIs,” said Martin Kalfatovic, BHL Program Director. “As a growing global consortium of biodiversity-related libraries and through cooperation with other biodiversity institutions, we are constantly working to refine BHL content and services to address the need for biodiversity literature. Through participation in GBIF, BHL will be able to share our own expertise, gain new competencies, and collaborate with new communities to ensure that the breadth of biodiversity knowledge is available to everyone, everywhere.”

GBIF participants include countries, intergovernmental and international organizations, and organizations with an international scope that seek to share data under common standards, inform and implement the GBIF strategic plan, and invest in tools, services, and capacity building within biodiversity information frameworks. To date, over 90 participants have signed the GBIF MOU.

About GBIF
The purpose of GBIF is to promote, co-ordinate, design, enable and implement the compilation, linking, standardization, digitization and global dissemination and use of the world’s biodiversity data, within an appropriate framework for property rights and due attribution. GBIF works in close co-operate with established programmes and organisations that compile, maintain, and use biological information resources. The Participants, working through GBIF, establish and support a distributed information system that enables users to access and utilize considerable quantities of existing and new biodiversity data. Learn more: http://www.gbif.org/.