All posts in Tech Updates

Building a More Resilient BHL: Improving Accessibility and Expanding Global Reach

What does it take to make millions of pages of biodiversity literature accessible to a global audience? For the Biodiversity Heritage Library (BHL) Technical Team, 2024 was a year of transformative milestones, new innovations, and overcoming challenges—all aimed at strengthening BHL’s mission of advancing biodiversity research.

Sixty-two million pages of scientific text, images, and metadata, representing 500 years of biodiversity data will be openly accessible via the Registry of Open Data on AWS

The BHL Technical Team is thrilled to announce that Biodiversity Heritage Library (BHL) datasets will be openly accessible on the Amazon Web Services (AWS) cloud, thanks to the AWS Open Data Sponsorship Program. Moving BHL data to the cloud allows researchers globally to explore and analyze over 500 years of biodiversity data, enhancing their ability to derive scientific insights from our shared past to inform future global environmental policy.

BHL data is now hosted on AWS, and comprises over 62 million pages of scientific text from the 15th to the 21st centuries. BHL’s vast collection represents an unparalleled biodiversity resource with enormous potential to be used for longitudinal studies and conservation efforts.

In 2023, BHL’s Technical Team dedicated significant efforts to improve our data ecosystem, now comprising 61+ million pages of biodiversity literature. Last year’s Technical Priorities underscored BHL’s steadfast commitment to data quality by focusing on both upstream and downstream data flows. Notable milestones include delivering refined taxonomic data to researchers, implementing interface improvements based on user feedback, and forging data pipelines for existing and new downstream data consumers. 

The planet is at a critical juncture, where urgent action is required to combat climate change, biodiversity loss, and secure a sustainable future for our planet. With the release of the recent BHL Wikimedia white paper entitled Unifying Biodiversity Knowledge to Support Life on a Sustainable Planet, the Biodiversity Heritage Library Secretariat hopes that an expanded data vision for the BHL community presents new opportunities to take bold steps forward into Wikimedia projects and the emergent semantic web. The white paper sheds light on BHL’s crucial role as a member of the biodiversity informatics community and reveals a series of key use cases and big data challenges that, if addressed, could be opportunities to enhance global biodiversity data infrastructure.

In the Spring of 2022, the BHL Cataloging and Metadata Committee investigated the possibility of harvesting persistent identifiers (PIDs) from Wikidata as part of the group’s longstanding project to disambiguate and deduplicate author records in the BHL database. The motivation behind this one-time experimental data harvest was to see if BHL could:

1. Enhance BHL author records with additional PID data points;
2. Improve the committee’s ability to disambiguate author names in the BHL database; and
3. Respond to an outstanding user request from two of Wikimedia’s super star editors, Siobhan Leachman and Andy Mabbett, to expose BHL’s author data on BHL and include hyperlinks to other authoritative knowledge bases on the web.

In particular, Wikimedians wanted to see the Wikidata Q identifier exposed, providing a link to the corresponding creator item record in Wikidata.

There are multiple motivations for undertaking this work. By adding the BHL Creator ID to the corresponding Wikidata item, Wikidata editors help link BHL to the richer biographical data about that person held in Wikidata. The Wikidata item for a person may contain links to their Wikipedia page or to images of the person held in the image repository Wikimedia Commons. Wikidata items also act as identifier hubs and contain links to other databases and identifiers.

By adding the BHL Creator ID to this list of identifiers, the Wikidata editor is linking the content held in BHL to the content held in multiple other datasets and repositories.

These extra author data points provide Wikimedians and BHL catalogers with crucial clues that aid in name disambiguation. In particular, hyperlinks to other knowledge bases are incredibly valuable because they lead to new knowledge pathways that help confirm a person’s identity in a complex game of “Who’s Who?”

Recently, BHL performed a comprehensive review of all live data feeds and outputs to ensure that we are providing robust metadata to our downstream consumers. Live BHL data can be found at BHL’s Developer and Data Tools. BHL’s live data outputs include:

• API v3
• OAI-PMH

OAI-PMH is an acronym for the Open Archives Initiative Protocol for Metadata Harvesting. It allows other discovery services and aggregators to harvest BHL’s metadata in standard formats such as Metadata Object Description Schema (MODS) and Dublin Core (DC).

To provide more robust data in BHL’s OAI-PMH Dublin Core feed, three changes have been made to the feed:

1. Creative Commons (CC) license information was added as a second <rights> element;
2. A <relation> element was added to titles that are part of a monographic series, allowing BHL to model more complex bibliographic relationships that exist in the BHL database; and
3. A non-standard “type” attribute was removed from the <relation> element for parts.

Important: If you are a developer, using the non-standard “type” attribute in your code at the part-level, this is a breaking change. Please take note and update your code accordingly.

For BHL, 2022 was a year to focus on critical upgrades for the BHL platform to ensure the sustainability of our services for our global users. Although BHL’s basic technical infrastructure remains the same, consisting of years of refinement, knowledge, and reliability, a few updates were definitely in order. Most of these upgrades were “behind-the-scenes” work and would not be noticeable to a majority of our users. However, keeping up with these important enhancements is a crucial component of any technology project.