The BHL Tech Team is pleased to announce a new form of content available in BHL: Article PDFs. While this may not sound like anything new, after all, we have had a tool to download PDF content for some time, this update changes both how the PDFs are created and maintained, and how BHL is viewed by content aggregators on the internet, most notably Unpaywall.

The new Download PDF icon

How to use it? While browsing an article, you will now see a Download PDF icon below the View Article link on the right side of the page. Clicking the link will immediately download the PDF to your computer (or view it in your web browser, depending on your settings.)

The benefits of the immediate download are:

• No waiting.
• No selecting pages.
• The PDF contains embedded, searchable, copy-paste-able text.†
• The PDF contains rich XMP-based metadata about the article.

An important change to note is that when viewing an article within an item at BHL, the Download Contents > Download Article link will now direct the visitor’s browser to the new PDFs for immediate download. This is a departure from what we had before in that the pages of the article were pre-selected for download and the visitor was then required to complete the process and wait for the PDF to be generated. We expect the new PDFs to be an improvement for our visitors who come to download articles. View the How do I download a PDF of an article? FAQ for simple download instructions.

Visitors to BHL are still able to manually create PDFs using the Download Contents > Select Pages to Download feature. This feature has not been removed, but it still means that it takes some time to create those PDFs and email the person when the PDF is ready. This option is useful for articles that have not been indexed in BHL, and therefore do not have a Download Article link. View the How do I generate a custom PDF of selected pages from the book? FAQ for complete instructions.

The most important feature of the new Article PDFs is the embedded text† within the document. The select-able text is an invisible text layer in the PDF, but it appears when you select or search for text within the document:

An example of select-able text in an Article PDF.

While the appearance of the text may look… less than ideal, rest assured that the text can be copied out intact and used in another program. Example:

It is perhaps needless for me here to reiterate the great importance
of arriving at a final decision as to the real nature of
the haloliranic forms, for it will be obvious that if they have
nothing to do with the normal fresh-water series, and are to
be regarded as the remnant of an ancient sea, our views
respecting the past history of the African interior must be
greatly changed.

Other, less visible benefits to the PDFs are that they are directly linked from the citation_pdf_url meta-tag on the web page which makes them more findable by Google Scholar, Unpaywall, and potentially other aggregators.

For the technical-minded, the PDFs (many tens of thousands of them) are created in advance and stored on BHL’s servers. Changes to data within BHL will cause the PDF to be updated automatically, usually within several hours.

We hope that this is a welcome addition to BHL.

† – Please note that the text is only as good as the OCR that was generated for the text on the page. While the OCR text is probably very good for the prose sections of an article, titles, tables, and other special content may not appear as expected.

Ten years ago in June 2010, the Atlas of Living Australia and Museums Victoria signed an agreement with the Biodiversity Heritage Library – and BHL Australia was born.

Dr. Mark Lonsdale (left), the then Chief of the Commonwealth Scientific and Industrial Research Organisation’s (CSIRO’s) Entomology Division, and Martin Kalfatovic (right), BHL Program Director, signing the Relationship Agreement between the Atlas of Living Australia, Museums Victoria and BHL. Image sourced from Martin Kalfatovic.

BHL Australia’s mission is to make Australia’s biodiversity literature freely accessible and discoverable. Ten years ago, we started with a single contributing organisation, Museums Victoria, and a team of five incredibly dedicated volunteers.

Over the past 10 years, BHL Australia has grown considerably. Our operation is still hosted by Museums Victoria (at the Melbourne Museum), but we now digitise literature (and ingest born-digital material) on behalf of 27 organisations across the country.

We are now a truly national project, representing Australia’s state and territory museums, herbaria, royal societies and field naturalists clubs, as well as government agencies and natural history publishers. Together these organisations have contributed more than 350,000 pages from over 2,400 volumes.

Dr. Elycia Wallis, Project Lead for BHL Australia at the Atlas of Living Australia, with our Zeutschel 16000 Book Scanner, purchased in 2017. Photo Credit: Nicole Kearney.

These volumes include treasures such as George Shaw’s The Naturalist’s Miscellany (1789-1813), Helena Forde and Harriet Scott’s Australian lepidoptera and their transformations, drawn from the life (1890-1898) and John Gould’s A synopsis of the birds of Australia, and the adjacent Islands (1837).

BHL Australia has also uploaded an extensive list of journals onto BHL (they may not be as pretty, but they’re just as important). To peruse all 2,400+ volumes, see our full BHL Australia Collection.

The Naturalist’s Miscellany includes the first published scientific description and illustration of the Duck-billed Platypus (Ornithorhynchus anatinus). Shaw, George. The Naturalist’s Miscellany. Volume 10. 1799. Contributed to  BHL by Museums Victoria. DOI: https://doi.org/10.5962/p.304567. 

Our volunteer team has also grown since 2010; BHL Australia now has 15 amazing volunteers who do the majority of our scanning, cropping, image processing and metadata addition work, as well as three science communication volunteers. (You may have seen their hugely successful takeover of the BHL Instagram account during #BirdWeek last year.)

BHL Australia volunteer, Tiziana Tizian, cropping page images in preparation for upload onto BHL. Photo Credit: Nicole Kearney.

Of course, like so many other digitisation operations around the globe, BHL Australia is now in lockdown (and has been since mid-March). But out of adversity comes opportunity. While in lockdown, we’ve switched our focus from physical to born-digital material. We’ve welcomed new contributors and have uploaded journal volumes published as recently as 2020. We’ve also started a major project (in collaboration with BHL superuser Rod Page) to upload article metadata for every Australian journal on BHL (we’re pretty obsessed with discoverability).

The Records of the Western Australian Museum is now complete on BHL from 1910 to 2019 as a result of our efforts (during our COVID-19 lockdown) to upload born-digital material from Australia’s journals.

And, like so many others, we’ve also had to postpone our celebrations. We had planned to invite all our Australian contributors and volunteers to a big BHL birthday bash. That’s on hold for now, but in the meantime, here are our BHL staff — Cerise, Chris, Veronica and myself — waving our thanks to all those who support BHL Australia. To our volunteers, our contributors, the Atlas of Living Australia and our BHL community around the world — thank you for a wonderful 10 years!

The BHL Australia team: Manager Nicole Kearney (top right); Digitisation Coordinators Veronica Scholes (top left) and Cerise Howard (bottom right); and Technician Chris Healey (bottom left).

Earlier this week, Rod Page and I received an email from Richard Orr, the Lead Developer at Unpaywall, telling us that he had created a work-around that would finally enable the Unpaywall extension to discover content in BHL. And I (Nicole) literally spent the rest of the day jumping for joy. 

Let us explain: 

Firstly, what’s a DOI?

DOIs (Digital Object Identifiers) are used throughout the scholarly research community to uniquely identify academic articles. They help readers locate the definitive version of a published article, and they make linking together the academic literature much easier – look at any recent paper and you’ll see that most of the references cited have DOIs.

DOIs do two things: 1) they uniquely identify an article, and 2) they point to the online location of the definitive version of that article (typically hosted by the article’s publisher). 

Many articles are not free to read: a significant proportion of both recently-published and historic articles are locked behind paywalls. However, with the rise of open access, it’s increasingly the case that there may be a free version of an article available somewhere online. BHL, for example, has scanned and made available tens of thousands of articles that also exist on commercial publishers’ websites. DOIs always direct users to the definitive version of an article. But if the definitive version is behind a paywall (as they so often are), how do we tell users that BHL has a free version available? Enter Unpaywall.

What’s Unpaywall?

Unpaywall finds (legally) open access versions of paywalled literature. Since its launch in 2016, Unpaywall has become an indispensable tool for scientists  (see “How Unpaywall is transforming open science”). Unpaywall’s free browser extension (downloadable via their website) displays a discrete padlock symbol on the side of your browser whenever you are on a paywalled paper. If Unpaywall is able to locate a freely-accessible copy of the article elsewhere, the padlock symbol appears green and clicking on it will take you directly to the open access version. To discover whether an article is free, Unpaywall scans a database of millions of articles compiled from over 50,000 sources. Until this week, BHL wasn’t one of them. 

Why couldn’t Unpaywall link to BHL?

BHL contains hundreds of thousands of journal articles. More than a quarter of a million of these articles have been indexed (the vast majority by Rod Page via Biostor), which means that they now have article-level landing pages containing their article-level metadata. Tens of thousands of these article landing pages now have DOIs. This should have made them discoverable, but Unpaywall still couldn’t find them. 

In June, we contacted Unpaywall to find out why. It turns out that the reason BHL content has never been picked up by Unpaywall is because of the way BHL uploads and presents its journal content. Most providers of online journals present each article neatly packaged as an individual PDF. BHL, however, is first and foremost a virtual library. We upload complete volumes of journals made up of individual page images. Our article landing pages don’t link to PDFs; they link to the page in the volume upon which the article starts. Unpaywall looks for a PDF link to confirm that the document is actually available. Thus, the 57 million pages of open access content on BHL was excluded from Unpaywall’s database.  

After we explained to Unpaywall how significant BHL’s content was, Richard Orr, Unpaywall’s Lead Developer, very kindly agreed to create a work-around, specifically for BHL, that would enable Unpaywall to link to BHL article landing pages. The result of this work-around is that (as of this week) 43,000 journal articles on the BHL website are now discoverable via Unpaywall. 

To demonstrate how useful this is, here are two examples:

The first description of the iconic (sadly now-extinct) Thylacine (or Tasmanian Tiger) was published in the Transactions of the Linnean Society of London in 1808. This article is well and truly out of copyright and yet the definitive version of this article is behind a paywall on the Wiley Online website: https://doi.org/10.1111/j.1096-3642.1818.tb00336.x. Downloading the PDF of this out-of-copyright article from the Wiley website will cost you $42 USD.

The definitive DOI version of this article is behind a paywall on the Wiley Online website, but it is freely available on the BHL website. With the Unpaywall extension, you can now easily navigate to the free version on BHL.

Now that BHL’s content is discoverable via Unpaywall, anyone directed to the Wiley version via the article’s DOI can discover the free version on BHL (via the Unpaywall extension).

Article freely available via the Biodiversity Heritage Library.

It’s important to note that making BHL discoverable via Unpaywall doesn’t just enhance access to legacy literature such as the Thylacine paper; it also applies to much more recent research. For example, “The generic relationships of the new endemic Australian ant spider genus Notasteron (Araneae, Zodariidae)” was published in The Journal of Arachnology in 2005. This article has the DOI (https://doi.org/10.1636/04-56.1) and is behind a paywall on BioOne. With Unpaywall’s extension in your browser you can now discover the free version hosted by BHL.

How to make even more BHL content discoverable via Unpaywall

43,000 may seem like a large number, but it’s actually only a tiny fraction of the articles freely available on BHL. For the Unpaywall extension to be able to locate all the journal content on BHL, we need to unlock the rest of the journal articles on BHL by 1) adding more article-level metadata, and 2) ensuring that, for every article on BHL that has an existing DOI, we include that DOI in the article-level metadata. That’s our next task…

We would like to thank Unpaywall for providing access to an ever-increasing number of open access scholarly articles (23,943,966 at 14/8/19) and to particularly thank their Lead Developer, Richard Orr, for making it possible for BHL (a square peg) to fit into their open database (a round hole).