As has been previously discussed, BHL has uncorrected text generated by Optical Character Recognition (OCR) software for each of its scanned volumes, and that uncorrected text has implications for data mining and accurate search. OCR results are notoriously poor because the technology hasn’t improved much since it was “solved” for forms processing in the mid-1980’s, which doesn’t really help BHL at all with our challenge of getting accurate text for our heterogeneous digital library spanning more than 500 years of printed publications.

Luckily some new work is happening around OCR and text correction from various projects, including:

• Rod Page’s examples with Wikisource and a Google-like text selector
• National Library of Australia’s Trove project (with their code)
• EU-funded Improving Access to Text (IMPACT) project

The BHL development team and other stakeholders recently highlighted “improvements to OCR text” as an emerging priority for BHL to pursue. Now that we have a sizable repository we’d like to turn development efforts towards enhancing our texts and making them as accurate as possible for our users and data consumers. A subset of BHL is required for various tests and experiments, and that list has begun forming at:
BHL Sample Texts for Retrieval and Evaluation

• https://spreadsheets.google.com/ccc?key=0AiGG0YxTuAn2dGN4R0ZyTTNTZUhfUlU1cHJ3T2JGdUE&hl;=en

If you encounter other texts that pose a unique challenge, please indicate them either via BHL’s Feedback form or on a comment below.

In November 2009, the BHL started ingesting biodiversity related content from the Internet Archive. Since then, the BHL collection has grown significantly. Each week, the BHL brings in new content from the Internet Archive based on a criteria of selected Library of Congress Subject Headings and call numbers, with the aim of bringing in content related to the literature of biological diversity.

As described by the Convention on Biodiversity,

Biological diversity – or biodiversity – is the term given to the variety of life on Earth and the natural patterns it forms.…This diversity is often understood in terms of the wide variety of plants, animals and microorganisms.…Biodiversity also includes genetic differences within each species – for example, between varieties of crops and breeds of livestock. Chromosomes, genes, and DNA-the building blocks of life-determine the uniqueness of each individual and each species. Yet another aspect of biodiversity is the variety of ecosystems such as those that occur in deserts, forests, wetlands, mountains, lakes, rivers, and agricultural landscapes. In each ecosystem, living creatures, including humans, form a community, interacting with one another and with the air, water, and soil around them. http://www.cbd.int/convention/guide/

The BHL collection reflects this wide interdisciplinary spectrum. When ingesting content from the Internet Archive, the criteria of LCSH and LC call numbers are used to target as much relevant biodiversity literature as possible, but it is not fool proof.

Recently, the Ingest Criteria was reviewed by the BHL Collections Committee to identify subject terms and call numbers that were missing the mark. We noticed books were being brought in that were clearly not relevant to the BHL mission such as books about human anatomy, taxation, and hockey. The Ingest Criteria have since been revised to better target biodiversity relevant materials and most of the irrelevant content has been removed from our collection. Users should not have to weed through irrelevant materials in the BHL collection when access to this content remains available via other open access digital collections. For example, access to Mary Louise Serafine’s Measure of meter conservation in music based on Piaget’s theory will be via the Internet Archive website rather than through the BHL.

For more information about BHL’s de-accession policy, please see our documentation on our wiki http://biodivlib.wikispaces.com/Deaccession+Policy. Should you feel that any content has been removed from BHL unnecessarily, please feel free to send us your comments to our feedback form.

So, many of you participated in our BHL User Survey 2010, and we greatly appreciate your contributions! One of the most prevalent themes throughout the range of responses that we received was that our users want to be able to submit requests for scanning. So, you spoke; we listened.

Introducing the new scanning request form on BHL!

You as a user can now submit requests for items to be scanned and added to the BHL collection, free of charge to you, no less! The process is easy, too. Simply access the “Feedback” form on the BHL website (located in the upper right hand corner of the header through the “Feedback” link, or at the title and item level in BHL by clicking on the “Report an Error” link on the right hand side of the title screen or above the scan images in the book viewer) and choose the option “Scanning Request.” You can then fill out and submit the form indicating your request for scanning.

We do ask that you keep a few things in mind. First, due to copyright restrictions, requests for scanning are limited to items published before 1923. Secondly, our ability to scan a request is limited both to the collections of our BHL member institutions and the condition of the volumes they might hold. If we don’t have it, we can’t scan it. If it’s falling apart, we probably won’t be able to scan it. Finally, we scan requests in the order they are received, and depending on certain circumstances, such as the rarity or fragility of the item in question, it may take longer to scan certain requests. Nevertheless, we are excited about this new feature and the opportunity it gives you, the user, to contribute to the building of our ever-increasing digital collection. So, pull out those lists of items you wished BHL had and start submitting your requests! We’re waiting…

New updates have been released for the BHL API, plus new documentation about data exports and other services are now available on the Developer Tools and API section of the BHL wiki.

The BHL Application Programming Interface (API) is a set of web services that can be invoked via HTTP queries (GET/POST requests) or SOAP. Responses can be received in one of three formats: JSON, XML, or XML wrapped in a SOAP envelope.

Version 1 (formerly the BHL Name Services): Updated documentation for the first version of the API can be found at http://www.biodiversitylibrary.org/api/docs/docs.html. This version of the API is provided solely to maintain backwards compatibility.

Version 2: The documentation for the latest version of the API can be found at http://www.biodiversitylibrary.org/api2/docs/docs.html. The first version of the API was limited to data related to scientific names found in the BHL collection; version 2 adds access to title, author, volume, and page information. Please note that users are required to obtain an API Key from http://www.biodiversitylibrary.org/getapikey.aspx in order to use version 2 of the API. This is the preferred version of the API.

View the API Privacy Policy and Terms Of Use.

BHL has released a beta version of its OpenURL Resolver API for testing. A full description of the service is available at http://www.biodiversitylibrary.org/openurlhelp.aspx.

Any repository containing citations to biodiversity literature can use this API to determine whether a given book, volume, article, and/or page is available online through BHL. The service supports both OpenURL 0.1 and OpenURL 1.0 query formats, and can return its response in JSON, XML, or HTML format, providing flexibility for data exchange.

One issue still under consideration is whether to assign an API key to each user of the service, similar to Google Maps and many other data providers. Some advantages to assigning a key include having a method by which to contact users to notify them of updates & service availability, and yes, to track usage. Some disadvantages include a perceived restriction on access to the service and concerns about privacy & data tracking, among others. In truth we are less concerned about restricting users and are more interested in finding a way to monitor use of the service and to communicate with its users.

We started a discussion of the pros and cons of the API key approach on Twitter but the text size restriction made responses laughable. We’re interested in your views as either potential consumers of the BHL OpenURL API or from your experience in managing similar services for your project. We’re also especially interested in viewpoints from those with experience deploying OpenURL, which we’ve come to learn is a fairly niche group. Please leave your comments below so that we can continue the dialogue using more than 140 characters!

BHL developers have incorporated the Internet Archive’s open source book viewing application into the BHL portal, providing a new interface for using BHL’s digital books. Check out last week’s BHL Book of the Week, The Edible Mollusca of Great Britain and Ireland with Recipes for Cooking Them, online through the BHL portal at:
http://www.biodiversitylibrary.org/item/46647

Now try out the new book viewer:
http://www.archive.org/stream/ediblemolluscaof00love
*This is also available underneath the “Download/About this book” link on each book. Click “Alternate Page Viewer (beta)”.

There are some key pieces of information missing from the book viewer display that are important for BHL users, like the corresponding volumes if the book is a journal/series, lists of names, view of all page numbers & page descriptions, etc. Those have been documented and discussed at:
http://blog.chrisfreeland.com/2008/10/internet-archives-new-book-viewer.html

Look for these improvements to be incorporated into the viewer in the coming months, as the viewer code is open and available for enhancement. For now, the BHL development team would appreciate your feedback on this new viewing option via comments on this post. Let us know what you like, dislike, want larger/smaller, wish would go away and never return, find confusing, find useful…everything – all feedback is useful for improving the display.

We’ve been investigating options for storage and distribution of citation data in the Biodiversity Heritage Library. In particular, we are searching for an appropriate “core” format. The thought is that with an appropriately verbose, open, standard core format for our citations, we can transform that format into whatever other format we might want to support. By “verbose”, we mean a format that can support all of the information that we need to preserve. By “open”, we’re looking for a format that’s not tied exclusively to one system or vendor. And by “standard”, we’re hoping to identify a format that is widely recognized by the library community.

Some of the information found in this Wikipedia article has guided the research: http://en.wikipedia.org/wiki/Comparison_of_reference_management_software. Specifically, the information found there about which formats are supported in each of the various applications is useful.

Following is a brief description of the format candidates we’ve investigated, as well as our preliminary conclusions.

• The following formats appear (at the first look) to be the most open, verbose, and recognized formats.METS/MODS – Library of Congress standardshttp://www.loc.gov/standards/mets/
  http://www.loc.gov/standards/mods/ – examples can be found under the “Guidance” section

  NLM – National Library of Medicine format

  http://dtd.nlm.nih.gov/ – DTDs
  http://www.ncbi.nlm.nih.gov/staff/beck/citations/citationtags.html – examples

  EndNote (RIS/XML) – this seems to be the most widely adopted format

  http://www.endnote.com/support/ensupport.asp – XML DTD is here
  http://refdb.sourceforge.net/manual-0.9.4/c2166.html – RIS format description

• The following format is also a possibility, but it may be overly complex for our needs.RDFhttp://en.wikipedia.org/wiki/Resource_Description_Framework
  http://www.w3.org/RDF/
• Here are other formats that have been looked at, but appear to be deficient in one way or another.UniXRef – this is the XML format CrossRef returns from their OpenURL resolver.The verbosity of this format is good; it appears that a document using this format it could contain all of the information that we require. However, it is unclear how much this format has been adopted outside of specialized custom applications.

  http://www.crossref.org/help/Content/04_Queries_and_retrieving/Bulk%20metadata%20distribution.htm – schema found by clicking on “Unified XML Schema – Overview”

  CoiNS – not widely adopted
  MARC – doesn’t support article-level metadata (pages, etc)
  Dublin Core – Not detailed enough
  BibTeX – Not detailed enough
  OAI outputs – only a few defined outputs, which happen to be formats that are defined elsewhere (Dublin Core, RFC1807, MARC)
  RefWorks – too proprietary

If you have experience with one or more of these formats and would like to help us make our decision, please post your comments below.