We have recently updated BHL to allow us to better handle situations where titles or items are taken offline. The key problem that occurs when things are taken offline is that the links to those items are now broken. Anyone who has linked to those titles/items or uses our data exports and/or APIs to access them is affected.

In some cases, titles/items are taken offline because of serious quality issues with the scans or because of copyright concerns. In those cases, there is little we can do other than “break” the links to those items.

In other cases, however, items may be rescanned and replaced by a new item. Or, two libraries might contribute the same title to BHL, and we at BHL combined those two titles into one. These are the cases for which we can preserve the links.

The change we have made is to allow our librarians to designate a “replaced by” identifier when titles/items are taken offline. So, if item 1000 is rescanned as item 2000, we can now take item 1000 offline, but redirect incoming requests for that item to item 2000. The same is now possible when two (or more) titles are merged into one… requests for the titles that are taken offline as a part of the merging process can now be redirected to the title that remains online.

Following is a more detailed explanation of how the various parts of BHL are affected.

biodiversitylibrary.org

If a title/item has been set to redirect to another title/item, then when you attempt to access the original page you will be automatically redirected to its replacement.

For example, if you navigate to http://www.biodiversitylibrary.org/bibliography/3935 you will be redirected to http://www.biodiversitylibrary.org/bibliography/7347 . Likewise, navigating to http://www.biodiversitylibrary.org/item/23127 will result in a redirect to http://www.biodiversitylibrary.org/item/28474 .

Data Exports

The bulk data exports that are generated on a monthly basis contain only active items and published titles. Because of this, redirects are not indicated in the data exports (since redirects apply to item that are no longer active and/or published).

Keep in mind, though, that if a third party has a copy of an older data export which contains titles/items that have subsequently been replaced, then when that third party follows the links contained in the exports they will be automatically be redirected to the correct title/item.

API

The following API methods honor the redirects:

GetItemByIdentifier
GetItemMetadata
GetItemPages
GetTitleBibTeX
GetTitleByIdentifier
GetTitleEndNote
GetTitleItems
GetTitleMetadata

Using the earlier examples, a call to GetTitlemMetadata with a parameter of 3935 will return the metadata for title 7347. Likewise, a call to GetItemMetadata with a parameter of 23127 will return the metadata for item 28474.

Detailed information our APIs can be found here: http://www.biodiversitylibrary.org/api2/docs/docs.html

Final Thoughts

It’s possible that you may find some titles/items that have a replacement item but have not been properly redirected. If that is the case, kindly let us know. Also, keep in mind that some titles/items may be taken offline and there may not be a valid replacement available. Please accept our apologies if you are affected by this situation.

We’ve been investigating options for storage and distribution of citation data in the Biodiversity Heritage Library. In particular, we are searching for an appropriate “core” format. The thought is that with an appropriately verbose, open, standard core format for our citations, we can transform that format into whatever other format we might want to support. By “verbose”, we mean a format that can support all of the information that we need to preserve. By “open”, we’re looking for a format that’s not tied exclusively to one system or vendor. And by “standard”, we’re hoping to identify a format that is widely recognized by the library community.

Some of the information found in this Wikipedia article has guided the research: http://en.wikipedia.org/wiki/Comparison_of_reference_management_software. Specifically, the information found there about which formats are supported in each of the various applications is useful.

Following is a brief description of the format candidates we’ve investigated, as well as our preliminary conclusions.

• The following formats appear (at the first look) to be the most open, verbose, and recognized formats.METS/MODS – Library of Congress standardshttp://www.loc.gov/standards/mets/
  http://www.loc.gov/standards/mods/ – examples can be found under the “Guidance” section

  NLM – National Library of Medicine format

  http://dtd.nlm.nih.gov/ – DTDs
  http://www.ncbi.nlm.nih.gov/staff/beck/citations/citationtags.html – examples

  EndNote (RIS/XML) – this seems to be the most widely adopted format

  http://www.endnote.com/support/ensupport.asp – XML DTD is here
  http://refdb.sourceforge.net/manual-0.9.4/c2166.html – RIS format description

• The following format is also a possibility, but it may be overly complex for our needs.RDFhttp://en.wikipedia.org/wiki/Resource_Description_Framework
  http://www.w3.org/RDF/
• Here are other formats that have been looked at, but appear to be deficient in one way or another.UniXRef – this is the XML format CrossRef returns from their OpenURL resolver.The verbosity of this format is good; it appears that a document using this format it could contain all of the information that we require. However, it is unclear how much this format has been adopted outside of specialized custom applications.

  http://www.crossref.org/help/Content/04_Queries_and_retrieving/Bulk%20metadata%20distribution.htm – schema found by clicking on “Unified XML Schema – Overview”

  CoiNS – not widely adopted
  MARC – doesn’t support article-level metadata (pages, etc)
  Dublin Core – Not detailed enough
  BibTeX – Not detailed enough
  OAI outputs – only a few defined outputs, which happen to be formats that are defined elsewhere (Dublin Core, RFC1807, MARC)
  RefWorks – too proprietary

If you have experience with one or more of these formats and would like to help us make our decision, please post your comments below.

I wanted to let everyone know about a change that has been made to the search function of the BHL portal.

Until now, letters that include diacritics (for example, ó, ö, è, é, û) were treated differently than letters without diacritics.

What this meant is that in order to find titles, authors, or subjects that included diacritics, you had to search for an exact match on the diacritic… for example, to find all titles about “invertebrate zoology”, you had to search twice: once for “invertebrate zoology” and once for “invertebrate zoölogy”. (Or you had to search for something like “invertebrate zo” and hope you didn’t get too much extra stuff in the search results.) Obviously, there are all sorts of problems with this limitation.

Starting immediately, searches in the BHL portal are accent-insensitive, so no distinction is made between letters with and without diacritics. This means that a search for “invertebrate zoology” will now find all nine titles that contain either “invertebrate zoology” or “invertebrate zoölogy”. See the search results here: http://www.biodiversitylibrary.org/Search.aspx?searchTerm=invertebrate%20zoology&searchCat;=. Another good example is searches for “Linne”, which now return instances of both “Linne” and “Linné”. http://www.biodiversitylibrary.org/Search.aspx?searchTerm=linne&searchCat;=T

While there is still more work to do to improve the search features, this is a good first step to improving the quality of our search results.

The BHL portal (http://www.biodiversitylibrary.org) has been updated with the following changes:

• A new option has been added to filter results by the language in which items are published. For example, Titles published in English, or Authors with works published in German. This option complements the pre-existing option to filter results by contributing institution.

• The display of search results has been enhanced. Results are now summarized at the top of the page, along with links to the different result categories (Titles, Authors, Names, or Subjects). Here is an example: http://www.biodiversitylibrary.org/Search.aspx?searchTerm=smith&searchCat;=. In addition, if more than 100 items are returned in any given result category, you are now given the option to expand the number of returned items to 500. For an example, follow this link (http://www.biodiversitylibrary.org/Search.aspx?searchTerm=poa&searchCat;=N) and scroll to the bottom of the page to see the “more names…” option.

• An advanced search page (http://www.biodiversitylibrary.org/advancedsearch.aspx) has been added. This page allows a user to search on any combination of search categories (Titles, Authors, Names, or Subjects), instead of just one or all of the categories. It also allows search results to be limited by the publishing language.

• OCLC numbers associated with each title have been cleaned up. This means that the “Find in a local library” link on each title’s bibliography page should now work correctly.

• Sorting of individual items within a single title has been improved. See the right side of this page (http://www.biodiversitylibrary.org/bibliography/702) for an example. You can see that the volumes are listed in order, v1 to v 92. Prior to the correction, the volumes were sorting as follows: v1, v10, v11, v12, … v2, v20, v21, v22, … v3, v30, v31, v32.

• Call numbers, when available, should now display correctly on bibliography pages.

• Recent Additions (http://www.biodiversitylibrary.org/Recent.aspx), and the associated RSS feeds, can now be filtered by Language and Contributor.

• Some minor updates have been made to the page that displays the discovered bibliography for a name. An example is http://www.biodiversitylibrary.org/name/Poa_annua. Changes have been made to retrieve data as needed, instead of all at once, which has improved performance greatly. However, there remains a lengthy delay in retrieving large data sets, so we know that more work is needed here. At a minimum, we know that we need to improve the feedback given to the user while large data sets are being retrieved.

Also of note:

Some inconsistencies with title information have been identified. We had believed that the MARC leader assigned to an item would be sufficient to uniquely identify a title. This has turned out to not be the case (affecting about one-half of one percent of the titles we’ve ingested from Internet Archive), so we’ve had to adjust how we identify which items belong to which titles. The cleanup of this data is ongoing.