Version 3 of the BHL API has been launched.

The development of a new API version was spurred by the recent introduction of full-text search to the BHL web site. In addition to the inclusion of full-text search, the entire API has been examined and updated. New methods have been added, existing methods have been modified, and many methods have been dropped entirely (or incorporated into other methods).

Because of the extent of the changes, version 3 of the BHL API has a separate endpoint from version 2. Both versions live side-by-side with one another, and do not conflict. Therefore, any existing users of the BHL API should see no disruptions.

API v3 endpoint: https://www.biodiversitylibrary.org/api3

API v2 endpoint: https://www.biodiversitylibrary.org/api2/httpquery.ashx

Long-term, it is expected that version 2 of the API will be deprecated and removed, so current API users are urged to look at version 3 and to begin moving to it.

The official documentation for version 3 of the BHL API is available at https://www.biodiversitylibrary.org/docs/api3.html. The remainder of this post describes the differences between version 3 and version 2.

Overall Changes

• No SOAP interface
• All “PrimaryTitleID” response elements have been renamed to “TitleID”
• All “GenreName” response elements have been renamed to “Genre”
• All “BibliographicLevel” response elements have been renamed to “Genre”
• All “Creator” response elements were renamed to “Author” (for example, <Creators> and <Creator> elements became <Authors> and <Author>, and <CreatorID> became <AuthorID>)

New Methods

GetAuthorMetadata

• Replaces and combines the old GetAuthorTitles and GetAuthorParts, as well as provides a way to retrieve basic Author metadata.
• Response includes author metadata, including a list of publications associated with the author.
• Response format:

<Result>
  <Author>
    <AuthorID></AuthorID>
    <Name></Name>
    <CreatorUrl></CreatorUrl>
    <Identifiers>
      <Identifier></Identifier>
      <Identifier></Identifier>
    </Identifiers>
    <Publications>
      <Publication></Publication>
      <Publication></Publication>
      <Publication></Publication>
    </Publications>
  </Author>
</Result>

GetSubjectMetadata

• Replaces and combines the old GetSubjectTitles and GetSubjectParts, as well as provides a way to retrieve basic Subject metadata.
• Response includes subject metadata, including a list of publications associated with the subject.
• Response format:

<Result>
  <Subject>
    <SubjectText></SubjectText>
    <Publications>
      <Publication></Publication>
      <Publication></Publication>
      <Publication></Publication>
    </Publications>
  </Subject>
</Result>

PageSearch

• Searches the text of a particular item (book) for a word/phrase
• Mimics the “Search Inside the Book” feature on the BHL web site
• Response format:

<Result>
  <Page></Page>
  <Page></Page>
  <Page></Page>
</Result>

PublicationSearch

• Replaces the old BookSearch and PartSearch methods
• Searches the text or text+metadata of items and parts
• Returns 200 results at a time; specify “page” to get a specific page of results
• Equivalent to the primary BHL site search
• Response format:

<Result>
  <Publication></Publication>
  <Publication></Publication>
  <Publication></Publication>
</Result>

PublicationSearchAdvanced

• Replaces the old BookSearch and PartSearch methods
• Search only title and part metadata by specifying a combination of “title”, “authorname”, “year”, “subject”, “language”, and “collection”.
• Search the full text of items matching the metadata criteria by including a “text” value.
• Returns 200 results at a time; specify “page” to get a specific page of results
• Equivalent to the “Advanced Search” feature of the web site
• Response format:

<Result>
  <Publication></Publication>
  <Publication></Publication>
  <Publication></Publication>
</Result>

Modified Methods

AuthorSearch

• Changed “name” argument to “authorname”

GetItemMetadata

• Renamed “itemid” parameter to “id”
• Added optional “idtype” parameter that defaults to value “bhl”. Valid values are: bhl, ia
• Defaulted “pages” parameter to “f”
• Defaulted “ocr” parameter to “f”
• Defaulted “parts” parameter to “f”
• Added <Item> container element around search results

<Response>
  <Result>
    <Item>…</Item>
  </Result>
</Response>

GetPageMetadata

• Defaulted “ocr” parameter to “f”
• Defaulted “names” parameter to “f”
• Added <Page> container element around search results

<Response>
  <Result>
    <Page>…</Page>
  </Result>
</Response>

• Moved <Name><NameBankID> and <Name><EOLID> elements into an <Identifiers> element to match how identifiers are formatted in other API responses.

Previous:

<Page>
  ... 
  <Names>
    <Name>
      <NameBankID></NameBankID>      <EOLID></EOLID>
      <NameFound></NameFound>
      <NameConfirmed></NameConfirmed>
    </Name>
  </Names>
</Page>

New:

<Page>
  ... 
  <Names>
    <Name>
      <Identifiers>
        <Identifier>
          <IdentifierName>NameBank</IdentifierName
          <IdentifierValue></IdentifierValue>
        </Identifier>
        <Identifier>
          <IdentifierName>EOL</IdentifierName
          <IdentifierValue></IdentifierValue>
        </Identifier>
      </Identifiers>
      <NameFound></NameFound>
      <NameConfirmed></NameConfirmed>
    </Name>
  </Names>
</Page>

GetPartMetadata

• Renamed “partid” parameter to “id”
• Added optional “idtype” parameter that defaults to value “bhl”. Valid values are: bhl, doi, jstor, biostor, soulsby
• Added “names” parameter to allow names to be included in the response. For example:

<Part>
  ...
  <Names>
    <Name>
      <Identifiers>
        <Identifier>
          <IdentifierName>NameBank</IdentifierName
          <IdentifierValue></IdentifierValue>
        </Identifier>
        <Identifier>
          <IdentifierName>EOL</IdentifierName
          <IdentifierValue></IdentifierValue>
        </Identifier>
      </Identifiers>
      <NameFound></NameFound>
      <NameConfirmed></NameConfirmed>
    </Name>
  </Names>
</Part>

• Added <Part> container element around search results

<Response>
  <Result>
    <Part>…</Part>
  </Result>
</Response>

• Renamed <PartIdentifier> element to <Identifier>

GetTitleMetadata

• Defaulted “items” parameter to “f”
• Added <Title> container element around search results

<Response>
  <Result>
    <Title>…</Title>
  </Result>
</Response>

• Renamed <TitleIdentifier> element to <Identifier>

NameGetDetail

• Renamed to GetNameMetadata
• Removed “namebankid” parameter
• Added “id” parameter
• Added “idType” parameter that accepts values: namebank, eol, gni, ion (index to organism names), col (catalogue of life), gbif, itis, ipni, worms
• To invoke this method, users should supply either a “name” parameter, or “idType” and “id” parameters

Examples:

op=GetNameMetadata&name=poa+annua

op=GetNameMetadata&type=namebank&value=123456

• Added <Name> container element around search results

<Response>
  <Result>
    <Name>…</Name>
  </Result>
</Response>

• Moved <Name><NameBankID> and <Name><EOLID> elements into an <Identifiers> element to match how identifiers are formatted in other API responses.

Previous:

<Name>
  <NameBankID></NameBankID>
  <EOLID></EOLID>
  <NameFound></NameFound>
  <NameConfirmed></NameConfirmed>
</Name>

New:

<Name>
  <Identifiers>
    <Identifier>
      <IdentifierName>NameBank</IdentifierName
      <IdentifierValue></IdentifierValue>
    </Identifier>
    <Identifier>
      <IdentifierName>EOL</IdentifierName
      <IdentifierValue></IdentifierValue>
    </Identifier>
  </Identifiers>
  <NameFound></NameFound>
  <NameConfirmed></NameConfirmed>
</Name>

NameSearch

• Identifiers are no longer included in the response.

Removed Methods

The following methods are not part of API v3, either because they were rarely used in API v2, their functionality was duplicated in other methods, or they were replaced with other methods.  Where appropriate, the replacement for a removed method is noted.

• BookSearch – replaced with PublicationSearch and PublicationSearchAdvanced
• GetAuthorParts – replaced with GetAuthorPublications
• GetAuthorTitles – replaced with GetAuthorPublications
• GetItemByIdentifier – merged with GetItemMetadata
• GetItemPages – same information available from GetItemMetadata
• GetItemParts – same information available from GetItemMetadata
• GetPageNames – same information available from GetPageMetadata
• GetPageOcrText – same information available from GetPageMetadata
• GetPartBibTex
• GetPartByIdentifier – merged with GetPartMetadata
• GetPartNames – same information available from GetPartMetadata
• GetPartRIS
• GetStats
• GetSubjectParts – replaced with GetSubjectPublications
• GetSubjectTitles – replaced with GetSubjectPublications
• GetTitleBibText
• GetTitleByIdentifier – merged with GetTitleMetadata
• GetTitleItems – same information available from GetTitleMetadata
• GetTitleRIS
• GetUnpublishedItems
• GetUnpublishedParts
• GetUnpublishedTitles
• NameCount
• NameCountBetweenDates
• NameList
• NameListBetweenDates
• PartSearch – replaced with PublicationSearch and PublicationSearchAdvanced
• TitleSeachSimple – replaced with PublicationSearchAdvanced (specify title parameter only)

The BHL API will be updated on 25 July 2016 to support changes to the BHL site. These changes will accommodate identifying additional Contributors for Items and Parts of items.

First are changes to the API that may affect your existing processes.

The Contributor and ContributorID elements in the result sets of API methods that return “Part” information will move. ContributorID will be included as a PartIdentifier in the Identifiers list. Contributor will be included in a new Contributors list.

These changes are being made to accommodate more than one contributor per part.  For example, if one institution researches/compiles the data and a second institution facilitates the inclusion of that data in BHL, both institutions may be listed as a contributor.  Initially, no more than two contributors per part will be allowed, but by adopting these changes to API responses we allow for additional (unlimited, actually) contributors in the future.

Here is an example that shows how the API responses are changing.  The examples shown here are an output of the GetPartMetadata method, and have been abbreviated for clarity.

Original API Results – highlighted elements are being moved:

<Response>
 <Status>okStatus>
 <Result>
 <PartUrl>
        http://www.biodiversitylibrary.org/part/1
 PartUrl>
 <PartID>1PartID>
 <ItemID>22498ItemID>
 <StartPageID>3190776StartPageID>
 <SequenceOrder>1SequenceOrder>
 <Contributor>BioStorContributor>
 <ContributorID>4443ContributorID>
 <GenreName>ArticleGenreName>
 <Title>
      Notes on certain species of Tetragnatha 
      (Araneae, Argiopidae) in Central America 
      and Mexico
 Title>
 <ContainerTitle>BrevioraContainerTitle>
 <Volume>67Volume>
 <Date>1957Date>
 <PageRange>1--4PageRange>
 <StartPageNumber>1StartPageNumber>
 <EndPageNumber>4EndPageNumber>
 <Authors> [...] Authors>
 <Subjects />
 <Identifiers>
 <PartIdentifier>
 <IdentifierName>ISSNIdentifierName>
 <IdentifierValue>0006-9698IdentifierValue>
 PartIdentifier>
 Identifiers>
 <Pages> [...] Pages>
 <RelatedParts />
 Result>
Response>

Updated API Results – Highlighted elements are the new locations of the moved data:

<Response>
 <Status>okStatus>
 <Result>
 <PartUrl>
        http://www.biodiversitylibrary.org/part/969
 PartUrl>
 <PartID>1PartID>
 <ItemID>22498ItemID>
 <StartPageID>3190776StartPageID>
 <SequenceOrder>1SequenceOrder>
 <GenreName>ArticleGenreName>
 <Title>
        Notes on certain species of Tetragnatha
        (Araneae, Argiopidae) in Central America
        and Mexico
 Title>
 <ContainerTitle>BrevioraContainerTitle>
 <Volume>67Volume>
 <Date>1957Date>
 <PageRange>1--4PageRange>
 <StartPageNumber>1StartPageNumber>
 <EndPageNumber>4EndPageNumber>
 <Authors> [...] Authors>

 <Contributors>
 <Contributor>
 <ContributorName>BioStorContributorName>
 Contributor>
 Contributors>
 <Subjects />
 <Identifiers>
 <PartIdentifier>
 <IdentifierName>BioStorIdentifierName>
 <IdentifierValue>4443IdentifierValue>
 PartIdentifier>
 <PartIdentifier>
 <IdentifierName>ISSNIdentifierName>
 <IdentifierValue>0006-9698IdentifierValue>
 PartIdentifier>
 Identifiers>
 <Pages> [...] Pages>

 <RelatedParts />
 Result>
Response>

Additionally, there will be two additions to the Item metadata.

The new data elements are: RightsHolder and ScanningInstitution.
These will optionally be displayed if there is data for the relevant organizations.

<Response>
 <Status>okStatus>
 <Result>
 <ItemID>59382ItemID>
 <PrimaryTitleID>20770PrimaryTitleID>
 <ThumbnailPageID>17605914ThumbnailPageID>
 <Source>Internet ArchiveSource>
 <SourceIdentifier>bulletin5160hatcSourceIdentifier>
 <Volume>v.51-60 1898-99Volume>
 <Year/>
 <Contributor>
      UMass Amherst Libraries (archive.org)
 Contributor>
 <RightsHolder>
 Biodiversity Heritage Library
 RightsHolder>
 <ScanningInstitution>
 Smithsonian Institution Libraries
 ScanningInstitution>
 <Sponsor>UMass Amherst LibrariesSponsor>
    [...]
 Result>
Response>

Detailed documentation for the BHL APIs is available at http://www.biodiversitylibrary.org/api2/docs/docs.html.  It will be updated to reflect these changes after they are moved into production on 25 July 2016.

Go to http://www.biodiversitylibrary.org/getapikey.aspx to get an API key for BHL.

Information about other BHL developer tools can be found at http://biodivlib.wikispaces.com/Developer+Tools+and+API.

If you have questions, please feel free to submit feedback via this form.

Tarantulas are amazing. Not only do they include the largest of all spiders, with some species reaching a legspan the size of a dinner plate, but they are arguably some of the most beautiful too. While famous for giants that inhabit the jungles of South America, some species barely grow larger than your thumb nail. Some species live on trees in damp forests while others live in self-constructed tubular burrows in the ground in some of the most inhospitable deserts. Some have special protective hairs on their bodies which cause extreme itching when they come into contact with the mucous membranes of potential predators, while others produce a hissing sound in self-defence. While they are the stuff of nightmares for some people, they are the source of absolute fascination for others.

The Tarantupedia is relatively new venture started about three years ago by a tarantula enthusiast in South Africa, Dimitri Kambas. His goal is to produce the authoritative online resource for information on tarantulas. Importantly, his focus is on scientific information, something that stands in stark contrast with the majority of tarantula websites which are centered on keeping tarantulas in captivity. The Tarantupedia is different. You won’t find captive care sheets or guidelines on how to breed a particular species. Instead you will find the kind of the information reminiscent of a detailed scientific publication, but presented in an easy-on-eye, comfortable-to-work-with format intended for scientists and interested members of the general public alike.

Dimitri Kambas, Co-Founder and Editor, The Tarantupedia.

The project began with the construction of a digital taxonomic catalogue, a presentation of tarantula classification linked to information on taxonomic authors and their publications. The Tarantupedia uses modern web technologies, so the data are presented in a dynamic format where the user can view the same information from multiple different perspectives. You may want to know who described a particular species, or how many species a particular author has described. You can get lists of species mentioned in a particular publication, or if you’re interested, you can get a short biography for some of the better known tarantula experts. You can even see where the original type specimen for a particular species was collected, presented neatly on Google Maps.

Tarantula taxonomy had early beginnings, with Linnaeus himself even responsible for the descriptions of one or two species. Developing the Tarantupedia has required investigation of some of the earliest literature, and this is where the Biodiversity Heritage Library has been invaluable. Through the efforts of the BHL, many obscure and largely forgotten taxonomic articles were available for consultation where they were needed. The Tarantupedia links directly to the BHL so users can examine relative literature for themselves at the click of a mouse. Using the BHL API, Tarantupedia finds instances of species names in the articles stored on BHL and creates hyperlinks directly to those article pages where the species names are mentioned. Anyone who has worked with old literature knows how daunting it can be trying to track down old but important works. Thanks to the BHL, not only do you get the relevant literature, but you can go right to the parts of the literature that are of interest to you! This creates a resource that taxonomists can use to find the information they need fast, lowering the barrier to further taxonomic research which is sorely needed for many tarantula taxa.

Tarantupedia record showing links to BHL literature.

If you’re interested to learn more please visit the Tarantupedia at www.tarantupedia.com. This is an on-going project with a growing base of contributors from around the world and new information is added almost daily. Suggestions for improvements are welcome, as are contributions in the way of photographs, sighting records, and missing literature. With continued support, particularly from other projects like the BHL, the Tarantupedia will become an invaluable resource for anyone interested in these amazing animals.

Enjoy a selection of historic tarantula illustrations from books in the BHL collection on Flickr.

2.  How long have you been in your field of study?
Broadly defined… about 25 years.

3.  When did you first discover BHL?

I think pretty much when you first opened.

4.  What is your opinion of BHL and how has it impacted your research?

BHL is a vital tool for me.  I live in West Kalimantan, Indonesia and so have no local physical access to botanical libraries. The nearest one is on Java, at the Indonesian National Herbarium (Herb. Bogoriense); it has a surprisingly good range of older publications, but I don’t get there as often as I’d like.  But with BHL, in a few seconds (or longer… depending on bandwidth here!), I have access to many of the original (and sometimes only) descriptions for the plant taxa out here.

5.  How often do you use BHL?

I use BHL a couple of times a month on average, sometimes more frequently.

6.  How do you usually use BHL?  

Almost always I’m looking for species descriptions, ecological details, and particularly images.

7. What are your favorite features and services on BHL?

So, I have to admit, as I was answering these questions, I started digging into the API options (Application Programming Interface) you offer, that I hadn’t really looked at before, and was blown away as to how many ways you offer to query your database and view the publication pages. One of the main challenges I have for using BHL, given the limited  bandwidth available here in Kalimantan, is the multiple webpage loadings required to get from submitting a taxonomic name to being able to check if a page will be worth reading, partially because the default page viewer loads a fairly high-resolution page image in the popup. So I hacked together a simple script (http://xmalesia.info/doc/bhl_pages.html) that takes a taxonomic name, calls your API, and returns a single page with embedded thumbnails of matching pages in BHL. This way, I can quickly identify which pages might be worth looking more carefully at.  So now I have to say that my favorite service on BHL is the API!  Thanks!

8.  What would you like BHL to focus on developing next?

For myself, I’d have to say that continuing to expand the coverage of the core document set would still be the highest priority.  There are a number of key references for Indonesia (many of them in Dutch)
that I have not found yet in BHL.

9. If you had to choose one title or item that has impacted your research or something that you love in BHL, what would it be?

It is having Flora Malesiana scanned and available online!

Thank you, Cam, for sharing your work and how you use the Biodiversity Heritage Library!

Research teams at the NESCent-EOL-BHL Research Sprint. Photograph by Cyndy Parr.

In early February, the National Evolutionary Synthesis Center (NESCent) hosted the EOL-BHL Research Sprint. NESCent, based in Durham, NC, is a non-profit science center supporting research in the evolutionary sciences. NESCent emphasizes an interdisciplinary approach to research, and so the idea behind the Sprint was to put together teams of programmers and life scientists to expose each other to questions and ways of thinking that they might not necessarily consider in their normal work. Informaticians could bring programming and data skills to bear on questions that scientists may not have had the programming expertise to implement effectively, using BHL‘s and EOL‘s now considerable amount of freely available data. Scientists could identify questions based on the data to programmers that they might not have considered. Plus, the meeting was useful in identifying how well researchers could identify and retrieve the data they needed from the BHL text corpus. To this end, William Ulate, BHL Technical Director and John Mignault, a member of the BHL Technical Advisory Group attended the meeting.

The teams covered a wide variety of interesting topics from studying the color of butterflies based on extracting color information from images to studying changes in ontologies over time based on an analysis of the text in the BHL corpus (see http://bit.ly/1dnnhG0). Over the course of the sprint, the teams began data mining EOL and BHL for their data sets and started preliminary analyses of their data. Each day, groups met at the end of the day to share experiences and progress. By the end of the sprint, each of the teams were sharing plans for further collaboration and completing their analyses. Plans for publication and grants proposals based on sprint ideas were also discussed. In an open, collaborative spirit, members shared the materials freely via Google Drive.

We learned some interesting things about the way people approach the BHL data set. Many of the teams on the first day wanted to use the BHL application programming interface for bulk data retrieval. Several team members asked us how they could download “all of the text.” When we told them that this was impractical and would result in a great deal of unwanted data, they asked how they could retrieve data based on, for example taxa – I want to harvest all pages with names from this taxon (Chordata) or this common name (Vertebrate). Others wanted data restricted by location. We tried to assist them given their specific needs rather than their initial request for the whole data set (see http://bit.ly/1rvbut3). This raised useful questions as to how we can provide the data to researchers need in the ways they need it – should we offer ways to request bulk data downloads based on a specific set of criteria? Should we alter the API (http://www.biodiversitylibrary.org/api2/docs/docs.html) in order to make it possible to retrieve more closely focused data sets? As BHL becomes better known as a source of “Big Data” for the biodiversity community, we will need to evolve our access to that data in order to better meet the needs of our users.

We were also surprised to discover the popularity of the R statistical programming language among scientists. Many team members used R in their work, to such an extent that a short R group discussion was scheduled for one morning during the meeting. Scott Chamberlain of Simon Fraser University has created an R interface to the BHL API, available at http://bit.ly/1oAFKjI. It is always good to see BHL and its data used in new and interesting ways. Follow up further results from this Sprint at: http://blog.eol.org.

The Sprint was a valuable meeting for BHL: it exposed our valuable data to more scientists and informaticians, and it gave BHL staff useful feedback on the uses of the BHL data corpus and its value to researchers. We would like to thank EOL, NEScent and the Richard Lounsbery Foundation for the opportunity and their collaboration in making this event a success.