The big day is here! BHL’s Tech Team is starting the three-week process of moving BHL’s technical infrastructure from the Smithsonian data center just outside of Washington, DC, to the Field Museum in Chicago, Illinois.

First, though, we are deeply grateful to the Field Museum for welcoming BHL’s technology and helping ensure that biodiversity knowledge remains available to the global community. The support from their administration and technical team is invaluable.

We’ve been testing and configuring the network at the Field Museum and BHL is already quietly running on their network, so we are confident the process will go smoothly when we ship and install the production database and full-text-search servers.

“How will this affect BHL?”, you say. Great question! As much as possible, the BHL website will remain online and we do not expect any time where BHL is unavailable. The shipments to Chicago will take place in two parts to ensure BHL is always available, with some limitations. The first shipment on 11 June 2026 will have no impact on the regular functions of the site.

The second shipment on 25 June 2026 will cause the usual full-text search function to be temporarily unavailable while the full-text-search server is in transit. This means users will still be able to search titles, authors, subjects, and other catalog metadata, but users will not be able to search the full text of books and articles. This will last for 5-6 days over a weekend.

The 11th of June is a warm up, but the big day is the 25th of June when we officially begin serving BHL from the Field Museum.

We’ve set up an informational page with more details and will update it as things progress. We’ve also provided a map to illustrate where BHL’s servers are during transit.

A map of the route the servers will take to get to Chicago.
Please note that the path is neither accurate nor to scale. (Map Source)

But will the BHL servers feel at home in Chicago? To help them acclimate, we looked to historical maps of rainfall, soils, and vegetation to compare BHL’s former and future environments.

From: Atlas of American agriculture, no.5 (Source)

In Chicago, BHL will be receiving slightly less rainfall (30-35 ” or 75-88 cm per year) than on the U.S. East Coast (35-40″ or 85-100 cm). This, however, is mitigated by its proximity to Lake Michigan, which is one of the largest freshwater lakes in the world.

This looks like sufficient water for BHL, but these are historical values and may not reflect the effect of global climate change. In fact, it’s reported that the north-central part of the United States is in a moderate drought.

From: Atlas of American agriculture, no.8 (Source)

What about soil type? BHL is leaving an area of Gray-Brown Podzolic soils to the much less prodigious Soils of the Northern Prairie. Looking closely at this map, we can see that BHL’s familiar soils are in close proximity to its new soil environment in Chicago, so we hope that this will help BHL rapidly make the transition to its new home.

From: Atlas of American agriculture, no.6 (Source)

Lastly, we look at vegetation. Historically the eastern seaboard of the United States is abundant in Oak-Pine and Chestnut-Oak-Yellow Poplar forests. The move to Chicago brings a potentially dramatic change as BHL moves from forests to the Tall Grasses of the Prairie Grasslands. Much like the new soil type, we see that BHL is still in close proximity to Oak-Hickory forests and we hope that the presence of the familiar Oaks will also help make the transition.

With familiar oaks nearby, sufficient water, and a welcoming new home at the Field Museum, we think BHL is well positioned to put down roots in Chicago.

Summary

Over the past year, BHL has been forced to contend with the issue of AI Bot (short for robot) Crawlers and the effect of their unreasonable requests for BHL’s content. I say unreasonable because while BHL’s content is made freely available and is probably beneficial to modern Large Language Models (LLMs), the behavior of these bots and their disruptive effect on BHL has proven that they are bad partners in the exchange of information.

BHL is not the only organization to contend with this in recent months. Other cultural heritage organizations have reported the same or similar experiences, most notably this detailed report from Michael Weinberg at the GLAM-E Lab.

By the time this and other articles had been posted in mid-2025, BHL had already crested the first wave of interruptions.

A bit of background

When a request is made to a website, all actors are expected to identify themselves. Your browser informs the website that it is Firefox or Chrome. Many of the bots that come to a site are there to fill the index for a search engine. They have clever names like “GoogleBot” or “BingBot”. Furthermore, many LLM bots identify themselves, too. “ClaudeBot” or “GPTBot”, for example.

On the other side of the equation we have the “robots.txt” file which is meant to create an informal agreement between website and Bot to set rules on how the bots should behave. It contains directives like “you can go to these these URLs”, “please don’t go to these other URLs” or “please don’t make more than 10 requests per second”. These directives can be set individually for different bots to help guide them to what we think is the most useful content for them.

Some LLM bots ignore all of this.

The disruptive effect

In fact, the Grok bot sneaks its way into web servers by lying about who it is. No one has seen a “GrokBot” visiting their sites. This behavior is recently well documented. This is bad behavior, and it’s not something a handful of humans can fight alone.

We found that this is what was happening at BHL: The traffic that we saw did not identify themselves and they also came from many thousands of locations around the internet, making it difficult to control or block them. The numbers show it:

This chart shows traffic to BHL’s “/search” URL for approximately the past year. The peaks in June 2025 and February 2026 exceeded BHL’s capacity.

Beginning in April 2025, we saw a gradual increase in traffic that peaked on 19 June 2025 when it dropped off completely. This was the first wave of disruptions which we saw again later in 2025 and into 2026.

During this time, the Smithsonian was the host for the BHL servers and making changes in their ecosystem was a challenge, but the firewalls (and the team) at the Smithsonian are amazing and played a large part in keeping BHL operational. Regardless, we believe that enough traffic was allowed to get through the firewalls to still cause disruptions and unexpected errors trying to get to BHL.

The Smithsonian’s firewalls were an important part of blocking traffic to BHL in February 2026. This chart shows requests per hour that were blocked by the firewalls for one week in February.

Our Response

Starting in June 2025 and continuing through September, Mike Lichtenberg, BHL Lead Software Engineer, made some small but effective changes to how BHL works internally. This allowed BHL to serve a new, higher level of traffic on the same technical infrastructure. Another change we made was to redirect certain requests to the Internet Archive, which was already protected by Cloudflare. This freed up capacity at BHL to serve more requests.

This chart shows per-day traffic to BHL. On the left side before August 2025, our regular traffic was around 2.5 million hits per day. On the right side, after Mike’s improvements the new baseline is closer to 5 million hits per day.

Unfortunately this was not enough. In January 2026, the bots returned, and worse than before.

The Cloudflare Cavalry

After a few weeks, it became clear that our efforts were no longer sufficient. Around 19 June 2025, BHL had responded to 870,000 search requests. This far exceeded our usual 22,000 searches per day.

On 10 February 2026, BHL received a peak of 943,000 search requests.

We’d been encouraged to pursue Cloudflare for some time, but it wasn’t until mid-February that our technical infrastructure was ready for the switch. On 20 February we officially started using Cloudflare and after a few missteps, everything was in place by 24 February.

But what is Cloudflare, you ask? It’s a service that sits between a web site and the rest of the internet and it looks at requests to that website to decide if they should be allowed through. As a very large player on the internet, Cloudflare has access to mountains of data on bot-like behavior to accurately analyze traffic in real-time.

After activating Cloudflare, we immediately saw a reduction in traffic and much more stability at BHL. In the following weeks, we made other changes, such as adding the ubiquitous “are you human?” checkbox that has become common across the internet.

This chart shows traffic to BHL’s servers dropped dramatically when Cloudflare was fully activated.

Cloudflare is not simply a gatekeeper preventing unwanted traffic from coming to BHL. They also cache regularly requested parts of BHL (images, scripts, and styles) to deliver them to you from a location that is geographically closer to you. This means BHL will often seem to load faster than it did before. This also accounts for the drop in traffic to BHL as indicated in the chart above.

Cloudflare is now blocking a large amount of automated traffic coming to BHL. 

This chart shows a bit more detail of how Cloudflare is affecting requests to BHL. The top line is requests blocked by Cloudflare, the middle line is requests served by BHL’s own infrastructure, and the bottom line is requests served by  Cloudflare’s caches.

In Closing

It’s been a long journey, but BHL is much safer from the army of bots. This doesn’t mean that our work is done. We’ve had to make special efforts to not block access to BHL’s APIs and data feeds, which are often used by software or code…and which often behave like a bot! These settings have been in place for a few weeks now.

Additionally, we are investigating whether to allow the LLM bots to resume accessing our high-quality content, but we must be able to limit their flow of traffic to prevent overwhelming BHL’s servers.

Rest assured that BHL’s Technical Team is committed to keeping BHL up and running as we always have, but now we have a powerful new tool to help us in those efforts.

With the Biodiversity Heritage Library now officially operating in its new chapter following its transition away from the Smithsonian Institution, we are delighted to announce that the Field Museum of Natural History, one of BHL’s longest-standing and most respected partners, will host BHL’s servers and core technical infrastructure.

This hosting arrangement ensures the continued stability, security, and reliability of the BHL website and the 63+ million pages of open biodiversity knowledge it makes freely available worldwide.

Paper Nautilus, from Zoography; or, The beauties of nature displayed. In select descriptions from the animal, and vegetable, with additions from the mineral kingdom (1807) by William Wood, illustrated by William Daniell. Contributed to BHL by the Field Museum of Natural History Library.

A founding partner supporting BHL’s next chapter

The Field Museum, based in Chicago, Illinois, is one of BHL’s ten original founding institutions and has been part of the consortium since its inception. For nearly two decades, the Field Museum has helped shape BHL’s mission to make biodiversity literature openly available to the world.

“Hosting and supporting BHL’s infrastructure reflects the power of partnership and our commitment to free and open data where appropriate. Grounded in strong technical foundations and a commitment to long-term stewardship, we are reinforcing the secure and scalable technical backbone that sustains one of the world’s most important open biodiversity resources.”
— Rob Zschernitz, Chief Technology Officer, Field Museum

As an independent museum supported by endowments and donor philanthropy, the Field Museum’s mission to connect all of us to nature and the human story aligns closely with BHL’s values and long-term goals.

What this means for BHL users

In practical terms, the Field Museum will:

• Coordinate with BHL and the Smithsonian Institution to relocate BHL’s servers to the Field Museum’s data center
• Provide essential services including domain hosting, systems operations, network security, and incident response
• Serve as custodian and host of BHL’s core technical infrastructure, under the direction of BHL’s Technical Team.

Importantly, BHL will continue to retain ownership of all its data, content, software, and equipment. The Field Museum will act as a trusted custodian, looking after the behind-the-scenes technical infrastructure that keeps BHL online and accessible to millions of users worldwide.

This arrangement builds on BHL’s new operational foundation and supports long-term continuity and resilience.

A carefully managed technical transition

Detailed planning for the move of BHL’s servers is underway. While the exact dates for the physical moves are still to be determined, the teams at BHL, the Smithsonian, and the Field Museum are working closely together to plan the move carefully and responsibly.

Our goal is to minimise disruption to public access. That said, there may be short periods of downtime while systems are moved and reconfigured. Any such interruptions will be temporary, and we will communicate scheduled downtime in advance via the BHL website and social media channels.

Why this matters

Secure, professionally managed infrastructure underpins everything BHL does, enabling global access to biodiversity literature and supporting research, conservation, and education worldwide.

The Field Museum’s commitment to hosting BHL’s technical infrastructure reflects the strength of BHL’s partnerships and depth of support across the BHL consortium.

“The Field Museum has been a valued member of the BHL consortium for many years. From hosting our Life & Literature conference in 2011 to hosting our Annual Meeting in 2015 and co-hosting in 2024, the Field has consistently demonstrated its commitment to BHL and our global community. I am delighted that the Field Museum has agreed to host BHL’s technical infrastructure at this pivotal time in BHL’s history. This represents a major milestone in our transition and reflects the strength and depth of collaboration across the consortium.”
— David Iggulden, Chair, BHL Executive Committee

We are deeply grateful to the Field Museum for stepping forward in this way, and to our global community for your continued trust, engagement, and support.

Continuity, access, and governance

The transition of BHL’s technical infrastructure does not change how BHL is governed or who makes decisions about its future.

The Field Museum’s role in hosting and stewarding BHL’s technical infrastructure does not affect who owns or controls BHL’s content or data. BHL will continue to operate as a member-governed consortium, with strategic oversight provided by the BHL Members Council, in accordance with the BHL Bylaws.

BHL’s collections will remain freely and openly accessible, as they always have been.

Looking ahead: a growing network of support

This hosting transition reflects the depth of BHL’s institutional relationships, its long history of collaboration, and the commitment of trusted partners across the consortium. The Field Museum is just one of the institutions that has stepped up in response to BHL’s Call for Support.

This work is part of a broader, coordinated effort across the BHL community, with multiple organisations offering support in different ways. Stay tuned for additional announcements as details are finalised.

How you can help

You can support BHL by:

• Staying informed: follow #BHLTransition on our website and social media channels
• Sharing our story: help amplify BHL’s Call for Support, and the importance of open biodiversity literature
• Support BHL directly: make a donation. Donations directly support BHL’s operations and long-term sustainability

As always, thank you for being part of the Biodiversity Heritage Library community.

In the past several weeks, we’ve seen a large, disruptive increase in traffic to the BHL main website. This blog post is meant to summarize the event, the effect on BHL and its servers, our response, and what we have planned should it happen again.

The problems experienced on BHL’s website are not related in any way to the transition away from the Smithsonian. We believe the timing of this activity and the resulting downtime to be purely coincidental.

In early June, the Technical Team observed increased load on the full-text-search server, though it remained within acceptable limits and BHL performance was unaffected. Since this is not unusual for short periods of time and the server continued handling traffic, we weren’t  too concerned.

In the week of 7 June, we started to see that BHL was sometimes slower to respond. It was then that we noticed that traffic to the search server was dramatically higher than normal. A quick analysis of the server logs found that BHL was serving more than 10 times the usual number of searches. And more importantly, none of this activity was appearing in Google Analytics. It was clear that we were dealing with traffic from a bot. The bot wasn’t simply performing searches, it was also loading other parts of the sites, usually calling each part with a different taxonomic name each time.

BHL’s Search Traffic slowly increased in April and May.

What is most surprising is that this was not API traffic. BHL’s API provides rich data to anyone who uses it — data that is meant to be consumed by software. This bot was loading the BHL web pages as a person would using their computer’s web browser. It was getting results that were less computer-friendly than it would if it had used the API!

Unfortunately, the traffic was coming from all over the world and the bot was clearly not identifying itself as GoogleBot, BingBot or even OpenAI’s GPTBot. Slowing it down or blocking it entirely was proving to be a challenge. We activated rate limits on the web server, but the distributed nature of the bot prevented that from being effective. We investigated implementing Cloudflare Turnstile, but configuring that on short notice is a challenge. We thought about slowing down all queries to the server, but we knew that would affect all BHL users. In the end, we found that the bots had a sort of “fingerprint” that was different than that of regular users. We were prepared to start limiting traffic using the fingerprint when…. the traffic stopped. On 19 June, BHL started once again responding with its usual speed.

Looking at the graph of how many times the /search URL is called every day, we see some surprising numbers before traffic dropped on 19 June.

BHL’s search traffic from May to June increased dramatically.

In general, BHL’s web infrastructure is able to easily support the current level of approximately 200k-300k searches per day. It is even possible to support up to about 400,000 searches per day, but that’s reaching the limit of what our infrastructure can handle. This is evidence that the hardware and software we have dedicated to search is very robust and well-suited to a busy site.

We haven’t performed an extensive analysis of the sources of this traffic, but at a glance they do seem to be from a variety of locations around the world. While the traffic may have been centered on certain countries or networks (such as the Google Cloud or others), we have no definitive evidence that it’s any one service.

One note: In the second chart, 6 June and 15 June show a much lower number of searches compared to the days before and after. This is not due to the bots slowing down, but instead were caused by BHL itself performing regular weekly and monthly tasks at the same time the bots were busy. This combination of activities overwhelmed the system and BHL was not responding at all.

When we look at the news, we find that our experience mirrors that of other cultural heritage sites, such as those reported in The Register:

Claburn, Thomas. (17 Jun 2025) Bots are overwhelming websites with their hunger for AI data. The Register.

…and also from smaller organizations such as our colleagues at FromThePage:

Brumfeld, Sara. (20 June 2025) Bot Traffic, AI Training, and Infrastructure Strain. FromThePage Blog.

Knowing we are not alone in this challenge is comforting, but it doesn’t eliminate the challenges. We continue to closely monitor BHL’s website performance and remain committed to keeping the website and API up and running as much as possible while we explore the best ways to mitigate the effects on BHL.

Our latest newsletter is now available! From collaborative transcription projects to a new Wikimedian-in-Residence, don’t miss the latest news from the BHL community.

View our December 2024 Newsletter.

Our latest bi-annual newsletter is now available! From flora and fauna in New Zealand to new Affiliates in Belgium and the United States, don’t miss the latest news from the BHL community.

View our November 2023 Bi-annual Newsletter.

Our latest bi-annual newsletter is now available! From BHL Day 2023 in Paris to releasing the new Ukrainian Українська Collection, don’t miss the latest news from the BHL community.

View our June 2023 Bi-annual Newsletter.