Data Centre on Crossref

We’ve migrated to the cloud; we hope you didn’t notice (but maybe you did)

Sara Bowman — Tue, 12 Aug 2025 00:00:00 +0000

TLDR: We’ve successfully moved the main Crossref systems to the cloud! We’ve more to do, with several bugs identified and fixed, and a few still ongoing. However, it’s a step in the right direction and a significant milestone, as, whilst it is a much larger financial investment, it addresses several risks and limitations and shores up the Crossref infrastructure for the future.

Some background

We have been doing a lot of thinking, planning, and working on paying down our technical debt and modernising our systems. It’s not fun and flashy work, but it is vital for sustaining our infrastructure, meeting the demand on existing services, and developing new services.

Just about a year ago, we completed phase one, migrating our main database from Oracle to PostgreSQL, an open-source database. This move brought us more in line with our commitment to the POSI principles, reduced our dependencies on costly private licenses, and opened up the possibility to use and offer additional and more contemporary features. With the transition to PostgreSQL we made upgrades to the operating system, the database software, and the underlying hardware, resulting in significant improvements to the overall throughput and capacity of the deposit system. Previously, we typically maintained a queue of more than 10,000 deposits waiting to be processed; now, the queue holds fewer than 100 deposits on average. Consequently, the average latency – the elapsed time from submission to deposit – has reduced from hours to seconds.

During phase one, a total of 35 new servers were created, and for the first time, the entire system configuration was defined through infrastructure-as-code, enabling the infrastructure to be recreated as necessary. This effort not only enabled the migration but also established a solid foundation for our cloud migration strategy, as the code was leveraged to configure our infrastructure on AWS. Additionally, it serves as a critical component of our disaster recovery planning.

Most importantly, phase one set us up for phase two and our next migration: moving the system into the cloud.

Why we moved to the cloud

We had been running most of our services in a physical data centre near Boston, MA, USA (there are a few exceptions: the REST API and our test system (test.crossref.org) were already in the cloud, as was the Crossref website). We’ve been planning to move to the cloud for ahem quite some time, but as always, competing priorities and limited resources have thwarted us, and the data centre was mainly serving us well.

But… with staff across 12 countries, and increased global use of our system, operating our own hardware in a physical data centre was becoming increasingly challenging and risky, not to mention, frustrating.

Moving to the cloud has solved several pain points for us:

Physical access to the data centre was required for various tasks (e.g., hardware upgrades, troubleshooting, general maintenance), but as Crossref grew as an organisation and became more distributed, we had fewer staff in the area. Hosting services in the cloud means staff around the world can access our servers remotely from anywhere (and we can leave the hardware upgrades to our vendor).
Scalability in the data centre required installing new hardware or upgrading connections, which also meant a good amount of time. In the cloud, we can scale up almost instantly.
We can maintain copies of our databases and services in distributed places, providing insurance against natural or other disasters. Upgrades now don’t involve buying physical hardware and installing it; it’s a much quicker and more straightforward process.

Moving from a physical data centre to the cloud also has some trade-offs; for instance, the cost will be approximately five times higher than running the system in the data centre; with initial data, it’s not unlikely the annual cost may be up to 2,000,000 USD. We aim to optimise and control this cost going forward.

What we did

The size of the undertaking was partly due to leaving it so long; technical debt has accumulated over many years of running the system in the data centre.

The whole plan was hugely detailed, but we can distil it to a few bullets:

We conducted an analysis of components, considered risks and sequencing, and created a test plan and timeline, including comms.
While most of the drive and work was on the shoulders of two infrastructure services colleagues, our software engineers were heavily involved too, and we had weekly check-ins with a cross-team group to review progress, reassess risks, and adjust timelines as we got closer to the migration date (or decided to move it once or twice).
We first created the deposit system in the cloud.
We then created other parts of our services that aren’t in the deposit system code base, but run alongside it, such as reports, querying, and other tools. We replicated our databases (of which there are several, in a few different flavours - PostgreSQL, MySQL).
We gave 14 days’ notice to our members, via email, and kept this maintenance notice up to date.
We commenced the migration on 8th July, which involved taking the whole system down and rejecting deposits for up to 24 hours.
In the process, we scripted the process to create CS and the other services using Terraform and Ansible, so that going forward, bringing up a whole new instance of CS (should we need to) won’t be a manual process.
We moved the DNS to point at our new system in the cloud, rather than the data centre. We brought the system back up on 9th July, after 14 hours of downtime, and watched the first few deposits come in, while testing thoroughly.
Alongside the technical team, the membership and support team was at the ready to work through the testing in the new live production environment.

The message we sent to members, Metadata Plus subscribers, and key integrators like PKP and Turnitin, listed which services would be down and described what changes they might see, such as:

The system timezone shifted from EST to UTC (universal coordinated time), which would be noticeable in the timestamps reported back to members after metadata deposits
Our IP address became dynamic and is no longer static. If members had hardcoded our previous IP static address to connect to our services, that would no longer work.
We previously allowed connections using the HTTP/1.0 protocol, but now require HTTP/1.1. Likewise, we previously allowed TLS version 1.1, but now require at least version 1.2. Older ciphers will not work. A list of accepted ciphers can be found on this page for “ELBSecurityPolicy-TLS13-1-2-2021-06”.

How it went and what’s next

We still have more to do, with both expected and unexpected issues arising from the migration. There are a couple of functions that still route through the data centre, configuration changes to wrangle, and processes to iron out, so we’ll be keeping that open for another couple of months.

Those were the known issues…

…we also uncovered a few bugs along the way, and we’ve been reporting those (and our progress toward fixing them) on our status page. See history.

A few diligent members also alerted us to problems they were having. In some cases, we could tell why, and in many cases, their systems needed to be upgraded to work with ours. Thanks go to mEDRA, Spandidos Publications, and Stichting SciPost who helped us identify gaps that resulted in configuration improvements and lessons learned (that we then shared with other members).

There were three issues that we were contacted about more than others:

Delayed delivery of notification emails which is partly due to the volume of backlogged notification emails in the system.
- Mostly solved: We have repaired delivery of notification emails for all metadata deposits and are working on a fix for the delivery of messages associated with very large queries.
A small percentage of registered records not being indexed in the REST API - this can cause downstream issues for a number of other services (e.g., Crossref metadata search - search.crossref.org, Participation Reports, ORCID auto-update, and for external services that make use of the metadata from our REST API).
- Mostly solved: All records in July are now indexed in the REST API, albeit we have new reports of a few records missing in the last week, which we are actively investigating.
Delayed delivery of July’s resolution reports.
- Solved - not only has July’s resolution report run completed, but we also completed August’s ahead of schedule.

This migration was a significant effort, and 2025’s top priority project for the Open and Sustainable Operations (OSO) program team. Overall, we’re happy with our progress toward making Crossref infrastructure more robust, reliable, and future-proof. And judging by the messages of support we received, you are too! Onwards to the next infrastructure project… check out our roadmap to see what’s up next.

References

‘Infrastructure as code’ (2025) Wikipedia, 12 August. Available at: https://en.wikipedia.org/wiki/Infrastructure_as_code (Accessed: 12 August 2025).
‘The programs approach: our experiences during the first quarter of 2025’ (2025) Crossref. Available at: https://doi-org.pluma.sjfc.edu/10.64000/4s2ee-wkr84 (Accessed: 12 August 2025).

Outage of March 24, 2022

Geoffrey Bilder — Thu, 24 Mar 2022 00:00:00 +0000

So here I am, apologizing again. Have I mentioned that I hate computers?

We had a large data center outage. It lasted 17 hours. It meant that pretty much all Crossref services were unavailable - our main website, our content registration system, our reports, our APIs. 17 hours was a long time for us - but it was also an inconvenient time for numerous members, service providers, integrators, and users. We apologise for this.

Like the outage last October, the issue was related to the data center that we are trying to leave. However, unlike last time, our single nearby network admin wasn’t in surgery at the time. Tim was alerted in the early hours of his morning and was able get up and immediately investigate.

Despite having both secondary and tertiary backup connections, neither activated appropriately.

The problem was with incomplete BGP (Border Gateway Protocol) settings on our primary connection’s network provider’s side. We never noticed this because our backup connection had the correct and complete BGP settings. But our backup circuit went down (we don’t know why yet), and when the router with complete settings went down, only the router with the incomplete settings was available and so everything went down.

We hadn’t yet fully configured the tertiary connection to cut over automatically. This meant cutting over to the tertiary during the outage would have required manual and potentially error-prone reconfiguration. Not something we wanted to do in a hurry with a sleep-deprived network admin.

It’s not an excuse at all. But we are currently down two people in our infrastructure group. One of our infrastructure staff recently left for a startup, and we are already hiring a new third position. In short, our one-long-suffering sysadmin had to field this all by himself. But hey - we are hiring a Head of Infrastructure, and if you are interested you can now see the work you’d have cut out for you!

So things are back up and we’ve resolved the incident but we are carefully and cautiously monitoring. We will further analyze what went wrong and post an update when we have a clearer picture.

I apologize for the downstream pain this outage will have inevitably caused. We realize that many people will now be scrambling to clean things up after this lengthy outage.

More when I have it… but for now I’ll mostly be curled up in a ball.

Update on the outage of October 6, 2021

Geoffrey Bilder — Wed, 27 Oct 2021 00:00:00 +0000

In my blog post on October 6th, I promised an update on what caused the outage and what we are doing to avoid it happening again. This is that update.

Crossref hosts its services in a hybrid environment. Our original services are all hosted in a data center in Massachusetts, but we host new services with a cloud provider. We also have a few R&D systems hosted with Hetzner.

We know an organisation our size has no business running its own data center, and we have been slowly moving services out of the data center and into the cloud.

For example, over the past nine months, we have moved our authentication service and our REST APIs to the cloud.

And, we are working on moving the other existing services too. For example, we are in the midst of moving Event Data and, our next target, after Event Data, is the content registration system.

All new services are deployed to the cloud by default.

While moving services out of the data center, we have also been trying to shore up the data center to ensure it continues to function during the transition. One of the weaknesses we identified in the data center was that the same provider managed both our primary network connection and our backup connection (albeit- on entirely different physical networks). We understood that we really needed a separate provider to ensure adequate redundancy, and we had already had a third network drop installed from a different provider. But, unfortunately, it had not yet been activated and connected.

Meanwhile, our original network provider for the first two connections informed us months ago that they would be doing some major work on our backup connection. However, they assured us that it would not affect the primary connection- something we confirmed with them repeatedly since we knew our replacement backup connection was not yet active.

But, the change our provider made did affect both the backup (as intended) and the primary (not intended). They were as surprised as we were, which kind of underscores why we want two separate providers as well as two separate network connections.

So both our primary and secondary networks went down while we had not yet activated our replacement secondary network.

Also, our only local infrastructure team member was in surgery at the time (He is fine. It was routine. Thanks for asking).

This meant we had to send a local developer to the data center, but the data center’s authentication process had changed since the last time said developer had visited (pre-pandemic). So, yeah, it took us a long time to even get into the data center.

By then, our infrastructure team member was out of surgery and on the phone with our network provider, who realized their mistake and reverted everything. This whole process (getting network connectivity restored, not the surgery) took almost two hours.

Unfortunately, the outage didn’t just affect services hosted in the data center. It also affected our cloud-hosted systems. This is because all of our requests were still routed to the data center first, after which those destined for the cloud were split out and redirected. This routing made sense when the bulk of our requests were for services hosted in the data center. But, within the past month, that calculus had shifted. Most of our requests now are for cloud-based services. We were scheduled to switch to routing traffic through our cloud provider first, and had this been in place, many of our services would have continued running during the data center outage.

It is very tempting to stop this explanation here and leave people with the impression that:

The root cause of the outage was the unpredicted interaction between the maintenance on our backup line and the functionality of our primary line;
Our slowness to respond was exclusively down to one of the two members of our infrastructure staff being (cough) indisposed at the time.

But the whole event uncovered several other issues as well.

Namely:

Even if one of our three lines had stayed active, the routers in the data center would not have cut over to the redundant working system because we had misconfigured them and we had not tested them;
We did not keep current documentation on the changing security processes for accessing the data center;
Our alerting system does not support the kind of escalation logic, and coverage-scheduling that would have allowed us to automatically detect when our primary data center administrator didn’t respond (being in surgery and all) and redirect alerts and warnings to secondary responders; and
We need to accelerate our move out of the data center.

What are we doing to address these issues?

Completing the installation of the backup connection with a second provider;
Scheduling a test of our router’s cutover processes where we will actually pull the plug on our primary connection to ensure that failover is working as intended. We will give users ample warning before conducting this test;
Revising our emergency contact procedures and updating our documentation for navigating our data center’s security process;
Replacing our alerting system with one that gives us better control over escalation rules; and
Adding a third FTE to the infrastructure team to help us accelerate our move to the cloud and to implement infrastructure management best practices.

October 6th, 2021, was a bad day. But we’ve learned from it. So if we have a bad day in the future, it will at least be different.

Outage of October 6, 2021

Geoffrey Bilder — Wed, 06 Oct 2021 00:00:00 +0000

On October 6 at ~14:00 UTC, our data centre outside of Boston, MA went down. This affected most of our network services- even ones not hosted in the data centre. The problem was that both of our primary and backup network connections went down at the same time. We’re not sure why yet. We are consulting with our network provider. It took us 2 hours to get our systems back online.

We are going to reprocess content that was in the process of being registered at the time of the outage in order to make sure everything gets registered correctly. This may take a few days to complete.

Why did we have such a complete outage and why did it take us so long to fix it?

We still run a significant amount of our infrastructure in a data centre outside of Boston that we manage ourselves. Even though we’ve been moving many of our services to the cloud, all our traffic was still routed through the data centre - so when it went down, most of our cloud services were unavailable as well.
It took us a long time to fix this because our infrastructure team only has two people in it. Only one of them is located near the data centre and was at the doctor’s when the outage occurred. Although we were alerted to the problem immediately, we had to send one of our development team members to the data centre to diagnose and fix the problem.

We have been aware of these weaknesses in our system since I took the role of director of technology in 2019, and we have been putting most of our efforts over the past two years into fixing them.

We know that an organisation of our size has no business trying to run and maintain a physical data centre ourselves. One of the strengths of cloud-based systems is that they can be administered from anywhere and don’t require anyone to physically go to a data centre to replace failed hardware or check that network connections are, in fact, live. We’ve been trying to move to the cloud as fast as we can. All new services that we build are cloud-based. At the same time we’ve been moving systems out of the data centre - starting with those that put the biggest load on our systems. To further aid this process we have budgeted to add an FTE to the infrastructure team in 2022.

What is really painful about this event is that we had just completed the last bit of work we needed to do before changing our traffic routing so that it would hit the cloud first instead of the data centre first. This would not have avoided the outage we just experienced, but it would have made it a bit less severe.

What is even more painful is that we had recently installed a third network connection with an entirely different provider because we were worried about just this kind of situation. But this third connection wasn’t yet active.

We already have a long list of tickets that we’ve created to address problems we faced in recovering from this outage. The list will undoubtedly grow as we complete a postmortem over the next few days. I will report back when we have more detail of what happened and have a solid plan for how to avoid anything similar in the future.

We know that an outage of this severity and duration has caused a lot of people who depend on our services extra work and anxiety. For this, we apologise profusely.

But at least we didn’t need to use an angle grinder.