Infrastructure on Crossref

Schema 5.5 now available: adding CRediT, new record types for blogs and posters, and more

Patricia Feeney — Thu, 09 Jul 2026 00:00:00 +0000

Research is rarely limited to a single contributor performing a single role. Behind every research output are people contributing in various ways: software development, data analyses, methodology design, and much more. Often, the same person contributes in several of these ways. Until now, Crossref metadata could only capture part of that picture, but this is changing with Schema 5.5.

Crossref Schema 5.5 includes several improvements across different content types, but its most significant enhancement is the expanded support for contributor roles through the introduction of multiple roles per contributor, option to specify the corresponding author, and compatibility with the CRediT (Contributor Roles Taxonomy): a community-owned taxonomy of 14 contributor roles, which has been adopted and made available in multiple languages.

These enhancements allow members to describe research contributions in much greater detail, creating richer metadata that better reflects how research is actually produced, and supporting greater accountability and more comprehensive research assessment.

If your workflow already distinguishes between different kinds of contributions, Schema 5.5 gives you a way to record that detail more accurately using the CRediT taxonomy values. CRediT can be adopted gradually, where it fits your editorial or production workflow.

Figure 1: Until now, contributors could be assigned a single contributor role using Crossref’s existing contributor role vocabulary. In Schema 5.5, members can indicate that the same contributor was responsible for different roles, such as corresponding author; writing: reviewing and editing; and data curation.

Existing deposits remain fully supported, and members can continue using the current contributor role attribute while planning implementation of the new repeatable role type element. For our members, who have been using CRediT in their workflows already, as ever – we encourage updating your metadata when practicable.

Why this update is kind of a big deal

This update gives more accurate credit to all of the people behind research outputs. Crossref vocabulary includes roles that aren’t recognised in CRediT, and vice versa. Capturing richer contributor metadata recognises contributions that may not be visible in a single author line and improves transparency around how research is produced, thereby enabling downstream systems to interpret that information more reliably. The update also offers better interoperability with CRediT, which is well recognised across the scholarly ecosystem.

Figure 2: Schema 5.5 is an expansion of Crossref contributor metadata. Members can describe contributors using Crossref’s existing contributor role vocabulary, as well as the internationally recognised CRediT taxonomy.

In turn, this strengthens metadata reuse across repositories, discovery services, funders, institutions and other infrastructure providers; and supports evaluation, reporting and discovery workflows. Better contributor metadata strengthens the connections that make up the Research Nexus.

What else is included in Schema 5.5?

Beyond the expanded contributor support, Schema 5.5 includes several additional enhancements across the metadata schema.

1. Updates to report series metadata

Support has been added for metadata elements that were previously missing from report series records, including Crossmark, funding, and licence information.

2. Posted content improvements: now including blogs and posters

Posted content includes preprints, eprints, and other types of content that have been posted to a stewarded host platform. We’re all about persistence, so it’s vital that everything registered with us be maintained. Note that accepted manuscripts are not considered posted content. Schema 5.5 refreshes posted content sub-types by introducing blog and poster.

At the same time, we are “retiring” working paper, dissertation, and report from posted-content sub-types. Over time, these have been developed into separate record types that benefit from richer, dedicated schemas.

Finally, archive locations can now also be included for posted content records.

3. Expanded archive support

A new archive location, CINES, has been added to the list of supported archive providers.

4. Clinical trial metadata across more record types

Clinical trial information is no longer limited to journal articles and conference papers. Schema 5.5 extends support across additional content types, including books, datasets, dissertations, reports, posted content, standards, and pending publications.

Schema adoption

Taken together, the updates in our latest schema support more holistic recognition of contributions to the research and its communication, as well as greater accountability and integrity in related processes.

To support gradual adoption, Schema 5.5 maintains backwards compatibility with existing deposits. Members can continue using the current contributor_role attribute while preparing to implement the new repeatable role element. We have prepared a migration guide to help members transition to Schema 5.5.

As you prepare to adopt Schema 5.5, we encourage members to include contributor roles whenever they are available from editorial workflows and to use recognised vocabularies consistently, including CRediT roles where appropriate.

Building better connections: the story of Crossref's metadata development

Patricia Feeney — Thu, 18 Jun 2026 00:00:00 +0000

Three years ago, we asked our members what they needed from Crossref’s metadata. We received confirmation that we were going in the right direction, as well as some new ideas to explore. This helped set the course for our metadata development work since then, and continues to guide where we’re headed next.

Every metadata update we make is driven by the same set of priorities: supporting metadata that reflects our organizational truths, focusing on what metadata our members can actually provide, and aligning with best practices, vocabularies, and standards that our wider scholarly community has established. More recently our Metadata Advisory Group has helped us explore both the minutia of working with metadata as well as larger ideas around the value and impact of the metadata we support.

What We’ve Accomplished

Our schema 5.4 update included several new or expanded types of metadata. First, citation metadata can now be labelled with a publication type. This means when a work cites an article, a preprint, a dataset, or software, that distinction is clear, helping make citations without an accompanying DOI metadata record easier to identify. Second, version information is now supported across all record types, giving the scholarly record a more precise handle on exactly which version of a work is being described.

We’ve also made two meaningful improvements to how funding relationships are captured. ROR IDs are now supported as funder identifiers in both our standard metadata schema and our grants-specific schema. Also, Grant DOIs can now be explicitly identified within funding metadata, making it possible to draw clearer lines between research outputs and the grants that supported them.

What’s happening now

A significant update is nearly here. Schema 5.5 will expand contributor metadata to support multiple roles per contributor, and will introduce support for CRediT — the ANSI/NISO taxonomy for contributor roles. This means that an individual’s complete contribution to a research output can finally be described in our metadata, rather than flattened into a single role or omitted entirely. The schema isn’t released yet, but the final version of the XML schema is available in our GitLab repository for those who want to get a head start.

We’ll next begin implementation work for a new Grants schema (0.3.0). This update will remodel investigator names to include a new role (beneficiary) as well as an organizational grant recipient, making it possible to include recipient info for grants given to organizations. Grant records include project metadata, so this update will also include support for RAiD, a persistent identifier for projects. The XML schema for this update is also available in a GitLab repository.

What’s up next

Our next planned major update will build substantially on the contributor work in version 5.5. In the next version (6.0) we will remodel names to expand our current limited structure to support a variety of name types as well as alternate names. We’ll also expand the contributor identifiers we collect to include ISNI and Wikidata identifiers, better supporting contributors for whom an ORCID is not possible. Our organizational contributor will be remodelled as well to include organization-level identifiers like ROR.

We’ll also introduce statements to Crossref metadata. Statements will allow members to include free-text statements including funding acknowledgements, ethics declarations, AI usage disclosures, and other important contextual information that doesn’t fit neatly into structured fields.

Other updates include expanding our support for abstracts encoding beyond JATS to include ONIX, BITS, and a generic markup option, and implementing better in-schema validation to avoid surprises at the time of deposit.

Progress means letting go of the past. We’re planning to deprecate all schemas prior to version 5.3.1 by the end of 2027, to be carried out in phases as outlined in our deprecation blog post. This is a necessary step to keep our infrastructure sustainable and to ensure members are working with schemas that reflect current capabilities and standards.

Looking further ahead

Beyond 6.0, we’re exploring further support for provenance in metadata (to establish who is doing what to a metadata record), a rethinking of how we handle dates so that they better capture the lifecycle of a research object, better support for research objects we don’t yet fully support, and making our metadata inputs more consistent. The Metadata Development roadmap has full details on what’s being explored and prioritized.

Each of these updates contributes to Crossref’s research nexus vision: strengthening connections between funders and research, more accurately capturing and recognizing contributor roles in the scholarly record, and collecting free-text content to fill in the gaps that structured metadata alone can’t address. Better metadata means better research integrity and more trustworthy infrastructure for everyone who depends on it.

Collaboration with Knowledge Futures to build support for high-volume DOI registration

Travis Rich — Mon, 08 Jun 2026 00:00:00 +0000

Cross-posted from the Knowledge Futures blog.

For many years, PubPub has made it possible for communities to assign DOIs to a range of outputs and component Pubs. Knowledge Futures and Crossref are building together to test the limits of what’s possible for high-volume, high-granularity DOI management. That means fast prototypes, real building, and learning through the process.

What this looks like

We’re starting by building. The goal is to get working prototypes in front of real use cases as quickly as we can, and let the technical, UX, operational, and infrastructure questions get answered through that process. What does it take to register and manage DOIs at a level of volume and granularity that goes beyond what most existing tools support? We’ll find out by trying.

A broader orientation for KF

This is also an example of where Knowledge Futures is headed more generally. We’re taking what we’ve learned from building publishing infrastructure and applying it across different parts of the scholarly communication ecosystem. Not siloed within PubPub development, but open to building more broadly and collaborating across organizational lines.

We’ve spent close to a decade learning what it takes to build and maintain reliable infrastructure for knowledge communities. That experience doesn’t have to live inside one product. We think working this way puts us in a stronger position as stewards of the things we maintain, and it opens the door to more collaboration across the ecosystem.

Aligned direction with Crossref

As Crossref adoption has skyrocketed, enabling DOIs for a vast range of research objects and organizations, they are looking to support these objects at scale and further upstream than traditional outputs. Alongside its fee remodelling effort begun in 2023, Crossref is backing this work with a $258k investment, partnering with Knowledge Futures to explore new models for the future of open research infrastructure.

Get involved

We’d love to hear your thoughts about high-volume, high-granularity DOIs. What’s your use case? What would it unlock for your community? Want to be involved in the design process? This collaboration with Crossref is just one piece of where we’re headed. If you’re curious about what we’re up to, or have something you’d like to share with us, get in touch. We’d love to tell you what we’re working on and hear what excites you too.

Hit refresh: redesigning our technical infrastructure

Paul Davis — Wed, 11 Mar 2026 00:00:00 +0000

With key milestones achieved in 2025, including the appointment of new Directors of Technology and Programs, a move to the cloud, and some key schema updates, we now have a firm foundation for our next challenge: a redesign of our core technical systems to make them more modern, robust, and easier to maintain and scale.

At a high level, our systems serve our community well. The deposit system handles over 107,000 DOI deposits and updates per day, and the REST API responds to 2 billion requests per month, serving up nearly 180 million open metadata records. These systems are reliable: since December 2025, the REST API pools exceeded 99.94% uptime, and the submission queue, since January 2024, has had an uptime of 99.90%.

Uptime for REST API pools and the submission queue, with minimal service interruption since January 2024.

It’s this reliability, we think, that has kept us from tackling this redesign earlier. But the reality is that when these systems were first built, some as long ago as 2005, Crossref and our community looked very different. In 2005, we had 318 members, and were creating DOIs and depositing metadata exclusively for journal articles, all supported by 5 Crossref staff. Today, we have 24,000+ members, representing publishers, societies, funders, universities, service providers, sponsors, and sponsored members; and we are assigning DOIs to >17 content types, from journal articles to book chapters, grants, conference proceedings, dissertations, and more. All supported by 50+ Crossref staff.

Like a stone monolith, a tightly coupled system can be solid, but hard to adapt.

Over the years, we’ve accumulated quite a bit of technical debt building support for new features and functions into one monolithic codebase.

We haven’t always been great at paying down that technical debt (because, as some say, if it ain’t broke don’t fix it), but we have made big strides in the last 18 months. Our main database moved to Postgres from Oracle and we migrated from a physical data centre to the cloud.

But still, the codebase has become hard to maintain; it’s not always clear that fixing something in one place won’t break something else. A more modern approach, breaking apart the monolith into separate, smaller services, will enable us to seamlessly maintain services, identify and debug issues efficiently, and build new features to support the ever-changing needs of our growing community.

Paul has recently started a new role at Crossref as the Product Manager for the Open and Sustainable Operations Program and will play a big role in this cross-functional effort. Having moved over from the technical support team, he brings a wealth of experience with all of our systems, how they work in detail, and where the pain points exist for internal users as well as members. We will rely on this experience to bring a better suited system to our members and colleagues.

Oh, the things he’s seen (and heard!). Here’s what rises to the top of his pain points list:

Our authentication service is difficult to administer with manual processes still in place for change requests
Title management causes headaches for our members and staff: members can’t modify or transfer titles between each other in our system themselves and rely on manual intervention from the Support team.
Members don’t have access to all of the details about their own records that we have in our system. This creates unnecessary barriers to stewarding their metadata.
Members can’t currently programmatically check the status of their submission in the system to learn in real-time whether it has been deposited or remains in the queue, which would be really useful.

How are we going to do this?

This won’t be one big bang – the scale is way too big, and it’s far too risky. We will instead break the work into a series of (many!) smaller projects, chipping away at the large monolith of Crossref code and building smaller, free-standing components which will be easier to maintain. We also don’t see this work as a separate project with a cleanly defined beginning and end - rather, gradually replacing parts of the system with more modern, better-designed components is simply part of ongoing maintenance.

We’ve already taken a hard look at what our system does (and it’s a lot!) and developed a list of ~14 core “functions” it serves, things like authentication and authorization, metadata deposit and validation, metadata distribution, and so on. We’ll work on replacing those functions with free-standing services, then pull out the (then-unnecessary!) code from the codebase. At each step, the monolith gets smaller and less complex.

We’ve already started doing this, and it feels great! Most recently, we rebuilt a component we lovingly call the ‘Pusher.’ Its function is to push the XML our members deposit to the REST API, where it can be distributed to users, and to keep citation counts updated. We deployed the new Pusher in two phases, in October 2025 and February 2026. It uses modern code libraries, is open source, and runs independently rather than being tangled up with the rest of the core system.

Another project that is currently underway is rebuilding of our metadata matching workflows using modern software development and data science practices. The goal is to create a dedicated, consolidated matching service that will eventually replace all existing matching processes, with results made available through the REST API. This project will start with matching funder names to ROR IDs, and eventually cover also bibliographic reference matching, preprint matching, affiliation matching, grant matching, and title matching.

We have yet to decide on the exact order of things, but that will likely be determined by the complex dependencies we touched on earlier. We’ll also consider urgency and risk - is something falling over too often and causing too much work to maintain it and keep it stable? That’s the reason we bumped up the priority on the Pusher work and rewrote it when we did. We’ll also consider benefits: quicker upgrades which will help both ourselves and our members will naturally have a higher priority than less impactful projects.

We do know that the top priority is the authentication service. The current one was not meant to be permanent, but rather a bridge until we build a permanent solution. It’s now beyond its useful life: it’s quite confusing, it doesn’t scale (and we are scaling), and it’s painful for sponsors, members, and Crossref staff. Tackling this piece of work first unblocks a lot of other important things we want to get done. It’s a big undertaking that we are excited to get started on to improve the user experience for everyone. Importantly, we will be consulting with the community along the way so that we get this right.

Dominika Tkaczyk, Crossref Director of Technology, emphasized the importance of maintaining open scholarly infrastructure at the Crossref annual meeting in October 2025. She said that, just as we maintain roads and bridges to make sure they’re safe, we must maintain scholarly infrastructure so it continues to serve the community far into the future.

This is a long road, but it’s one we’re excited to be on. We’ll have periodic updates on our progress as this work goes forward: what we’re getting done, where we need input, and what we’re tackling next. We’ll need a lot of feedback from you, our community, about what’s working well and where we might make improvements.

The sunset is on the horizon for Metadata Manager. What's next?

Lena Stoll — Thu, 06 Nov 2025 00:00:00 +0000

TL;DR. Metadata Manager will be retired at the end of 2025. Over the past four years, we have been developing a new helper tool to replace it, and that tool has now reached a stage of maturity that means we will be able to switch off Metadata Manager by the end of the year.

How did we get here?

In 2021, we said that we would be retiring the deprecated Metadata Manager as soon as we can offer members a suitable replacement for registering their journal content. So this news has been a long time coming - Metadata Manager has been very challenging for us to support, and we have found it impossible to develop additional features. However, we did not want to take the final step of switching off the interface until we were able to offer a suitable replacement for members who rely on manual helper tools to register their journal content.

That replacement, our new record registration form, has now been used by many members for over a year to register their journal content. The feedback so far has been positive, and we have been able to add functionality to the tool at a pace that we are happy with.

In July 2025, we contacted those members who are still using Metadata Manager to let them know that the tool will no longer be available after December 2025. So if you are affected by this news, you were probably already aware of it. But we wanted to go into a little more detail on the sunsetting of Metadata Manager, why we are doing it, and what’s next for Crossref’s content registration helper tools.

What has happened since 2021?

We have been developing the record registration form ever since that announcement in 2021. It began its life as a helper tool for registering grant records, but we knew we wanted to expand it to cover journal articles and other record types as soon as we could.

To see whether the concept behind the grants form could be applied to journal content, we first built an initial prototype and tested it with a number of Crossref ambassadors and volunteers. We wanted to ensure that the tool was intuitive to use, and to understand what functionality it would need to support for it to be truly useful to our members. Following some iteration on the invaluable feedback we received from our testers, we finally released the tool to production in September 2024 and began encouraging members to use it for their real-life article deposits.

We have been continuously adding new functionality since then, from additional fields for registering richer metadata to a feature that allows members to edit their articles’ metadata without having to re-enter everything into the form.

Now, about two months from the target date for retiring Metadata Manager, the record registration form is used by members to register about 200 articles per day, while Metadata Manager still sees about double that volume of submissions. So we have some way left to go.

Why is now the right time to retire Metadata Manager?

2025 has been a year of addressing technical debt for Crossref. My colleague Sara wrote about this co-ordinated push towards modernising our system in her post about our cloud migration in the summer.

Having the long-awaited replacement for Metadata Manager in place will allow us to free up the resources that have been tied up for years by troubleshooting Metadata Manager, in terms of both technology and user support, so that we can focus on projects and initiatives that align with our longer-term strategy.

How will we avoid the new tool developing the same problems as Metadata Manager?

As stated above, Metadata Manager has caused us many issues and headaches in different ways - but we have also learned a lot from dealing with these problems. As Bryan Vickery wrote in 2020, Metadata Manager is “not flexible enough to easily add other record types, like books/book chapters, or to include any changes we may make to our input schema.” To address this, we built the record registration form in a schema-driven way, which makes it adaptable to any future schema changes. It also means that we can spin up prototypes of new forms for additional record types quite quickly.

So while Metadata Manager was custom-built in a way that could only ever work for journal content, the record registration form already supports two record types and will support more in future. This is key for our goal of building a complete research nexus, which extends far beyond journal content, and even beyond “content” as such (did someone say grants?).

What happens next?

Metadata Manager will no longer be available from January 2026.
Starting next year, if you attempt to access Metadata Manager at https://www-crossref-org.pluma.sjfc.edu/metadatamanager/, you will be redirected to a deprecation note on https://www-crossref-org.pluma.sjfc.edu/deprecated/ which will link out to the new tool.

What options do I have for registering my journal content going forward?

If your organisation still uses Metadata Manager to register metadata for your journal articles, now is a good time to begin familiarising yourself with the alternatives available to you from 2026 forward - these include, but are not limited to, the new record registration form.

If your journal has an ISSN

We recommend you begin using the record registration form as soon as possible. Simply go to https://manage-crossref-org.pluma.sjfc.edu/records and sign in with your Crossref account credentials to register a journal article. You can also see a list of all the journal article records you have previously registered using our manual helper tools at https://manage-crossref-org.pluma.sjfc.edu/records/edit and edit their metadata using the form.

To help you make the switch from Metadata Manager, we will be hosting an interactive webinar on 13 November about how to transition to the new tool. Register here or look out for the recording, which will be shared in our events archive.

If your journal does not have an ISSN

The record registration form currently only supports ISSNs as journal identifiers. Title-level and volume/issue-level DOIs, which are at the core of how Metadata Manager handles journal metadata, have been the cause for some of the problems we have had over the years with that particular tool. Also, Crossref DOIs have always been intended primarily as citation identifiers, and entire journals/volumes/issues are very rarely cited. For that reason, we built the Record Registration Form such that it doesn’t support registering or using journal-level DOIs.

With that being said, if you do not (yet) have an ISSN for your journal for whatever reason, you can use our web deposit form to register your articles with journal DOI. If you do obtain an ISSN for your title later on, you can then simply begin using the record registration form from that point onward.

How will the new tool continue to be developed?

We will continue to work with our members and community to develop additional functionalities for the journal article form. Currently we are working on allowing relationships metadata to be registered using the form.

Ultimately, the goal is for the record registration form to become the one-stop shop for members who manually register and update their metadata. To this end, we are working on expanding the tool to cover additional record types - we have recently developed a prototype for registering books and chapters, and we will be looking to test this in the coming months with volunteers who are currently registering their book metadata via other avenues such as the web deposit form.

If you would like to support these efforts, or you have begun using the new tool and would like to share your feedback, come join the discussion in our community forum.

References

Bowman, S. (2021). Next steps for Content Registration. Crossref. https://doi-org.pluma.sjfc.edu/10.64000/30vzx-r5x16
Bowman, S. (2025). We’ve migrated to the cloud; we hope you didn’t notice (but maybe you did). Crossref. https://doi-org.pluma.sjfc.edu/10.64000/wd6rx-vpq73
Vale, P. (2022). Forming new relationships: Contributing to Open source. Crossref. https://doi-org.pluma.sjfc.edu/10.64000/cvq2e-q8t24

We’ve migrated to the cloud; we hope you didn’t notice (but maybe you did)

Sara Bowman — Tue, 12 Aug 2025 00:00:00 +0000

TLDR: We’ve successfully moved the main Crossref systems to the cloud! We’ve more to do, with several bugs identified and fixed, and a few still ongoing. However, it’s a step in the right direction and a significant milestone, as, whilst it is a much larger financial investment, it addresses several risks and limitations and shores up the Crossref infrastructure for the future.

Some background

We have been doing a lot of thinking, planning, and working on paying down our technical debt and modernising our systems. It’s not fun and flashy work, but it is vital for sustaining our infrastructure, meeting the demand on existing services, and developing new services.

Just about a year ago, we completed phase one, migrating our main database from Oracle to PostgreSQL, an open-source database. This move brought us more in line with our commitment to the POSI principles, reduced our dependencies on costly private licenses, and opened up the possibility to use and offer additional and more contemporary features. With the transition to PostgreSQL we made upgrades to the operating system, the database software, and the underlying hardware, resulting in significant improvements to the overall throughput and capacity of the deposit system. Previously, we typically maintained a queue of more than 10,000 deposits waiting to be processed; now, the queue holds fewer than 100 deposits on average. Consequently, the average latency – the elapsed time from submission to deposit – has reduced from hours to seconds.

During phase one, a total of 35 new servers were created, and for the first time, the entire system configuration was defined through infrastructure-as-code, enabling the infrastructure to be recreated as necessary. This effort not only enabled the migration but also established a solid foundation for our cloud migration strategy, as the code was leveraged to configure our infrastructure on AWS. Additionally, it serves as a critical component of our disaster recovery planning.

Most importantly, phase one set us up for phase two and our next migration: moving the system into the cloud.

Why we moved to the cloud

We had been running most of our services in a physical data centre near Boston, MA, USA (there are a few exceptions: the REST API and our test system (test.crossref.org) were already in the cloud, as was the Crossref website). We’ve been planning to move to the cloud for ahem quite some time, but as always, competing priorities and limited resources have thwarted us, and the data centre was mainly serving us well.

But… with staff across 12 countries, and increased global use of our system, operating our own hardware in a physical data centre was becoming increasingly challenging and risky, not to mention, frustrating.

Moving to the cloud has solved several pain points for us:

Physical access to the data centre was required for various tasks (e.g., hardware upgrades, troubleshooting, general maintenance), but as Crossref grew as an organisation and became more distributed, we had fewer staff in the area. Hosting services in the cloud means staff around the world can access our servers remotely from anywhere (and we can leave the hardware upgrades to our vendor).
Scalability in the data centre required installing new hardware or upgrading connections, which also meant a good amount of time. In the cloud, we can scale up almost instantly.
We can maintain copies of our databases and services in distributed places, providing insurance against natural or other disasters. Upgrades now don’t involve buying physical hardware and installing it; it’s a much quicker and more straightforward process.

Moving from a physical data centre to the cloud also has some trade-offs; for instance, the cost will be approximately five times higher than running the system in the data centre; with initial data, it’s not unlikely the annual cost may be up to 2,000,000 USD. We aim to optimise and control this cost going forward.

What we did

The size of the undertaking was partly due to leaving it so long; technical debt has accumulated over many years of running the system in the data centre.

The whole plan was hugely detailed, but we can distil it to a few bullets:

We conducted an analysis of components, considered risks and sequencing, and created a test plan and timeline, including comms.
While most of the drive and work was on the shoulders of two infrastructure services colleagues, our software engineers were heavily involved too, and we had weekly check-ins with a cross-team group to review progress, reassess risks, and adjust timelines as we got closer to the migration date (or decided to move it once or twice).
We first created the deposit system in the cloud.
We then created other parts of our services that aren’t in the deposit system code base, but run alongside it, such as reports, querying, and other tools. We replicated our databases (of which there are several, in a few different flavours - PostgreSQL, MySQL).
We gave 14 days’ notice to our members, via email, and kept this maintenance notice up to date.
We commenced the migration on 8th July, which involved taking the whole system down and rejecting deposits for up to 24 hours.
In the process, we scripted the process to create CS and the other services using Terraform and Ansible, so that going forward, bringing up a whole new instance of CS (should we need to) won’t be a manual process.
We moved the DNS to point at our new system in the cloud, rather than the data centre. We brought the system back up on 9th July, after 14 hours of downtime, and watched the first few deposits come in, while testing thoroughly.
Alongside the technical team, the membership and support team was at the ready to work through the testing in the new live production environment.

The message we sent to members, Metadata Plus subscribers, and key integrators like PKP and Turnitin, listed which services would be down and described what changes they might see, such as:

The system timezone shifted from EST to UTC (universal coordinated time), which would be noticeable in the timestamps reported back to members after metadata deposits
Our IP address became dynamic and is no longer static. If members had hardcoded our previous IP static address to connect to our services, that would no longer work.
We previously allowed connections using the HTTP/1.0 protocol, but now require HTTP/1.1. Likewise, we previously allowed TLS version 1.1, but now require at least version 1.2. Older ciphers will not work. A list of accepted ciphers can be found on this page for “ELBSecurityPolicy-TLS13-1-2-2021-06”.

How it went and what’s next

We still have more to do, with both expected and unexpected issues arising from the migration. There are a couple of functions that still route through the data centre, configuration changes to wrangle, and processes to iron out, so we’ll be keeping that open for another couple of months.

Those were the known issues…

…we also uncovered a few bugs along the way, and we’ve been reporting those (and our progress toward fixing them) on our status page. See history.

A few diligent members also alerted us to problems they were having. In some cases, we could tell why, and in many cases, their systems needed to be upgraded to work with ours. Thanks go to mEDRA, Spandidos Publications, and Stichting SciPost who helped us identify gaps that resulted in configuration improvements and lessons learned (that we then shared with other members).

There were three issues that we were contacted about more than others:

Delayed delivery of notification emails which is partly due to the volume of backlogged notification emails in the system.
- Mostly solved: We have repaired delivery of notification emails for all metadata deposits and are working on a fix for the delivery of messages associated with very large queries.
A small percentage of registered records not being indexed in the REST API - this can cause downstream issues for a number of other services (e.g., Crossref metadata search - search.crossref.org, Participation Reports, ORCID auto-update, and for external services that make use of the metadata from our REST API).
- Mostly solved: All records in July are now indexed in the REST API, albeit we have new reports of a few records missing in the last week, which we are actively investigating.
Delayed delivery of July’s resolution reports.
- Solved - not only has July’s resolution report run completed, but we also completed August’s ahead of schedule.

This migration was a significant effort, and 2025’s top priority project for the Open and Sustainable Operations (OSO) program team. Overall, we’re happy with our progress toward making Crossref infrastructure more robust, reliable, and future-proof. And judging by the messages of support we received, you are too! Onwards to the next infrastructure project… check out our roadmap to see what’s up next.

References

‘Infrastructure as code’ (2025) Wikipedia, 12 August. Available at: https://en.wikipedia.org/wiki/Infrastructure_as_code (Accessed: 12 August 2025).
‘The programs approach: our experiences during the first quarter of 2025’ (2025) Crossref. Available at: https://doi-org.pluma.sjfc.edu/10.64000/4s2ee-wkr84 (Accessed: 12 August 2025).

Crossref at 25: a short film on a quarter-century of open infrastructure

Crossref team — Sun, 19 Jan 2025 00:00:00 +0000

Crossref at 25

Crossref was incorporated on 19 January 2000 by a small group of publishers who agreed that persistent links between scholarly works were too important to leave to any one of them alone. Twenty-five years on, this short film looks back at what that founding decision made possible — and forward to the next quarter-century of open infrastructure.

Watch the film

Crossref at 25 — a short film marking the organisation's 25th anniversary, 19 January 2025

What the film covers

The founding moment — why eight publishers came together in 2000 to register the first Crossref DOIs and run a shared linking infrastructure
A growing network — how membership expanded from those eight founders to over 25,000 organisations in 167 countries, across every type of scholarly publisher
From DOIs to the Research Nexus — how the work has grown from persistent links to a connected graph of works, people, organisations, funders, and outputs
POSI in practice — what it means to run not-for-profit, community-governed infrastructure for the long term
The next 25 years — what the community is building together: richer metadata, broader participation, and a more equitable scholarly record

Anniversary context

Event	Date
Crossref incorporated	19 January 2000
First DOIs registered	2000
10th anniversary	2010
25th anniversary	2025
Members today	25,000+ across 167 countries

For the Crossref 10 years retrospective publication (2010, English and Japanese), see the parallel anniversary booklet from the 10-year mark.

Ed Pentz on building open scholarly infrastructure: a journey of collaboration and diplomacy

Ed Pentz — Fri, 01 Nov 2024 00:00:00 +0000

Ed Pentz on building open scholarly infrastructure: A journey of collaboration and diplomacy

This article expands on Ed Pentz’s Miles Conrad Award lecture at NISO Plus 2024. Drawing on three decades in scholarly publishing — including 24 years as Crossref’s Executive Director — it explores the collaboration, diplomacy, and key inflection points that shaped open scholarly infrastructure, and reflects on the challenges and opportunities ahead.

Strategists

Understand why collaboration and diplomacy are the foundation of open infrastructure. How inflection points create windows for change — and why patience, trust-building, and non-profit governance are what make those changes stick.

Decision-makers

Learn the principles behind Crossref’s founding and longevity. Why collective action, mission-aligned sustainability, and the right governance model matter more than technology alone — with hard-won lessons from 24 years of building open infrastructure.

Practitioners

Trace the technical and organisational milestones from DOI-X to open metadata. From the first cross-publisher reference linking experiments to POSI, ROR, and fully open metadata for 180 million research outputs.

What this paper covers

Inflection points — the moments that catalysed major change: the World Wide Web, DOI-X, open access, and more
The founding of Crossref — why collective action, not technology alone, was the solution to cross-publisher reference linking
Key lessons — focus on the problem, build trust, develop a sustainability model, and be prepared to compromise
Principles of Open Scholarly Infrastructure (POSI) — why governance and sustainability matter as much as technical standards
Diversity, equity, and inclusion — how these principles have shaped Crossref’s culture and the future of open infrastructure
Future challenges — AI, metadata quality, and the evolving landscape of scholarly communication

Read the full paper

Loading…

Celebrating five years of Grant IDs: where are we with the Crossref Grant Linking System?

Kornelia Korzec — Mon, 01 Jul 2024 00:00:00 +0000

We’re happy to note that this month, we are marking five years since Crossref launched its Grant Linking System. The Grant Linking System (GLS) started life as a joint community effort to create ‘grant identifiers’ and support the needs of funders in the scholarly communications infrastructure.

The system includes a funder-designed metadata schema and a unique link for each award which enables connections with millions of research outputs, better reporting on the research and outcomes of funding, and a contribution to open science infrastructure. Our first activity to highlight the moment was to host a community call last week where around 30 existing and potential funder members joined to discuss the benefits and the steps to take to participate in the Grant Linking System (GLS).

Some organisations at the forefront of adopting Crossref’s Grant Linking System presented their challenges and how they overcame them, shared the benefits they are reaping from participating, and provided some tips about their processes and workflows.

The funding organisations whose experiences were shared included Wellcome, FCT (Foundation for Science and Technology, Portugal), and NWO (Dutch Research Council). They were joined by a new group of foundations, research councils, and private research funders from around the world—from Kenya to Singapore to Estonia—to have a first introduction to the GLS and connect them with colleagues who are further along on their journey.

We also heard about tools such as a new open source Crossref plugin for the Fluxx platform, grant management systems with in-built Crossref integrations such as ProposalCentral, Europe PMC GrantFinder which was first to implement the GLS on Wellcome’s behalf and hosts their grants, and one of the first publishers, eLife to start referencing Crossref grant links in their publications both online and in the open metadata for others to retrieve.

Read on for further information or watch the recording of the event.

What is the Crossref Grant Linking System?

The Crossref Grant Linking System, conceptualised in 2017, and launched in 2019, captures and helps clarify funding relationships for scholarly outputs. Thanks to interconnectedness with the 160 million metadata records collected and curated by Crossref members, it enables funders as well as scholars to track and analyse funding patterns and evaluate programmes, and it supports assertions about the integrity of scholarly records.

Features of the GLS

Globally unique persistent link and identifier for each grant
Connected with 160 million published outputs
Funder-designed metadata schema, including project, investigator, value, and award-type information
Programmatic or no-code methods to send metadata
- Thanks to the Gordon and Betty Moore Foundation who funded development of the online grant registration form
Open search and API for all to discover funding outcomes; all metadata is distributed openly to thousands of tools and services
Crossref-hosted landing pages
A global community of ~50 funder advisors and 35+ funders already in the Grant Linking System
Membership of Crossref; influence the foundational infrastructure powering open research

The last five years has seen the GLS grow through membership, metadata, and community contributions.

The momentum for this programme is building - as illustrated by increasing numbers of metadata records (and related relationships we’re seeing). The 35 funder members represent over 100 funding programmes and have created 125,000 grant records already.

During last week’s call, it was helpful to hear from the community what they see as key benefits of the Crossref Grant Linking System:

Meaningfully delivering on and supporting Open Science policies and mandates, and contributing ‘their bit’ to the transparency of the evidence trail in the scholarly ecosystem.
Reporting and evaluating the funding programmes, essential for the public funders who need to demonstrate the value for money in allocating their funds and other support.
Supporting a more holistic assessment of scholarship and scholars, especially as and when metadata becomes included with a full array of outputs, not limited to books and articles.

How the Crossref Grant Linking System supports Open Science policy

Since 2020, all the grant records are openly available through our REST API which is queried more than 1.8 billion times every month so these metadata records are distributed to thousands of systems across the research enteprise. In a 2022 blog, Ed Pentz and Ginny Hendricks laid out guidelines for research funders to meet open science guidelines using existing open infrastructure such as Crossref, ORCID, and ROR. Syman Stevens, a grantmaking and private philanthropy consultant, highlighted on the call that the funders he works with are increasingly interested in ways to deliver on their open science policy and that participation in the GLS is a tangible thing they can do to meet this goal.

As part of its open science policy, NWO will start participating in the Crossref Grant Linking System from July 2025. Research funders are a part of the scholarly communications system; we not only provide the funding to do the actual research but can also be the authoritative source of data about the projects we have funded and the outputs arising from that funding. Increasingly, all these elements – grants, researchers, outputs - are linked with metadata and unique identifiers to ensure that research is findable and accessible.

– Hans de Jonge, Director of Open Science NL, part of the Dutch Research Council (NWO)

How funders leverage the Grant Linking System in their reporting and assessment

Looking back to the origins of the system, it’s important to recognise the work of the initial working groups. Through their contribution, funders helped design the initial metadata schema for grants as well as establish the governance and fees for this service, and our Advisory Group continues to inform further developments. In this way, the Grant Linking System enables the needs and wishes of funders to contribute and see their data as part of the wider ecosystem.

An excellent example of that synergy in action is the use case presented by Cátia Laranjeira, manager of the PTCRIS programme at the Foundation for Science and Technology, Portugal (FCT). PTCRIS is the Foundation’s integrated national information ecosystem that supports scientific activity management. Cátia reflected on the relative fragmentation of spaces where the scientific outputs are found, and PTCRIS’s ambition for aggregating metadata in one place to be able to trace and evaluate programmes in light of the related outputs. At the start of the programme, they identified lack of a persistent identifier for grants as a major shortcoming of the system. Crossref GLS naturally fits in with their goals.

The initiative by FCT to assign unique DOIs to national public funding through Crossref is a game-changer for open science, linking funding directly to scientific outcomes and boosting transparency. Join us in this effort—let’s make every grant count and ensure open access to research information!"

– Cátia Laranjeira, PTCRIS Program Manager at Fundacao para a Ciencia e a Tecnologis (FCT Portugal)

FCT initially piloted a small subset of their grants (approximately 6,000 recent awards) at the end of 2023. Cátia pointed to researchers’ keen participation in this programme as one of its successes – and thanks to the word of mouth, FCT has already been approached by researchers requesting unique Crossref links for their grants! This appetite for grant IDs will soon be more fully satisfied, as FCT is readying to register all of their grants with Crossref, to enable further insights into funding and outcome flows, supporting them in demonstrating the value for money for the public resources they manage. Via interfaces for grant management and standardised online CVs, the system is also enabling researchers to use the system in their own future reporting and career development.

In the ensuing discussion, Rachel Bruce of UKRI mentioned that she’s hopeful that GLS will help funders ‘close the loop’ on more holistic reward and recognition, allowing for inclusion of evidence for a broader set of outputs in those processes.

How the community is working to integrate open infrastructure

Melissa Harrison, Team Leader at EMBL-EBI, manages Europe PMC and a complementary data science team, who were part of the initial FREYA project – supporting infrastructure delivery for unique identifiers for grants. The team has been adding grant records to Crossref on Wellcome’s behalf since 2019. Melissa highlighted the shortcomings of internal award numbers, which don’t tend to be understood outside of the ecosystem where they are produced (that is the funder’s administrative system), are almost certainly not unique, and don’t resolve to or connect with anything in the wider ecosystem. Therefore internal award numbers can’t signify relationships with other outputs or assets in the wider world. By contrast, Crossref’s Grant IDs are unique, persistent, resolvable, and interrelated with other Crossref metadata, whilst being retrievable for other systems to link to too.

Persistent identifiers for grants was the next logical step after identifiers for funders - open metadata registered with a PID in a central service like Crossref is invaluable to build the full picture of the research enterprise.

– Melissa Harrison, Team Leader, Literature Services at EMBL-EBI)

Ease of execution is important for scaling the Grant Linking System, and enabling its use in a diverse set of circumstances in the open science ecosystem. Altum was the trailblazer, first integrating its grant management platform Proposal Central with GLS. It was good to hear that others are now joining the integration efforts. Syman Stevens talked about the recent work initiated by Joe McArthur at OA Works, to develop a simple, open-source plug-in for any of the major grant management systems, to enable funders to deposit their grant metadata with Crossref GLS with a click of the button. Syman demonstrated the resulting interface in Fluxx, that allows for creating a record and sending grant metadata to Crossref as part of the regular grant management within the platform. He pointed out that, while this integration was developed for Fluxx, all code and documentation is openly available on GitHub and this can potentially be forked or adapted as necessary for reuse in other grant management systems.

It is heartening that others in the community are seeing such a need for this that they’re funding and creating their own tools to advance participation and use of the GLS.

Finally, Fred Atherden, Head of Production Operations at eLife, presented how they include Crossref grant identifiers in publication metadata for the version of record of the works published on their platform. eLife is the first publisher to fully integrate Crossref grant identifiers both within the article display and in the metadata. Fred shared that in addition to collecting the data from the authors, eLife also attempts matching, albeit using very restrictive methodology, to enable more grant metadata in their publication records. They recognise that so far there are very few publishers including persistent links for grants in this way, and talked about plans to start collecting and including this data further upstream, and including them in the future for reviewed preprints.

Acknowledgements and how to participate in the GLS

Reflecting on the last five years, thanks must go to the >35 funders who are already participating (see logo mashup below), to our current volunteers and to those partners working to promote and make use of the Grant Linking System. We also acknowledge that the GLS would not have been possible without the Crossref board members at the time, our staff including alumni Josh Brown, Jennifer Kemp, Rachael Lammey, and Geoffrey Bilder, or without the early dedicated time and input from the following people and organisations on our working groups for governance and fees, and for metadata modelling:

Yasushi Ogasaka and Ritsuko Nakajima, Japan Science & Technology Agency
Neil Thakur and Brian Haugen, US National Institutes of Health
Jo McEntyre and Michael Parkin, Europe PMC
Robert Kiley and Nina Frentop, Wellcome
Alexis-Michel Mugabushaka and Diego Chialva, European Research Council
Lance Vowell and Carly Robinson, OSTI/US Dept of Energy
Ashley Moore and Kevin Dolby, UKRI (Research Councils UK / Medical Research Council)
Salvo da Rosa, Children’s Tumor Foundation
Trisha Cruse, DataCite

To learn more about the Crossref Grant Linking System, the best place to start is our service page. And for the next step, please reach out to us for a conversation about any questions specific to your organisation and any questions that may need to be addressed in order to enable your full participation.

Grant DOIs enhance the discovery and accessibility of funded project information and are one of the important links in a connected research ecosystem. I’m grateful and proud to contribute to the robustness and interconnectedness of the research infrastructure. Few funders are currently participating in the Crossref Grant Linking System, and I encourage others to consider doing so. This adoption follows the “network effect,” where the value and utility increase as more people participate, encouraging even wider adoption.

– Kristin Eldon Whylly, Senior Grants Manager and Change Management Lead at Templeton World Charity Fund (TWCF)

You can email me via feedback@crossref.org or set up a call with me when it suits you (you can overlay your own calendar using the toggle at the top right). We look forward to welcoming even more funders and to see those relationships in the open science infrastructure grow even further in the coming years.

Rebalancing our REST API traffic

Stewart Houten — Tue, 04 Jun 2024 00:00:00 +0000

Since we first launched our REST API around 2013 as a Labs project, it has evolved well beyond a prototype into arguably Crossref’s most visible and valuable service. It is the result of 20,000 organisations around the world that have worked for many years to curate and share metadata about their various resources, from research grants to research articles and other component inputs and outputs of research.

The REST API is relied on by a large part of the research information community and beyond, seeing around 1.8 billion requests each month. Just five years ago, that average monthly number was 600 million. Our members are the heaviest users, using it for all kinds of information about their own records or picking up connections like citations and other relationships. Databases, discovery tools, libraries, and governments all use the API. Research groups use it for all sorts of things such as analysing trends in science or recording retractions and corrections.

So the chances are high that almost any tool you rely on in scientific research has somewhere incorporated metadata through us.

Optimising performance

For some time, we’ve been noticing reduced performance in a number of ways, and periodically we have a flurry of manually blocking/unblocking IP addresses from requesters that are hammering and degrading the service for everyone else, and this is of course only minimally effective and very short term. You can always watch our status page for alerts. This is the current one about REST API performance: https://status-crossref-org.pluma.sjfc.edu/incidents/d7k4ml9vvswv.

As the number of users and requests has grown, our strategies for serving those requests must evolve. This post discusses how we’re approaching balancing the growth in usage for the immediate term and provides some thoughts about things we could try in the future on which we’ll gladly take feedback and advice.

Load balancing

In 2018, we started routing users through three different pools (public, polite, and plus). This coincided with the launch of Metadata Plus, a paid-for service with monthly data dumps and very high rate limits. Note that all metadata is exactly the same and real-time across all pools. We also, more recently, introduced an internal pool. Here’s more about them:

Plus: This is the aforementioned premium option; it’s really for ‘enterprise-wide’ use in production services and is not really relevant here.
Public: This is the default and is the one that is struggling at the moment. You don’t have to identify yourself and, in theory, we don’t have to work through the night to support it if it’s struggling (although we often do). Public currently receives around 30,000 requests per minute.
Polite: Traffic is routed to polite simply by detecting a mailto in the header. Any system or person including an email is being routed to a currently-quieter pool, this means we can always get in touch for troubleshooting (and only troubleshooting). Polite currently receives around 5,000 requests per minute.
Internal: In 2021, we introduced a new pool just for our own tools where we can control and predict the traffic. Internal currently receives around 1,000 requests per minute.

The volumes of traffic across public, polite and internal pools are very different and yet each pool has always had similar resources. The purpose of each of these pools has been long-established but our efforts to ask the community to use polite by default have not been particularly successful and it is clear that we don’t have the right balance.

The internal pool has been dedicated to our internal services that have predictable usage and that have requests that are not initiated by external users. The internal pool has previously included reference matching but not Crossmark, Event Data, or search.crossref.org, which all use the polite pool instead, along with the community. We have the capacity on the internal pool to shift all of this “internal” traffic across, and in doing so we will create more capacity for genuine polite users and redefine what we consider to be “internal”.

Creating more capacity on polite will also give us the opportunity to load-balance requests to both polite and public across the two pools. We are at a point where we cannot eke more performance out of the API without architectural changes. In order to buy ourselves time to address this properly, we will modify the routing of polite and public and evenly distribute requests to the two pools 50/50.

The public and polite pools have equal resources at the moment yet handle very different volumes of traffic (30,000 req/min vs 5,000 req/min), and with the proposed changes to internal traffic the polite pool would handle a fraction of this. The result would look something like 31,000 req/min evenly distributed across public and polite.

Rate limiting

Our rate-limiting also needs review. We track a number of metrics in our web proxy but only deny requests on one of them - the number of requests per second. On public and polite we limit each IP address to sending 50 req/sec and if this rate is exceeded users are denied access for 10 seconds. These limits are generous and we cannot realistically support this volume of request for all users of the public or polite API.

However, when requests are taking a long time to return, we potentially have a separate problem of high concurrency as hundreds of requests could be sent before the first one has returned. We intend to identify and impose an appropriate rate limit on concurrent requests from each IP to prevent a small number of users from disproportionately affecting all users with long-running queries.

Longer-term

So, in the short-term we will revise our pool traffic as described above. We’ll do that this week. Then we will review the current rate limits and reduce them to something more reasonable for the majority of users. And we’ll identify and introduce a rate limit for concurrent requests from each user.

Longer-term, we need to rearchitect our Elasticsearch pools so that we can:

Reduce shard sizes to improve performance of queries
Balance data shards and replicas more evenly
Optimise our instance types for our workload

Want to help?

Thanks for asking!

Firstly, please, everyone, do always put an email in your API request headers - while the short term plan will help stabilise performance, this habit will always help us troubleshoot e.g. we can always contact you instead of blocking you!

Secondly, we know many of you incorporate Crossref metadata, add lots of value to it in order to deliver important services, and also develop APIs of your own. We’d love any comments or recommendations from those of you handling similar situations on scaling and optimising API performance. You can comment on this post which is managed via our Discourse forum. We’ll also be adding updates to this thread as well as on status.crossref.org. If you’d like to be in touch with any of us directly, all our emails are firstinitiallastname@crossref.org.

Now is the time to work together toward open infrastructures for scholarly metadata

Ginny Hendricks — Wed, 27 Oct 2021 00:00:00 +0000

Now is the time to work together toward open infrastructures for scholarly metadata

The closure of Microsoft Academic — and its records not covered by Crossref — was a wake-up call for the scholarly community. Published on the LSE Impact Blog in October 2021, this piece draws lessons from that moment and makes the case for open, collectively governed, and sustainable metadata infrastructure.

Strategists

Understand why open scholarly infrastructure is fragile — and what makes it resilient. How the loss of Microsoft Academic revealed gaps in coverage, governance, and sustainability, and why POSI offers a framework for building something more durable.

Decision-makers

Know what your organisation should commit to. A direct call to action for publishers, funders, institutions, and infrastructure providers — and why mandating open, FAIR metadata is essential to the health of the scholarly record.

Practitioners

See the gaps and how to fill them. What the Microsoft Academic coverage data revealed about grey literature and Global South scholarship, and how open metadata workflows and full-text access can address those gaps.

What this piece covers

The Microsoft Academic closure — what was lost, and what the coverage gap revealed about equity and representation in open scholarly infrastructure
Four lessons learned — why metadata infrastructure must be POSI-compliant, collaborative, supported by systematic policy, and paired with open full-text access
A call to action for each stakeholder group:
- Publishers — deposit complete metadata (references, abstracts) in open infrastructures; support the Initiatives for Open Citations and Open Abstracts
- Researchers — choose journals that offer open access to both full text and complete, validated metadata
- Funding agencies — mandate that metadata from funded research be made openly available
- Institutions and libraries — require complete, open metadata availability in contracts with publishers
- Infrastructure and service providers — simplify metadata deposition for smaller publishers with limited technical capacity
- Metadata disseminators — ensure enriched metadata is openly available with full provenance using common standards and open licences

Read the article

Read on LSE Impact Blog

Open-source code: giving back

Joel Schuweiler — Fri, 30 Apr 2021 00:00:00 +0000

TL:DR;

Hi, I’m Joel
GitLab UI unsatisfactory
Wrote a UI to use the API
Wrote a missing API
Open company contributes changes back to another open company
Now have a method for getting work done much easier
Hurrah!

I’m Joel, a Senior Site Reliability Engineer here at Crossref. I have a long background in open source, software development, and solving unique problems. One of my earliest computer influences was my father. He wrote software to support scientists in search of things like the top quark, the most massive of all observed elementary particles.

One day my father came home with over 40 floppy disks, excited to have this cool, free operating system called Linux. Together we installed Linux and ended up with a fully functional computer. Learning and using Linux opened up an entirely new world to me of amazing open-source software that I could use freely. As I enjoyed all this new software now available to me, I tried to fix any bugs or problems I’d encounter and report solutions for them to the software developers. It felt great to be able to contribute back so others could benefit.

Software teams tend to manage their workflow by writing issues, reviewing them to make sure they make sense and have an achievable goal, estimate how much time it will take to complete, and finally––the crucial step––putting the issues in the order in which they should be completed. To manage my work, I’ve always used Jira––a product designed to help teams of all types prioritize work––and for the first time in over a decade, I find myself not using it in my work.

Product development tracking with GitLab

The Crossref team took the decision a few years ago to move all our development and product tracking work via GitLab––a commercial open-source product anyone can use to help keep track of software throughout the development life cycle––with an open-by-default policy. Work is tracked using the issues feature of Gitlab. GitLab will host it, so you don’t have worry about maintenance and backups. One major drawback I discovered with GitLab, is a lack of maturity when it comes to doing light project management work.

This is where the trouble begins with GitLab.

In the board view of your issues, you can transition your issues from waiting, to in progress, from in progress to done. The problem with this view is its width-restricted, and things like tags on issues, which are used to help categorize, take up valuable vertical space. With enough tags and a long enough subject line, you can only see five issues at a time on a MacBook Pro monitor, for example.

In the list view of your issues, you get a clean compact view; the perfect view to order issues. However there’s one major flaw, it’s paginated. (You know when you’re shopping and they make you click to see another page of goods? Yes, like that.) The problem with GitLab’s implementation is you can drag and drop issues on a given page, but there is no way to move the issues to another page in the list of results. Additionally, all newly-created issues are added to the end of the list.

The solution

I went about finding a solution by visiting GitLab’s own public issue page and found that requests requiring user interface (UI) changes would languish; in some cases, they would go years without getting approval. Instead of putting in all the work to open an issue with them, only to have it be discarded or ignored, I decided to look for another way.

GitLab has an API, what more could I need? I discovered I could log in and get a list of all the issues, by project, and by group. “This is perfect!”, I thought. I can write my own UI around it. It took three evenings writing a UI that was satisfactory to me. When I started writing javascript to interact with the UI, I learned that the ’re-ordering of issues’ didn’t actually have an API. Further investigation lead me to the issue tracker where I found an issue by a GitLab employee asking for the same functionality––the ability to re-order issues.

While in a chatroom for GitLab development, I was genuinely surprised by my experience. There was quick attentive help on locating the file I would need to implement the change, they set up a development environment, and even helped submit tests for my code while I worked on updating documentation and writing a changelog entry. It felt like GitLab must’ve designated an employee to work with the community on submitting improvements. In no time, the API for re-ordering was implemented. After the scheduled monthly release of GitLab rolled out with my new API, I was able to easily re-order issues.

GitLab’s response when help was needed along the way was impressive. Now there is a much easier method for getting work done that everyone can use. It’s rewarding when you can contribute back to the community for all to benefit.

Is GitLab as polished as Jira? No. Did they embrace me making changes by being open from the start and providing help along the way? Yes. Do I see Jira shifting its culture to match? Unlikely.

By emulating GitLab, an open organisation like Crossref has a shot at encouraging community development.

Crossref: the sustainable source of community-owned scholarly metadata

Dominika Tkaczyk — Wed, 01 Jan 2020 00:00:00 +0000

Crossref: The sustainable source of community-owned scholarly metadata

Published in Quantitative Science Studies (2020), this paper describes the scholarly metadata collected and made openly available by Crossref — its history, scale, content types, and role in the research ecosystem.

Strategists

Understand why Crossref metadata is foundational scholarly infrastructure. The history, governance, and community model that make Crossref a sustainable, neutral, member-owned source of scholarly data.

Decision-makers

Know what’s in the metadata and what it enables. 106 million records, 13 content types — not just bibliographic basics but funding, licences, citation links, corrections, and retractions.

Practitioners

Learn how to access Crossref metadata at scale. A technical overview of the REST API and OAI-PMH, with context on citation data provision and metadata quality trends.

What this paper covers

History and governance — how Crossref was founded in 2000 and why its member-owned, not-for-profit structure matters for long-term sustainability
Scale of the metadata set (in 2021) — over 106 million records across journals, conference papers, books, datasets, preprints, peer reviews, grants, and more
Richness beyond basic bibliographic data — abstracts, full-text links, funding and licence information, citation links, corrections, updates, and retractions
Role in the research ecosystem — how Crossref metadata directly supports scientometrics, research assessment, and scholarly communications research
APIs and access — the REST API and OAI-PMH as the primary routes to Crossref metadata at scale
Citation data — the evolution of Crossref’s citation data provision, including the move to open references
Metadata quality and curation — trends in completeness and accuracy over time, and plans for improvement

Read the full paper

Loading…

Accidental release of internal passwords, & API tokens for the Crossref system

Geoffrey Bilder — Fri, 04 Oct 2019 00:00:00 +0000

TL;DR

On Wednesday, October 2nd, 2019 we discovered that we had accidentally pushed the main Crossref system as part of a docker image into a developer’s account on Docker Hub. The binaries and configuration files that made up the docker image included embedded passwords and API tokens that could have been used to compromise our systems and infrastructure. When we discovered this, we immediately secured the repo, changed all the passwords and secrets, and redeployed the system code. We have since been scanning all of our logs and systems to see if there has been any unusual activity that could be related to the exposure of the container.

Please note that no external data e.g. member passwords or personal information were exposed; our source code contains only internal passwords and ‘secrets’ such as API tokens.

Thankfully, the way in which these secrets were exposed (in compressed, binary files which were, in turn, in a Docker image) means that they were probably overlooked by the automated exploitation tools which focus on scanning source code. And, so far, we have seen nothing that would indicate that these passwords and secrets have been exploited. We will, of course, inform our members directly (and update this blog) if that changes.

More than you probably want to know

If you are continuing to read this, my guess is that you might have questions like:

Why are you doing something as silly as embedding secrets and passwords in your code?
And wait a minute… I thought Crossref code was open source?
And why is the director of strategic initiatives announcing this?

Let me answer these questions in random order.

In March 2019 I took over Crossref’s technical teams when Chuck Koscher announced that he would be retiring at the end of the year. I’m now the director of technology & research.

A few months earlier we had already concluded that a major portion of the Crossref system had accumulated 20 years of technical debt and that we were going to spend a significant portion of 2019 and 2020 paying down that debt.

Specifically, a lot of the code that runs Crossref was inherited from a third party who developed it back in the early 2000s. This means that, even though any new systems that we’ve developed since 2007 have been open-source, the code for the oldest parts of the system has remained closed because it contained potentially proprietary code as well as a lot of deprecated coding practices. Also - the architecture, the tooling, and the development processes behind the Crossref system had not changed much in those twenty years. It was fantastic architecture, tooling, and code for its time. But architectures that scale to millions of records need to change to handle hundreds of millions of records. Processes that work for configuring one service need to change when you are managing dozens of services. And support tools that work for a few hundred members break down when you are dealing with tens of thousands of members.

These parts of the Crossref system were decidedly not 12 factor. We were not using DevOps or SRE working practices to run them. And the bulk of that part of the system is still being run in a traditional data center.

But since March we have been slowly fixing that. In incremental steps. Some of which are visible as a side effect of the security incident that precipitated this blog post. For example, one of our first moves was to move our development to Gitlab. Even though a big chunk of the base Crossref code is still closed source, we saw moving to Gitlab as a priority because Gitlab offers a fantastic suite of tools to help automate and manage our deployments. Similarly, we have been Dockerizing the Crossref system so that it is easier to scale and run in different environments. And as part of this effort, we have spent a lot of time on the issue of how to best handle secrets. We knew our secrets management in this part of the codebase was horrible. We have been developing some experiments and infrastructure for handling these secrets securely. But we haven’t finished this work yet. And so the system slipped out into a public repo too early. Ironically, this too illustrates a fundamental change in the way we develop things. Our default is to be open and transparent. This case is currently an exception. An exception we want to eliminate, but one we are not ready to do yet. We have to audit and scrub the code first.

Yes, this incident has been embarrassing. But not nearly as embarrassing as the fact that Crossref has succumbed to a technology industry cliche. That we spent so much time growing and focusing on new features for our members, that we neglected some of the creaking infrastructure of our infrastructure.

And I should be clear about two things:

First, not all of our code is like this. We have, for a long time, been building open source software and using modern best practices for secrets management in our newer subsystems and services. The problems described above are confined to twenty-year-old-code that we didn’t write in the first place and that we had been avoiding refactoring.

And second, the technology team has been marvelous at responding to the challenge we face. They have adopted new processes and tools. They are learning new techniques. We are steadily chipping away at these problems.

It is generally considered bad practice to praise or reward technology teams for fire-fighting instead of fire prevention, but this may be the exception that proves the rule.

I was blown away by how the technology, product, and support teams worked together. When we discovered this problem, I sat at my desk in rural France and watched as staff from the UK, and all three US time zones shut down this problem in just a couple of hours. Obviously, I wish we hadn’t had the problem in the first place, but seeing their response did a great deal to encourage me that we are on the right track.

In any case, it looks like we’ve been lucky. And we’ll be working even harder to refactor our code, tools, and processes so that this kind of thing doesn’t happen again.