Citation on Crossref

Two billion citation links in Crossref help research travel further

Kornelia Korzec — Tue, 26 May 2026 00:00:00 +0000

We’ve recently reached an important milestone for the research nexus: the works in our metadata corpus are now connected with over 2 billion citation links! This is a great opportunity to share a dedicated dataset and discuss why these are important for science.

The reference metadata is a lifeline of discoverability. Scholars use citations to critique and build on existing research. They acknowledge the contributions of others through references. Our members can then deposit those references as part of metadata with Crossref, and we use those to link the cited and citing objects. This results in complex thematic networks that can be explored by interested researchers. Many tools for research discovery use the linked reference metadata in Crossref to support searches of related content.

The citation links are derived from bibliographic references in the metadata of one work that include DOIs of materials it cites (scholarly works, data, code, etc.). It’s always best if the members can deposit these relationships in full. In a recent post, we shared that nearly half of these links are asserted by our members through metadata deposits, and the other half are created thanks to our automated matching. This form of metadata enrichment happens when members include some information about the references but without the DOI of the cited work, and it’s enough to automatically find and add that DOI. The enrichment supports making data more useful for the community.

The most important impact of citation links is the increased discoverability of connected works. Reference metadata is an important tool for improving visibility and readership of our members’ content. These links are also the foundation of our Cited-by service, which enables implementing members to display citation counts of the work they published on their landing pages.

The chart below shows the cumulative count of citations over time, by the created date of the citing DOI’s record. These include records linked by DOI either through member-submitted metadata or matched by Crossref, as well as records that are unmatched. Unmatched records can include records that we were unable to match with the information we have, but also records that truly have no DOI to link to. You can explore the full citation dataset of all 2 billion citation links between Crossref DOIs available now as a (somewhat hefty) download.

Cumulative count of references deposited to Crossref by created date of citing DOI

The push for open citation data is something that has unfolded over the last few decades, making more and more of these relationships public. Notably, the growth in citation links reflects not just the output of new scholarship, but also a sustained effort to extend coverage of the historical scholarly record. We can see evidence of this playing out over time by looking at our historical data—periodic snapshots of Crossref’s metadata going back to 2019. When comparing successive snapshots and examining the publication dates of citing and cited works, we can classify each newly appearing citation as either a new paper citation, or a retrospective one. A new citation is where the citing work was published since the previous snapshot, representing real growth in the scholarly record. A retrospective citation is where both papers already existed but the link between them had not yet been captured by Crossref, and these represent indexing catchup rather than new publishing activity.

The chart below shows the cumulative count of citations added in each category since 2019. In the early years of our data, retrospective backfill was the dominant source: the blue line climbs steeply from 2019 to 2021 as a large volume of previously uncaptured historical citation relationships entered the corpus. Over time, however, that rate of backfilling has levelled off. New paper citations, meanwhile, have grown steadily throughout the period, and by 2025 they surpassed the cumulative retrospective total. The open citation ecosystem continues recovering historical links, but the citation network’s growth is now increasingly driven by the natural momentum of scholarly publishing itself.

Cumulative citations added to Crossref by type, 2019–2026. Retrospective citations (blue) represent links to and from works that existed before the previous snapshot; new paper citations (green) come from works published since the last snapshot.

Combined with other metadata for more context, reference metadata supports bibliographic and meta-research on different aspects of the scholarly process, and can support judgements about research integrity and conflicts of interest.

Stereotypically, when talking about references, we consider links to published works (whether preprints, journal articles, or books). However all types of records in Crossref can be cited. Thanks to the changes in our latest schema, members can now signal the types of content that is being referenced. And with our new Data citations endpoint, the community can explore specifically links from Crossref-registered records to research data, including citation links to works within Crossref, as well as DataCite’s corpus.

Close to half of all records registered with Crossref still have none or not enough reference information to make such connections. We invite members to regular Metadata health-check webinars to support them in improving completeness of their records for increased transparency and visibility.

Strengthening support for data citations and saying goodbye to Event Data

Martyn Rittman — Tue, 24 Mar 2026 00:00:00 +0000

We’re excited to announce a new data citation API endpoint and are seeking your feedback. The new service makes existing data citation relationships in our metadata available, thereby surfacing this part of the research nexus. At the same time, we’ve decided that it’s time to move on from Event Data.

Time to say goodbye

Metadata about published scholarly research has evolved, and continues to evolve in exciting ways. A published article, book, or conference paper is only a single piece of the puzzle. A host of digital identifiers and items can be put together to form a more complete picture of a research project. This is what forms the basis of the research nexus—a rich and reusable open network of relationships connecting research organisations, people, things, and actions; a scholarly record that the global community can build on forever, for the benefit of society.

Ten years ago, we launched Event Data to surface mentions of research around the Internet. What were people saying about published research? Could this discourse contribute to post-publication review and validation? We set up Event Data to capture use of Crossref DOIs in the online world from a variety of sources, including blog posts, social media, websites, and annotations. The idea was that diverse mentions (or “events”) could supplement traditional citation counts as a way to capture the value of research.

Today, the focus is increasingly on transparency, research integrity, and the completeness of outputs. Trust in research is shaped by knowing who the funder was, being able to reanalyse the original data, or checking for bugs in the analysis code. There are also more identifiers for objects within the research space and they are used more widely. This shift is evident in the relatively low usage of Event Data. We can no longer justify the resources and cost that goes into maintaining it as a service. Instead, we will focus on enabling and surfacing relationships between different types of research outputs, starting with links to datasets.

For these reasons, we have decided to sunset the Event Data API and from 23 April 2026 it will no longer be available (although historical data will still be available). In its place, we’re making available an API endpoint for data citations.

Visibility for data citations

The new API endpoint focuses solely on data citations and uses references and relationships deposited by Crossref members, including Crossref articles referencing datasets with either Crossref or DataCite DOIs. While our metadata contains many data citations, and some of them are labelled as data citations, it is often difficult to find them because they are swamped in number by other citations. If you are trying to get data citations directly from our REST API, it’s like looking for a needle in a haystack. This new endpoint makes connections easier to find, enabling organisations to track when research is reused, cited, and built upon. By putting this metadata into a dedicated service, we are making it easier for interested organisations to track and find data citations. Our goal is to make existing sets of connections easier to access, giving clarity to how scholarly works link to the data that supports them.

This beta version will allow us the opportunity to incorporate feedback from the community and make changes to improve delivery. We received early positive feedback from a number of interested organisations. Later this year we will assess whether it is ready for production, needs more work, or if insufficient interest from the community suggests we should pursue a different solution.

Anyone interested in data citations is invited to try the new endpoint. Please let us know your feedback via the community forum. You can find supporting documentation on our website and Swagger documentation, including the opportunity to try out features.

Edited 15 June 2026: link to historical Event Data added.

On metadata enrichment

Dominika Tkaczyk — Thu, 19 Mar 2026 00:00:00 +0000

Metadata is communication; it can tell a story about research and paint a picture for others to respond to and learn from, across the world and throughout the forthcoming generations. Metadata can feel technical with words like ‘infrastructure’ and ‘schema’, and sometimes, like tech in general, it comes with hyperbole. But metadata really is part art (storytelling and pictures) and part science (structured models and standards) with both aspects being equally important, and requiring people as well as systems. That necessary combination of human and machine involvement also makes metadata challenging.

Crossref, as the earliest adopter of DOIs specialising in scholarly research, became synonymous with DOIs in this community. However, not everyone realises that DOIs can be registered with any one of nine different agencies, which are all separate organisations with entirely separate systems that do not at present integrate or connect. And what’s more – there isn’t a central or shared “DOI schema” – each agency develops the metadata for the purposes of their organisation or community. In Crossref’s case, with our vision to create the research nexus as a complete and robust network of relationships between objects, people, and institutions of scholarship – that community encompasses the whole of the research enterprise.

The immense 180 million records of research outputs in Crossref are maintained in a system that 24,000 member organisations have already invested in. Those records benefit from rich and format-appropriate metadata schema, developed in close collaboration with the community, which makes it possible for our members to offer contextual information about each object they register. We have a long history of working with our members on recording that context, creating tools, and providing support to adopt standard metadata, enriching the context for the benefit of the scholarly community, and society at large.

Of course, those metadata records are not perfect, both in terms of quality and completeness, and the frustration around gaps in metadata is particularly strong. We are working to improve the quality and completeness of the metadata from many angles: by working with the community to understand their needs and obstacles, by identifying and analysing potential sources for additional metadata, by maintaining and adopting the existing system to changing environment, and by planning a new flexible system that will allow third-party assertions and automated enrichment workflows.

In 2020, we published a paper for the inaugural issue of Quantitative Science Studies on Crossref: The Sustainable Source of Community-Owned Scholarly Metadata and blogged an introduction to it under Crossref Metadata for Bibliometrics. One of the things our analyses in 2019 showed was that over 80% of records between 2013-2016 had been updated. Reviewing the numbers recently, we continue to see this stewardship and maintenance of metadata, amounting to almost 70% of records from the past decade being updated at least once. On the dawn of reaching 2 billion citation links, we’d like to share our experience, plans, and views on this ubiquitous activity of updating and connecting metadata – by our members and by automations built into the system by us. Altogether, these constitute the enrichment process to improve the usability of the information for the community.

Metadata available through Crossref

Crossref collects, processes, stores, and shares metadata records for a wide range of research outputs. While each record describes an individual research output, it also mentions other entities and their attributes - and, most importantly, the relationships between them. Two works identified by DOIs, for example, may be linked by a citation relationship. A person identified by an ORCID may be connected to an institution identified by a ROR ID through an affiliation relationship. A preprint and its corresponding journal article, each with its own DOI, can be linked by an “is preprint of” relationship. A research output may be associated with a grant through a “financed by” relationship. Together, these entities and relationships form the foundational building blocks of the research nexus.

As of March 14, 2026, the Crossref database contains 180,034,490 metadata records describing research outputs. You can download all the records and examine them yourself in the latest public data file. The plot below illustrates how the number of works has changed over time, showing that the rate of growth is accelerating.

The metadata records describe research outputs of various types, including:

journal articles
books and book chapters
conference proceedings
peer reviews
reports
datasets
preprints
dissertations
grants
and more

The majority of works in the Crossref database (67%) are journal articles. However, the distribution of record types has changed considerably over time. Newer types, such as components, datasets, and posted content, are growing more quickly than more traditional ways of communicating research:

Research outputs in the Crossref database are represented by rich metadata records, which may include:

basic bibliographic metadata (title, publication dates, contributors, journal title, conference name, volume and issue numbers)
authors’ affiliations and ORCID identifiers
abstracts and links to full text
funding metadata, including funders and grant details
license metadata
bibliographic reference lists
clinical trial numbers
updates such as corrections or retractions
relationships between works and other entities, such as “is translation of”, “is review of”, “is preprint of”, or “is version of”
components associated with the work, such as figures, tables, and supplemental materials

All metadata is freely available through the Crossref REST API, and additional services, such as Crossref Search, are also provided.

A natural question is: where does all this metadata come from? This is important for two main reasons. First, it helps address the question of trust, as understanding the origin of the metadata allows users to better assess its reliability. Second, it points us to the right place when investigating or addressing issues or gaps in the data.

At first glance, the answer might seem straightforward: from Crossref members. Crossref members, such as publishers, research institutions, universities, funders, museums, libraries, data and subject repositories, and conference providers, register metadata for the outputs they publish. Crossref stores this metadata and makes it available to the community.

In reality, however, the story is more complicated.

Metadata enrichment layers

The initial metadata deposit is only the beginning of what can become a long and rather fascinating journey. What users can see in our REST API is often the result of a series of updates and additions that occur over time, sometimes coming from multiple sources and happening in different ways. We can think of these ways as enrichment layers.

Each enrichment layer offers opportunities to improve the metadata while also introducing its own considerations and challenges. Rather than forming a sequence of clearly separated stages, these layers intertwine, overlap, and affect one another, collectively shaping how a research output is represented within the research nexus.

Enrichment layers are essential for completeness of the research nexus. If we relied solely on the original, one-off deposits from members, the metadata would be full of gaps, limiting the usefulness of any analysis or assessment based on it. While the scholarly metadata will never be perfectly complete, applying these enrichment layers is how we gradually and collectively build a fuller, more accurate picture of the research nexus.

One important caveat is that more metadata doesn’t magically equal better metadata. In fact, there’s often a delicate tradeoff between completeness and quality: the harder one pushes to fill every gap, the greater the chance of introducing errors. At Crossref, we believe quality comes first. We recognise that no dataset will ever be perfect, but we’re equally unwilling to apply enrichment processes without quality control. Any enrichment we introduce must meet a high bar for accuracy — no exceptions, no shortcuts.

The order of the enrichment layers discussed here loosely reflects how established they are within the scholarly ecosystem. There also might be a correlation, or at least a perceived one, between this ordering and the reliability of the underlying processes. That said, one must tread carefully when making such interpretations: perceived reliability is not the same as actual reliability.

Layer 1: Member updates

Crossref members not only deposit metadata, but also update it over time. This is an essential part of the system for several reasons. There may be errors in the originally deposited metadata that need to be corrected. Also, the initial record may contain gaps that can be filled later as more information becomes available. In addition, many changes naturally occur: landing page URLs may change, works may be archived in new locations, or identifiers for affiliated organisations may become available. Those situations also ideally result in an update.

This update process is well established. Over 24,000 Crossref members form a large global community that operates under shared membership terms. As part of these terms, members are responsible for maintaining and updating their metadata records. In this governance framework it is clearly defined who owns and stewards the metadata associated with each record, and who is responsible for the quality level and issues.

Member updates are very common. As an example, over 80% of works deposited between 2013 and 2020 were updated at least once. This demonstrates the community’s commitment to improving completeness and quality of the scholarly record. The plot below shows the percentage of works created in a given month that were updated at least once.

However, this layer also comes with challenges. It relies on members actively meeting their obligations to maintain and improve their metadata. As a result, gaps and inconsistencies can remain, and overall metadata quality is never perfect.

Our plans for the future in this area largely build on what is already happening. This includes developing and maintaining effective user interfaces for updating metadata, evolving the input metadata schema to keep pace with changes in the scholarly landscape, offering regular workshops on metadata improvements, and collaboratively establishing best practices while educating members on how to apply them.

Layer 2: Community feedback loop

Crossref metadata is widely used and examined by a large community of consumers. As a result, issues with metadata are sometimes identified by community members and reported back to us. When this happens, Crossref does not directly correct the metadata records. Instead, we contact the relevant member responsible for the record and able to deposit an update.

In this layer, the stewardship of metadata remains with the member, while responsibility for metadata quality broadens to include other actors in the community. This creates significant potential for scaling by involving a large community in identifying and reporting metadata issues.

At present, however, this process is not automated. Crossref staff effectively act as intermediaries between those reporting issues and the responsible member. As a result, the process has limited scalability. It also depends on the willingness of members to act on the reports they receive, as they are not obligated to respond to such reports.

In the future, we may explore automating portions of this workflow to handle community feedback more efficiently and lighten the load on everyone involved.

Layer 3: Metadata matching

Metadata matching is the task of finding an identifier for an item based on a structured or unstructured description of it. Matching strategies run as fully automated processes that analyse information deposited and updated by members and add identifiers, filling gaps in the metadata.

There are many instances of metadata matching problems, for example:

bibliographic reference matching: finding a DOI for a cited paper based on a bibliographic reference,
funder matching: finding the ROR ID for a funder based on its name,
affiliation matching: finding the ROR ID for an organisation based on an affiliation string,
preprint matching: finding the DOI for a preprint that precedes a given journal article,
grant matching: finding the grant DOI based on an award number and a funder name.

This layer is unique, as it focuses on a crucial type of gap in the scholarly record: the missing relationships between entities. Indeed, adding an identifier for an entity mentioned within a metadata record of a research output is typically an equivalent of asserting a relationship between that output and the matched entity. For example, bibliographic reference matching inserts citation relationships, and funder name matching - funding relationships between a research output and a funding organisation. These relationships form the foundation of the research nexus.

Currently, at Crossref, we perform two types of matching. We match bibliographic references to the DOIs of cited outputs, and funder names to Funder IDs. Both processes rely on fuzzy comparisons and other heuristic approaches to identify likely matches.

In the case of bibliographic reference matching, as it turns out, more than half of the cited DOIs (1 billion) available in the Crossref database originate from automated metadata matching:

In the case of funder name matching, the distribution is very different, but the matching strategy was still able to fill in some of the gap:

Metadata matching is a particularly valuable form of enrichment for several reasons. Matching strategies can often achieve high levels of accuracy while working in a fully automated way. This makes them highly scalable and drastically reduces the need for human oversight. Their focus on relationships also strengthens the foundations of the research nexus.

At the same time, this enrichment layer presents a number of challenges.

Its most fundamental limitation to remember is that metadata matching can only fill gaps when there is at least some useful information to work with. For example, it can identify a cited document only using structured or unstructured citation data, and the funding organisation can only be identified if some funding information is available. But if citation information, or funding information, is completely absent, as is the case for 101M (56%) records and 166M (92%) records respectively, then matching simply isn’t possible.

Matching strategies can also be complex and time-consuming to research, develop, and maintain. They require additional considerations of issues such as openness, explainability, complexity, flexibility, and cost.

Perhaps most importantly, in the case of matching, it becomes less clear who is responsible for the information introduced through the matching process. This is particularly important because matching results are never perfect, meaning there is always a risk of introducing errors. The risk is further amplified by the fact that matching strategies typically operate in a fully automated, unsupervised manner. As a result, careful evaluation of matching performance, as well as maintaining accurate provenance records, becomes increasingly important.

At Crossref, we have ambitious plans in this area. We intend to rebuild Crossref’s metadata matching workflows using modern software development and data science practices. The goal is to create a dedicated, consolidated matching service that will eventually replace all existing production matching processes, with results made available through the REST API. This project will cover six matching tasks: bibliographic reference matching, funder name matching, preprint matching, affiliation matching, grant matching, and title matching. You can learn more about metadata matching at Crossref at a dedicated project page.

Layer 4: Third-party datasets

There are many databases containing scholarly data, and one way to fill gaps in Crossref member-provided metadata is to incorporate additional metadata from those external sources.

We already have one example of this. Crossref ingests data from the Retraction Watch database to supplement information about retractions and other updates to records:

This layer has several advantages. It draws on subject-specific and metadata-specific expertise, avoids reinventing work that has already been done elsewhere, and reflects a collaborative community-driven approach to improving the scholarly record.

However, there are also important challenges to consider. Integrating external data often involves multiple data licenses or acquisition arrangements, and there may be less control over data quality compared to metadata that comes directly from members. There is also a risk that relying too heavily on external sources could shift responsibility away from the member stewards of the metadata. Finally, it can be difficult to determine which external datasets provide sufficient value and longevity to justify long-term integration.

Looking ahead, we plan to explore further opportunities to incorporate third-party datasets, carefully considering the value they bring, as well as issues of licensing, sustainability, and data quality.

Layer 5: Unstructured content scraping

A significant amount of scholarly information still exists in fully unstructured forms, such as full-text PDF documents and web pages. In principle, extracting information from these sources could help fill many gaps in existing metadata.

In a lighter-touch approach, analysing full-text documents can also help verify existing metadata elements. If such a check fails, the unverified element may be removed from the record — which, perhaps counterintuitively, can also count as enrichment, since improving accuracy is every bit as important as adding new information.

There are also important challenges to consider. Extracting metadata directly from unstructured sources could substantially shift responsibility away from the original data stewards or owners, weakening the current stewardship model. The results of automated extraction may also be inconsistent or of relatively low quality. In addition, there are potential legal and rights-related concerns, particularly when processing full-text materials. Finally, developing reliable extraction methods would require substantial research and engineering effort.

For all these reasons, the practical usefulness of this approach remains uncertain, and Crossref currently has no plans to run such processes in production. We will, however, keep a close eye on emerging extraction technologies and may consider adopting them in some form if future evaluations show clear value.

Summary

Metadata is far more than a technical afterthought of the publishing process. It is the connective tissue of the scholarly ecosystem, linking research objects, people, and institutions into a coherent, navigable network. At Crossref, this takes the form of a vast and continually evolving corpus of more than 180 million metadata records, all contributing to the emerging research nexus, being built through collective community effort to help the global research community discover, interpret, and reuse knowledge effectively.

The initial metadata record deposited by members is only the beginning. Its quality and completeness can improve over time through multiple enrichment layers: member-driven updates, community feedback, automated metadata matching, and the incorporation of third-party datasets. These processes help fill gaps and strengthen the reliability of the scholarly record, all while upholding a firm commitment to accuracy and stewardship.

Taken together, these layers reflect a long-term, collaborative effort across technology developments, community participation, and responsible automation, to ensure that scholarly metadata becomes richer, more interconnected, and more useful for everyone who relies on it.

Start citing data now. Not later

Geoffrey Bilder — Thu, 23 Mar 2023 00:00:00 +0000

Recording data citations supports data reuse and aids research integrity and reproducibility. Crossref makes it easy for our members to submit data citations to support the scholarly record.

TL;DR

Citations are essential/core metadata that all members should submit for all articles, conference proceedings, preprints, and books. Submitting data citations to Crossref has long been possible. And it’s easy, you just need to:

Include data citations in the references section as you would for any other citation
Include a DOI or other persistent identifier for the data if it is available - just as you would for any other citation
Submit the references to Crossref through the content registration process as you would for any other record

And your data citations will flow through all the normal processes that Crossref applies to citations. And it will be distributed openly to the community (including DataCite!) via Crossref’s services and APIs. All data citations deposited with Crossref will be exposed in the (soon-to-be launched) Data Citation Corpus.

And then, you can sit back and congratulate yourself for making your publication more useful to researchers who want to be able to reuse the data underlying your publications.

Background

You might ask, “So if submitting Data Citations to Crossref has long been possible, why do you have to write this?”

Historically, authors did not cite data in the way they cited publications. Instead, they would often refer to the data in the main text of the article. This has made it hard to determine what data lay behind the research and/or access the data.

But the research community has increasingly recognized that data is a first-class research output and that we should treat it as such. In short, we should formally cite data.

But because citing data is a comparatively new practice, it has been subject to a lot of new analysis. And unsurprisingly, people analyzing data citation have discovered that there is a lot of nuance to citation of any kind.

There are lots of reasons for citing something. There are lots of internalized conventions for citing things. And there are different conventions for citation for different research objects. And SSH citation practice differs from STEM. And legal citation practices are different from scholarly citation practices. And citation practices even vary by subdiscipline and by journal.

Those who have been looking at what it means to “cite data” have naturally stumbled into a thicket of divergent practices - some of which are historical holdovers, some of which are stylistic preferences, and some of which are clearly adaptations to deal with the specific needs of certain research objects/containers or different disciplines.

The temptation has been to try and rationalize this before extending the practice of citation to data.

“Maybe because data is a distinct record type, we should include the fact that it is a data citation in the citation itself?”

“Maybe because people cite data for different reasons, we should include a typology of citation types in all data citations?”

And so you may hear some people say, “hold off on data citation - we don’t have an optimal way to do it yet, and it can be very complicated.”

But guess what?

We currently don’t label citations to monographs as “citation to monograph.”

And we don’t currently include the reason for citation when we are citing a journal article.

It would be very cool if we did. And it would likely make citations even more useful if we did.

But citations are already useful even without these features. And so, to delay citing data indefinitely because we have an opportunity to improve the act of citation is just perverse. Our community has always opted for progress over perfection.

For one thing - the efforts are not mutually exclusive. We can start citing data with the current limitations of citation practices and simultaneously propose mechanisms for making citation more useful in the future, including new guidelines to deal with the unique issues that citing data poses.

But in the meantime, we will be doing researchers a giant favour if we at least include our imperfect and ambiguous, and unconventional references to data in the references section of an article so that they can be accessed and processed along with all the other imperfect, ambiguous and variant citations that we find so useful.

Some of our members are already doing this. They have been for a long time. And they haven’t found it any more complicated than managing non-data references in the past.

Join them and make your metadata more useful.

Cite data now. Don’t put it off.

And Crossref will continue to work with DataCite and the rest of the community to make the distribution even easier and more useful.

So who is already citing data?

Top 10 members depositing data citations from November-May 2022

(broken down by DOI prefix, which is why you see some publishers listed twice):

Prefix	Member name	Data citations deposited
10.1038	Springer Science and Business Media LLC	7174
10.1016	Elsevier BV	6527
10.1007	Springer Science and Business Media LLC	4748
10.5194	Copernicus GmbH	3017
10.1080	Informa UK Limited	2346
10.1177	SAGE Publications	2082
10.1002	Wiley	2048
10.1111	Wiley	1888
10.1108	Emerald	1876
10.3390	MDPI AG	1827

Top 10 data citations per deposited work

(again, broken down by prefix)

Member name	Prefix	Data citations deposited	Data citations per work
Consortium Erudit	10.7202	580	1.149
SLACK, Inc.	10.3928	462	0.646
S. Karger AG	10.1159	1653	0.532
Proceedings of the National Academy of Sciences	10.1073	973	0.502
American Academy of Pediatrics (AAP)	10.1542	486	0.397
F1000 Research Ltd	10.12688	552	0.341
American Association for the Advancement of Science (AAAS)	10.1126	952	0.317
Springer Science and Business Media LLC	10.1038	7174	0.231
JMIR Publications Inc.	10.2196	864	0.187
American Geophysical Union (AGU)	10.1029	692	0.166

These are for the prefixes with the most data citations deposited (>500 in 6 months) so there might be smaller members doing better than this.

Summaries are great, but I want to see some actual examples!

Here are some examples showing how data is cited by our members:

This eLife article: https://doi-org.pluma.sjfc.edu/10.7554/eLife.26410 cites this dataset in Dryad https://doi-org.pluma.sjfc.edu/10.5061/dryad.854j2.
This Copernicus article: https://doi-org.pluma.sjfc.edu/10.5194/acp-22-7105-2022 cite to this dataset https://doi-org.pluma.sjfc.edu/10.24381/cds.bd0915c6
This Sciendo article: https://doi-org.pluma.sjfc.edu/10.2478/plc-2021-0008 cites this APA-hosted language competence test https://doi-org.pluma.sjfc.edu/10.1037/t15159-000
This De Gruyter article: https://doi-org.pluma.sjfc.edu/10.1515/opth-2020-0160 cites this bibliography at Oxford Bibliographies: https://doi-org.pluma.sjfc.edu/10.1093/OBO/9780195396584-0012

And here are some example API requests for discovering more metadata citations. You can use these API requests as examples and adapt to your own needs.

Service Provider perspectives: A few minutes with our publisher hosting platforms

Jennifer Kemp — Mon, 24 May 2021 00:00:00 +0000

Service Providers work on behalf of our members by creating, registering, querying and/or displaying metadata. We rely on this group to support our schema as it evolves, to roll out new and updated services to members and to work closely with us on a variety of matters of mutual interest. Many of our Service Providers have been with us since the early days of Crossref. Others have joined as scholarly communications has grown and services have evolved. Though fewer than 20 in number, their impact far outweighs the size of the group.

They, like us, work with a great variety of members and have a broad view into publishing trends. In this post, we focus on views from some of the publishing hosting platform Service Providers, who’ve taken the time to share their thoughts on a few questions:

What is the biggest change you’ve experienced working with publisher metadata over the last few years and how have you adapted to it?

It has become more and more important that not only the DOIs are registered with the minimum of necessary metadata to get the DOIs registered, but that a most complete set of metadata is being sent along – including author identifiers, funding information, abstracts, licenses, to support other Crossref services and improve discoverability.

– de Gruyter

Our clients are increasingly aware of the key role metadata plays in the effective dissemination of research. With an increasing number of published articles and a clear domination of “search engines” and aggregation of content, metadata is the primary means of making sure that publications reach the right audience. Publishers’ value-add includes not just copy editing, formatting, and packaging, but also now creating journal articles for the digital age that are discoverable and well linked to the research corpus. Furthermore, we sense a clear move toward standardization, which goes beyond the structure to introduce standardized semantics: adopting common taxonomies for classifying content in different dimensions. Our response is to introduce effective, automated and consistent services that capture, and surface metadata throughout the value chain from authoring to publication and search.

– Atypon

Highwire’s publishers are always looking to use the latest DTD (Document Type Definition) for the content to stay up to current standards. Currently this would be JATS 1.2. They are choosing to remain current so that they can stay on top of all or new metadata that can enrich their deposits. We have handled this well and offer support for the latest version of DTD when they are released, but some publishers are not always familiar with what can/should be deposited with their content and this can be a learning process for them.

– MPS Limited

How do you explain to clients (and others!) why correct, quality metadata is important?

In the digital age, metadata is the key to enabling effective content consumption. Publications that cannot be effectively discovered are of little value. We can only increase the impact of research with “discoverable” and “machine readable” publications. So ensuring correct and quality metadata is the key to optimizing not only the processing (finding the right journal, editor, reviewers) but also to positioning each publication properly. As the volume of published scientific research increases, article metadata is the way forward — it brings “order” and enables our community to manage this volume.

– Atypon

Highwire always positions itself as “good content in” means “good content out”. This is true for our own content stores. Strong and valid metadata will result in valid and strong deposits. We explain this to all new clients on-boarded with Highwire and the use of current standards and for current client projects where content should/can be enriched through re-load.

– MPS Limited

Getting our journals to care about metadata is a two step process: First, make sure they understand how metadata will help their journal succeed (i.e. why it matters to them). Second, make it easy for them to produce metadata while minimizing the cost, time, or complexity of their workflow. The first step – making a case for why metadata matters – is often easier than you’d think. At the very least, most journal editors understand that metadata, e.g., JATS or DOI registration, is an important signifier of professionalism / prestige. In other words, they see that top journals publish metadata and want the same for their journal. From a more technical standpoint, metadata is important because that’s the format computers understand and, like it or not, the publishing ecosystem relies on computers to deliver all sorts of critical services – such as indexing, archiving, and discoverability. So, if you’re not publishing metadata, you’re likely missing the benefit of these services. The second step – making it easy to produce metadata – is more difficult. Journal editors generally understand metadata matters but often lack the technical skills or resources necessary to create metadata. This is where a platform, such as Scholastica, can be very helpful. Because platforms work with many journals, they can invest in tools to automate the creation of metadata, reducing costs for all their clients. For example, most platforms offer integrations to support automatic DOI registration. At Scholastica, we’re pushing this idea even further with automatic integration to more complicated services such as PubMed Central. By reducing cost and complexity, we can help new or small-budget journals have the same quality metadata normally reserved for large, established journals.

– Scholastica

We are sending other publishers’ metadata to academic libraries and distribution channels. Erroneous metadata will have a direct impact on how discoverable a title may be. The more uniform and correct the metadata, the better it will be indexed in other places.

– de Gruyter

What is the one industry development or trend you’re most excited about for the near future and why?

Open Science and the ability to deliver research with the tools for reproducing it is the most exciting and game changing trend. Technology has enabled the output of science to transition from two-dimensional printed text delivery into globally accessible and responsive web-based delivery. We are now taking the next steps to further leverage web technology to enhance research output with rich assets ranging from audio and video, datasets, executable code, high-resolution imagery, interactive applications and more. As more assets accompany research publications, viewing these assets as modular, individually citable, and reusable becomes a requirement. We are reviewing the whole research output flow from authoring to publishing, and most importantly to its dissemination through the myriad of discovery tools now available.

– Atypon

The move of everything to the cloud – this is changing and improving our infrastructure, our possibility to scale and to stay on top of technological development.

– de Gruyter

Thanks very much to the interviewees for their time and thoughts. We look forward to working with our entire Service Provider group on questions like these and many more. If you’d like more details, you can read about our Service Provider program or contact me for more information.

Event Data: A Plan of Action

Martyn Rittman — Mon, 01 Feb 2021 00:00:00 +0000

Event Data uncovers links between Crossref-registered DOIs and diverse places where they are mentioned across the internet. Whereas a citation links one research article to another, events are a way to create links to locations such as news articles, data sets, Wikipedia entries, and social media mentions. We’ve collected events for several years and make them openly available via an API for anyone to access, as well as creating open logs of how we found each event. Some organisations are already using Event Data and we are keen for more to come on board.

Last year we gave an update on Event Data with apologies for being so quiet and a promise of more information at a later date. It’s been some time, so here goes…

I joined Crossref in the middle of last year as a Product Manager and was tasked with looking into Event Data. The first thing I found was a large amount of enthusiasm for Event Data, both within Crossref and further afield. The idea of gathering information beyond the metadata deposited by our members is popular, and creates valuable connections between DOIs and a range of other sources. Interest spans the spectrum of academic research, publishing, bibliometrics, and beyond.

At the same time, I found a project with a very solid, well-built code base but unstable performance. After being put into production in 2018, we didn’t provide sufficient support. Coupled with staff changes and other competing priorities, Event Data hasn’t had the opportunity to live up to early expectations.

To address these issues, we have embarked on a plan to make the server infrastructure more robust, improve monitoring, and make sure that the future of Event Data makes the best use of the resources we have without over-stretching. It means working with the community to determine the most essential aspects of Event Data, and providing support where it’s needed.

The steps below are not necessarily sequential and some depend on the completion of work in other parts of Crossref, but they outline the priorities we have for Event Data in 2021.

The Plan

Stability

Since we put in place our original Event Data infrastructure, the amount of incoming data has grown, and at an ever-increasing rate. In 2017 we were creating 2 million new events per month, that number is now over 20 million. We have known for some time that we need to refresh the infrastructure, but didn’t have the resources to move forward: now we do.

In the first part of the plan we will renew the server infrastructure that underpins Event Data. Maybe not a headline-grabbing move, but the aim is to reduce downtime and pull in missing data. Through improving our monitoring and shortening the response time when things go wrong, we will be able to ensure that events are added on a regular basis and the API can reliably handle requests.

We’ve made the first steps in this direction by upgrading our API infrastructure and making some other tweaks to improve performance. There is still work to do, but we’ve already seen a significant improvement in performance with nearly >99.99% uptime in December.

Consolidation

The second component of the plan is to review performance and data quality. We will evaluate the event sources, update artefacts (such as the lists of publisher landing pages and news websites, and review performance reporting. This will help us to have a better understanding of Event Data in its current form: if the stability component is about improving what comes in and goes and out, this part will give us increased confidence in what Event Data already contains.

Future roadmap

While the two steps above are being carried out, we will revisit the applications of Event Data and talk to organisations that currently use it or have expressed an interest. These conversations will feed into future development in which we will evaluate new sources and other ways to optimize the service.

Central to the roadmap will be continued support of the data citation endpoint in Scholix format, which we run in close collaboration with DataCite. Additionally, we will add new data from relationships between Crossref works, for example a preprint is matched to a journal article, or where there are corrections, retractions, or translations of works.

We expect to continue supporting the current sources of events and where there are organisations with either a strong interest in a particular source or a database of events that they can send directly, we are keen to build collaborations. Event Data, like everything that Crossref does, is a community-based effort.

Staying in touch

To join the conversation about Event Data and keep informed, head over to our Community pages. You can also check out our Gitlab pages. At the end of last year we updated the Education pages where you can learn more about Event Data.

Fast, citable feedback: Peer reviews for preprints and other record types

Martyn Rittman — Wed, 09 Dec 2020 00:00:00 +0000

Crossref has supported depositing metadata for preprints since 2016 and peer reviews since 2018. Now we are putting the two together, in fact we will permit peer reviews to be registered for any record type.

Currently, peer reviews can be registered for journal articles, but that means that they can only be related to some of the content our members deposit. Preprints, books, chapters, working papers, dissertations, and a host of other works can also be registered with Crossref. A number of these frequently undergo some form of review and many of our members and voices in the community have called for us to widen the net on peer reviews, including journal publishers, book publishers, review platforms, and preprint servers. We’ve listened and taken action, and from now on Crossref members can add relationship metadata that links peer reviews to any record type. The metadata will also contain the type of review, stating whether it is a referee report, author response, or community comment, etc. This allows accurate reporting on whether the peer review is happening within a traditional editorial process or elsewhere.

Reviews for preprints

In the last decade there has been an increase in the number of disciplines using preprints. Since enabling registration of preprint metadata, it has become our fastest-growing record type. Preprints, working papers, and other forms of early publication help to accelerate dissemination of the latest research and discovery. They can also promote discussion on important topics, and help authors to improve papers before an editorial decision for journal publication. During the COVID-19 pandemic, preprints have become invaluable for speeding the publication of vital research and case studies.

On the other hand, preprints do not undergo formal review and editorial approval, leading to concerns about the dissemination of false information. While the issue of misinformation in preprints has been discussed for some time, the COVID-19 pandemic has brought it more sharply into focus. organisations that post preprints need to balance the benefits of rapid dissemination with promoting their responsible use.

To support the feedback process, preprint servers along with a growing number of other platforms and services offer scholars the opportunity to post public comments on preprints. By doing this, they give extra context for readers, provide suggestions for authors, and raise awareness of work that could be flawed or too preliminary.

Another growing trend is journal publishers adopting editorial processes that involve preprint-first options and open peer review. As Dr. Stephanie Dawson from ScienceOpen says:

“We have long believed in rewarding reviewers by assigning Crossref DOIs to their open reviews to make them citable objects and we were one of the first users of Crossref’s peer review schema. However, a large percentage of the articles reviewed on ScienceOpen are publicly available preprints. The UCL Open: Environment journal hosted on the platform, for example, is based on a workflow of open peer review of preprints. Our customers, editors, reviewers and authors are therefore extremely happy that these reviews can now also be assigned a Crossref peer review DOI for more accountability and transparency in scholarly publishing.”

At Crossref, we’re continually looking to support more record types and relations between them to build trust, support reproducibility and increase discoverability of content. This is another small step in building the research nexus and we look forward to working with members depositing peer reviews of preprints.

EASE Council Post: Rachael Lammey on the Research Nexus

Rachael Lammey — Mon, 12 Oct 2020 00:00:00 +0000

This blog was initially posted on the European Association of Science Editors (EASE) blog: “EASE Council Post: Rachael Lammey on the Research Nexus”. EASE President Duncan Nicholas accurately introduces it as a whole lot of information and insights about metadata and communication standards into one post…

I was given a wide brief to decide on the topic of my EASE blog, so I thought I’d write one that tries to encompass everything - I’ll explain what I mean by that.

In the past, Crossref has had the opportunity to talk to EASE members about the importance of registering content whose metadata contains important information related to the article. Richer metadata helps to connect the content to other key information such as who wrote it, who it was funded by, the relevant license, the research it cites, any updates to the work such as corrections and retractions, and the data that underpin the research. The use of open persistent identifiers like DOIs, funder IDs, ORCID iDs and ROR IDs are always recommended.

Such rich and connected metadata also helps discoverability of the published research in a different way than just direct access; if you can find something based on looking at the publications related to a particular funder, author, or institution, then there are more ways to come across what you’re looking for. Making links between objects underpinning the research also helps put the research in context and can help further research by making connections to other valuable information that may have been more difficult to make otherwise.

I’ve mentioned the Research Nexus in the title of this post. It’s achieved by declaring relationships between publications and other associated research objects, and from those objects to related publications. The metadata that reveals relationships between research objects can be as informative as the objects themselves. These relationships can assert certain facts that may not be otherwise obvious: this is our goal with the Research Nexus. These relationships and assertions need to exist not just on the web pages of the outputs, but also reflected in a standard way in the metadata so that the information is computer-readable and can be used at scale. As Jennifer Lin, who coined the term, explains:

“Researchers are adopting new tools that create consistency and shareability in their experimental methods. Increasingly, these are viewed as key components in driving reproducibility and replicability. They provide transparency in reporting key methodological and analytical information. They are also used for sharing the artefacts which make up a processing trail for the results: data, material, analytical code, and related software on which the conclusions of the paper rely. Where expert feedback was also shared, such reviews further enrich this record.”

In her Crossref blog, Jennifer goes on to give some examples, including:

Linking to an entire collection of methods and video protocols via Protocols.io
Linking to software and peer reviews in JOSS
Linking to preprint, data, code, source code, peer reviews in Gigascience

I’d include an additional example of linking research to the grant using the grant identifier and associated metadata from the funding section of this PLOS paper (read more about the example from EuroPMC who register grants with Crossref for Wellcome).

These links can be established by adding them into the Crossref relationship metadata schema. The information is then made available to anyone via our open APIs, so that they can easily see and use the information.

In all of these, publishers and other parties are linking to associated research outputs to support the reproducibility and discoverability of content.

The reproducibility point is worth reiterating; EASE has always supported projects to maintain high standards around the review of research, publication standards and ethics, and the reduction of research waste. And connecting articles to data, preprints, protocols, and peer reviews, and making the relationships open for analysis will help achieve this.

We also know that there are work and cost involved in establishing these links, and we’re working on ways to lower the barriers in doing so by:

Revisiting what we charge to encourage best practice. Starting in 2020, we have removed fees for registering vital information on corrections, retractions and other Crossmark metadata. This is timely in light of the updates to the EASE Standardised Retraction form.
We’re also working to remove fees for translations and versions that are linked together by the appropriate relationship metadata so that publishers posting translations or different versions of an article don’t have to pay multiple times for these. Our Membership & Fees Committee is currently reviewing other ways we can support publishers keen to make these connections.
Finding ways to make it easier for publishers to collect this information from authors e.g. submission systems integrations with data repositories to collect robust information on article/data links.
Allowing the registration of peer review metadata for content other than journal articles e.g. books, preprints (coming soon).
Making it easier for publishers to register this information with us at Crossref via the provision of simple to use tools, interfaces and reporting.

The outputs of the research process, such as journal articles, don’t exist in isolation - you only have to look at the interest in the corpus of COVID-19 publications, preprints and associated data to see this. This thinking is also supported by campaigns like Metadata 2020 advocating for “richer, connected, and reusable, open metadata will advance scholarly pursuits for the benefit of society.” The relationships revealed by the Research Nexus may one day help progress research to realise benefits that help us all, providing we all make efforts to effectively support them. More to come…

Events got the better of us

Bryan Vickery — Fri, 27 Mar 2020 00:00:00 +0000

Publisher metadata is one side of the story surrounding research outputs, but conversations, connections and activities that build further around scholarly research, takes place all over the web. We built Event Data to capture, record and make available these ‘Events’ –– providing open, transparent, and traceable information about the provenance and context of every Event. Events are comments, links, shares, bookmarks, references, etc.

In September 2018 we said Event Data was ‘production ready.’ What we meant was development of the service had reached a point where we expected no further major changes to the code, and we encouraged you to use it. What normally would have followed was a detailed handover to our operations team, for monitoring and performance management, and for Product Management to expand Event Data by adding new Crossref member domains and evaluating additional event sources.

Why so quiet?

But many things changed on the staff front, meaning 2019 was a year of reinvention for the Technical and Product teams and of critical knowledge sharing and learning –– Event Data had to take a back seat as we focused resources on other key projects (more on that later). From a technical perspective, we’ve found the Elasticsearch index is not performing well and the approach taken to specifically support data citations through Scholix has not really scaled.

When things go wrong, whether in ways you can or can’t anticipate, the most important thing is communication –– in dealing with the challenges we forgot to do that. We understand how frustrating that can be and we’re extremely sorry to have gone so quiet.

So, where are we today?

Event Data is important to us and clearly important to you too as you’ve contacted us about your use-cases and the reliability of the service. Event Data remains available and you’re welcome to use it, but you should expect instability to continue and be aware that it does not find events for DOIs/domains of our newer members (who joined Crossref since 2019) –– so we’re conscious it might be hard to say whether it’s a good fit for your project at this point.

What are we doing?

We have brought in additional expert Elasticsearch resources to assist with a separate project to migrate our REST API from SOLR to Elasticsearch. We’re making fantastic progress on this. As soon as we’re confident we can make this switch, we will move those same Elasticsearch resources to shoring up Event Data. The REST API takes priority over Event Data because we need to add support for important new record types (like research grants) that aren’t yet available via the API.

We’re also concluding the process of hiring two new Product Managers which means we’ll be in a position to assign someone to head up the product management of Event Data. When we do return to Event Data in the coming months, our initial priority will be increased support for data citation and Scholix. If that means radical changes to the rest of the service, we’ll let you know.

Opening up the discussion

We will have more news on Event Data in mid-2020. We’d love you to join the Crossref Community Forum; we’ve created a new Category for Event Data where you can post details of how you are using, or plan to use Event Data; post questions to the group; suggestions for future development and provide general feedback on the Event Data service.

Crossref metadata for bibliometrics

Ginny Hendricks — Fri, 21 Feb 2020 00:00:00 +0000

Our paper, Crossref: the sustainable source of community-owned scholarly metadata, was recently published in Quantitative Science Studies (MIT Press). The paper describes the scholarly metadata collected and made available by Crossref, as well as its importance in the scholarly research ecosystem.

Containing over 106 million records and expanding at an average rate of 11% a year, Crossref’s metadata has become one of the major sources of scholarly data for publishers, authors, librarians, funders, and researchers. The metadata set consists of 13 record types, including not only traditional types, such as journals and conference papers, but also data sets, reports, preprints, peer reviews, and grants. The metadata is not limited to basic publication metadata, but can also include abstracts and links to full text, funding and license information, citation links, and the information about corrections, updates, retractions, etc. This scale and breadth make Crossref a valuable source for research in scientometrics, including measuring the growth and impact of science and understanding new trends in scholarly communications. The metadata is available through a number of APIs, including REST API and OAI-PMH.

In the paper, we describe the kind of metadata that Crossref provides and how it is collected and curated. We also look at Crossref’s role in the research ecosystem and trends in metadata curation over the years, including the evolution of its citation data provision. We summarize the research that used Crossref’s metadata and describe plans that will improve metadata quality and retrieval in the future.

Using the Crossref REST API (with Open Ukrainian Citation Index)

Rachael Lammey — Wed, 05 Feb 2020 00:00:00 +0000

Over the past few years, I’ve been really interested in seeing the breadth of uses that the research community is finding for the Crossref REST API. When we ran Crossref LIVE Kyiv in March 2019, Serhii Nazarovets joined us to present his plans for the Open Ukrainian Citation Index, an initiative he explains below.

But first an introduction to Serhii and his colleague Tetiana Borysova.

Serhii Nazarovets is a Deputy Director for Research at the State Scientific and Technical Library of Ukraine. Serhii has a Ph.D. in Social Communication Science. His research interests lie in the area of scientometrics and library science. Serhii is the Associate Editor for DOAJ (www.doaj.org) and the Regional Editor for E-LIS (Eprints in Library and Information Science). Serhii has worked in different scientific libraries of Ukraine for more than 10 years. Tetiana Borysova is a Senior Researcher at the State Scientific and Technical Library of Ukraine. Her research interests are focused on topics such as research data management, journal management and scientometrics.

Introducing OUCI

OUCI (Open Ukrainian Citation Index) is a new search engine and a citation database based on publication metadata from Crossref members.

OUCI is intended to simplify the search of scientific publications, to attract the editors’ attention to the problem of completeness and quality of the metadata of Ukrainian scholarly publications, and will allow bibliometricians to freely study the relations between authors and documents from various disciplines, in particular in the field of social sciences and humanities. OUCI is open for every user in the world without any restrictions.

OUCI launched in November 2019. The project is being implemented by the State Scientific and Technical Library of Ukraine with the support of the Ministry of Education and Science of Ukraine.

In Ukraine, we do not have a national citation database, and this significantly impedes the search and analysis of information about Ukrainian publications. According to preliminary estimates, more than 3,000 titles of scientific journals are currently published in Ukraine. At the same time, only around 100 Ukrainian journal titles are indexed in authoritative citation databases, such as Scopus and Web of Science Core Collection. Thus, researchers and managers lack this citation data to understand the impact of Ukrainian journals and their demand in the scientific communication system. Our approach is that OUCI database contains metadata from all publishers that use the Crossref’s Cited-by service and who support the Initiative for Open Citations by making the reference metadata they publish with Crossref openly available.

How is Crossref metadata used in OUCI?

A publication can only be indexed in OUCI if there is a DOI. At first glance, the idea of creating an index of national publications based on this condition may seem too optimistic. However, in January 2018, a new requirement was adopted by the List of scientific publications of Ukraine (a list of Ukrainian journals recognized by experts as qualitative for publishing their research results for a scientific degree), which listed a DOI as one of the requirements for inclusion. After that, the number of publishers who received the DOI prefix from Crossref has tripled, to 352 in November 2019.

Another important feature of OUCI is that publishers have to use Crossref’s Cited-by service and support the Initiative for Open Citations. We are working to build a new fair infrastructure where everyone who is interested in the dissemination of scientific knowledge can present their publications to the community, develop expert judgment skills and access citations to explore the links between documents. The philosophy of the index is to use only open resources to fill it.

In addition to standard filters from Crossref metadata (such as publisher, publication, type, year), OUCI offers to refine search results by:

indexation in Web of Science and/or Scopus,
journal category (A or B according to the List of scientific publications of Ukraine),
the field of knowledge and scientific specialties (according to the Ukrainian legislation) and other aspects important to Ukrainian users characteristics.

Figure 1: OUCI search and filter options

Beyond the ability to search articles, OUCI displays profiles for Ukrainian journals (the titles of these journals will include hyperlinks in the search results). Administrators can manage them, add and edit information about their journals: web-site, aims and scope, scientific fields of the journal according to the Ukrainian classification. Also, you can see some quantitative characteristics of journals: number of publications, number of citations, h-index, i10-index etc.

Figure 2: Display of journal information in OUCI

In addition, we have implemented an analytics module. Using the data about the number of articles and citations from Crossref, it allows users to analyze Ukrainian journals by field.

Figure 3: Publication and citation information

What are the future plans for OUCI?

In the near future, we plan to add:

the ability to export search results for further analysis;
integration with Unpaywall;
alternative metrics from Crossref Event Data.

In the ideal future for our index, every Ukrainian article will be registered with Crossref and have open references. We plan to promote the importance of reach and quality metadata in Crossref among Ukrainian publishers. We also encourage all publishers to support the Initiative for Open Citations.

What else would OUCI like to see in Crossref metadata?

One of the main problems we encountered when creating OUCI was the metadata about the authors. Very few publications contain data about the author’s ORCID iD. Focusing publishers on the need to transmit full metadata to Crossref, as well as monitoring their quality is a must for the resources like this. Also we look forward to the growing usage of ROR (Research Organization Registry) - identifiers for research organisations, similar to the way that ORCID offers identifiers for researchers. We believe that the ROR will help to obtain reliable data for analyzing the scientific activity of Ukrainian institutions.

Another issue we’ve identified in some Ukrainian journals that some of the small publishers that register content via Crossref Sponsors did not take care getting their own prefix, so it can be difficult to see their publications - this is something that showing the metadata via an index can help them see and therefore fix.

Questions?

We’ve had lots of questions about OUCI in the run up to the launch and now that it’s live. Here is a selection of our FAQs, all available on our website. You can also get in touch directly if you have another question we haven’t answered yet.

Metadata Corrections, Updates, and Additions in Metadata Manager

Shayn Smulyan — Mon, 13 Jan 2020 00:00:00 +0000

It’s been a year since Metadata Manager was first launched in Beta. We’ve received a lot of helpful feedback from many Crossref members who made the switch from Web Deposit Form to Metadata Manager for their journal article registrations.

The most common use for Metadata Manager is to register new DOIs for newly published articles. For the most part, this is a one-time process. You enter the metadata, register your DOI, and success!

But everything doesn’t always go quite as expected. Humans make mistakes, and typos in metadata are bound to happen on occasion, even for the most careful users.

We always want to make it as easy as possible for our members to find and correct metadata errors, and to add additional metadata when it becomes available. Our Schematron, Conflict, and Resolution reports can help you identify existing metadata errors. We never charge content registration fees for metadata updates, additions, or corrections, so cost won’t be a barrier to getting the most accurate and thorough metadata possible. And, now, Metadata Manager can make those corrections easier to do.

Correcting Errors

Because accurate and comprehensive metadata is so important for the linking and discoverability of your publications, it’s important to catch these occasional errors and correct them.

We send out reports that automatically screen for particular types of metadata errors, and we pass along comments from users who contact us with concerns about metadata quality to our contacts at the relevant publisher.

The “Review all” feature in Metadata Manager also allows you to do a final check of all the metadata you entered right before you’re about to submit your deposits. So, we also rely on you to evaluate your own accuracy there as well.

Once you’ve identified an error, you’ll need to correct it. To do that, you must resubmit a whole new metadata deposit for the affected item. The newly deposited metadata will entirely overwrite the previously deposited metadata.

If you’re used to using the Web Deposit Form, you know that the redeposit can be a little tedious. For example, if you find that you misspelled an author’s last name, you’d have to manually type in or copy-paste not just the corrected last name, but all of the journal-level, issue-level, and article-level metadata that applies to the article.

Using Metadata Manager, the process is much simpler. The full metadata record is retained or imported and you only need to correct the error itself.

For articles originally registered using Metadata Manager

If you find a metadata error in an article which you initially registered in Metadata Manager itself, you can locate the article in one of two ways:

Navigate through the list of Accepted articles within a given journal
Or, search by article title in the Deposit History

Once you’ve located the relevant article, click on the article title to open the article’s metadata record. From there, you can make the necessary corrections. With the corrections complete, click “Continue” and then “Add to deposit.” After that, the process is exactly the same as depositing a new article.

For articles registered using the Web Deposit Form or any other deposit method

If you registered an article using the Web Deposit Form, an XML deposit, or the OJS plugin, you can still use Metadata Manager to quickly correct an error. But, first you have to import the article’s metadata into Metadata Manager.

To do this, click into the relevant journal from your Metadata Manager home page. Then, search for the article title using the “Add existing article” search box. Select “Add” next to the article title in the search results, which will import the article’s metadata record into Metadata Manager.

From here, make any necessary corrections and click “Continue” and then “Add to deposit.” Navigate to the “To deposit” tab and “Review all” to ensure that your metadata record is accurate. Then select “Deposit” to finalize your submission. You’ll receive immediate feedback as to whether your metadata deposit was successful or not.

Adding additional metadata

Perhaps there are no problems with your metadata, and everything is completely accurate. That’s great! But, we encourage our members to submit metadata that is not just accurate, but also as thorough as possible. Check your Participation Report to see if there are any types of metadata that you haven’t been submitting yet, or that you haven’t been submitting for certain journals.

Metadata Manager allows you to deposit references, licenses, and relationships between your articles and other DOIs, which weren’t possible to add using the Web Deposit Form. The same process described above for corrections will allow you to import previously registered articles and add in these new metadata elements.

We also know that many of our members register DOIs for their articles when they’re first published online, but aren’t yet included in an issue. When the articles are published in their final versions, there is important metadata added which wasn’t yet available when the DOI was first registered. This includes things like volume number, issue number, page numbers, and full publication date, all of which are extremely important for linking and discoverability. Sometimes the resolution URL changes when the article is moved from its pre-publication status to its final version.

So, when each issue is published, you can use Metadata Manager to pull up all the already-registered articles included in that issue and add in the newly relevant metadata like page numbers, issue number, URL, etc. Then add them to a new deposit, review, and submit.

Please check out the full Metadata Manager help documentation for more details, or join us on an upcoming workshop to test out Metadata Manager in real-time with us. And, as always, feel free to email us at support@crossref.org with any questions.

What's your (citations') style?

Dominika Tkaczyk — Tue, 29 Oct 2019 00:00:00 +0000

Bibliographic references in scientific papers are the end result of a process typically composed of: finding the right document to cite, obtaining its metadata, and formatting the metadata using a specific citation style. This end result, however, does not preserve the information about the citation style used to generate it. Can the citation style be somehow guessed from the reference string only?

TL;DR

I built an automatic citation style classifier. It classifies a given bibliographic reference string into one of 17 citation styles or “unknown”.
The classifier is based on supervised machine learning. It uses TF-IDF feature representation and a simple Logistic Regression model.
For training and testing, I used datasets generated automatically from Crossref metadata.
The accuracy of the classifier estimated on the test set is 94.7%.
The classifier is open source and can be used as a Python library or REST API.

Introduction

Threadgill-Sowder, J. (1983). Question Placement in Mathematical Word Problems. School Science and Mathematics, 83(2), 107-111

This reference is the end result of a process that typically includes: finding the right document, obtaining its metadata, and formatting the metadata using a specific citation style. Sadly, the intermediate reference forms or the details of this process are not preserved in the end result. In general, just by looking at the reference string we cannot be sure which document it originates from, what its metadata is, or which citation style was used.

Global multi-billion dollar fashion industry proves without a doubt that people care about their fashion style. But why should we care about the citation style used to generate a specific reference? This might seem like an insignificant piece of information, but it can be a powerful clue when we try to solve tasks like:

Reference parsing, i.e., extracting metadata from the reference string. If the style is known, we also know where to expect metadata fields in the string, and it is typically enough to use simple regular expressions instead of complicated (and slow) machine learning-based parsers.
Discipline/topic classification. Citation styles used in documents correlate with their discipline. As a result, knowing the citation style used in the document could provide a useful clue for a discipline classifier.
Extracting references from documents. Conforming to a specific style might suggest that the reference string was correctly located within a larger document.

Even though the style is not directly mentioned in the reference string, the string contains useful clues. Some styles will abbreviate the authors’ first names, and others won’t. Some will place the year in parentheses, others separate it with commas. The presence of such fragments in the reference string can be used as the input for the style classifier.

I used these clues to build an automatic style classifier. It takes a single reference string on the input and classifies it into one of 17 styles or “unknown”. You can use it as a Python library or via REST API. The source code is also available. If you find this project useful, I would love to hear about it!

And if you are interested in more details about the classifier and how it was built, read on.

Data

The data for the experiments was generated automatically. The training and the test set were generated in the same way but from two different samples. The process was the following:

5,000 documents were randomly chosen from Crossref collection.
Each document was formatted into 17 citation styles. This resulted in 85,000 pairs (reference string, citation style).
Very short reference strings were removed. A short reference string typically results from very incomplete metadata of the document.
From a number of randomly selected references, I removed fragments like the name of the month. These fragments appear in the automatically generated reference strings because sometimes months are included in the metadata records in Crossref collection. However, they rarely appear in the real-life reference strings, so removing them made the dataset more reliable.
5,000 strings labelled as “unknown” were also added. These were generated by randomly swapping the words in the “real” reference strings.

This process resulted in two sets: training set containing 87,808 data points and test set containing 87,625 data points. The training set was used to choose various classification parameters and to train the final model. The test set was used to obtain the final estimation of the classifier’s accuracy.

Styles

The classifier was trained on the following 17 citation styles (+ “unknown”):

acm-sig-proceedings
american-chemical-society
american-chemical-society-with-titles
american-institute-of-physics
american-sociological-association
apa
bmc-bioinformatics
chicago-author-date
elsevier-without-titles
elsevier-with-titles
harvard3
ieee
iso690-author-date-en
modern-language-association
springer-basic-author-date
springer-lecture-notes-in-computer-science
vancouver

These 17 styles were chosen to cover a vast majority of references that we see in the real-life data, without including too many variants of very similar styles.

If you need a different style set, fear not. You can use the library to train your own model based on exactly the styles you need.

Features

Our learning algorithm cannot work directly with the raw text on the input. It needs numerical features. In the case of text classification (and reference strings are text), one very common feature representation is bag-of-words. In the simplest variant, each feature represents a single word, and the value of the feature is binary: 1 if the word is present in the text, 0 otherwise.

There are many variants of this representation, for example:

The input text typically undergoes normalization before the features are extracted. Depending on the use case, this might include lowercasing, removing punctuation, bringing the words to their canonical form by stemming, etc.
We do not have to use single words as features. In some use cases, it is beneficial to use n-grams, which correspond to fixed-length sequences of words.
Instead of binary values, we might want to use some other feature weight schemes, such as the famous TF-IDF representation.

Our use case is not a typical case of text classification. We cannot use raw words as features, as words do not carry the information about the citation style. Imagine the same document formatted in different styles –– those reference strings will contain the same words, and the learning algorithm won’t be able to distinguish between them.

As a side note, in some cases, some specific words might be important. For example, if the reference contains the word “algorithm”, chances are the document is from computer science. If so, then perhaps the citing paper is from computer science as well. And in computer science, some styles are more popular than others. Machine learning algorithms are pretty good at detecting such correlations in the data. In the first version of our classifier, however, we do not take this into account. This keeps things simpler.

If not words, then what matters in our case? It seems that the information about the style is present in punctuation, capitalization and abbreviations.

To capture these clues, before extracting the features we first map our reference string into a sequence of “word types” (or “character types”). The types are the following: lowercase-word, lowercase-letter, uppercase-word, uppercase-letter, capitalized-word, other-word, year, number, dot, comma, left-parenthesis, right-parenthesis, left-bracket, right-bracket, colon, semicolon, slash, dash, quote, other.

In addition, we mark the beginning and the end of the reference string with special types start and end.

So for example this string:

Eberlein, T. J. Yearbook of Surgery 2006, 322–324.

is mapped into this sequence:

start capitalized-word comma uppercase-letter dot uppercase-letter dot capitalized-word lowercase-word capitalized-word year comma number dash number dot end

This transformation effectively brings together different words, as long as their form is the same.

After transforming the reference string we extract 2-grams, 3-grams and 4-grams. The values of the features are TF-IDF weights.

Some example features in our representation include:

lowercase-word lowercase-word lowercase-word lowercase-word - a sequence of four lowercase words. It is most likely the part of the article title and won’t have a huge impact on the decision about the citation style.
capitalized-word comma uppercase-letter dot - typical representation of an author in some styles, where the first name is given as an initial only and follows the last name.
left-parenthesis year right-parenthesis - typical for styles that enclose the year in parentheses.
number dash number - this sequence is most likely pages range.

Learning algorithm

I tested four learning algorithms (naive Bayes, logistic regression, linear support vector classification and random forest) in a 5-fold cross validation on the training set. The plot shows the distribution of accuracies obtained by each algorithm:

Based on these results, logistic regression was chosen as the algorithm with the best mean accuracy and the lowest variance of the results.

Final accuracy estimation

The final model was trained on the entire training set and evaluated on the test set. As evaluation metric accuracy was used. In this case, accuracy is simply the fraction of the references in the test set correctly classified by the classifier.

The accuracy on the test set was 94.7%. The confusion matrix shows which styles were most often confused with each other:

The most often confused styles are chicago-author-date and american-sociological-association. Let’s see some example strings from these two styles:

Legros, F. 2003. "Can Dispersive Pressure Cause Inverse Grading in Grain Flows?: Reply." Journal of Sedimentary Research 73(2):335–335
Legros, F. 2003. "Can Dispersive Pressure Cause Inverse Grading in Grain Flows?: Reply." Journal of Sedimentary Research 73 (2) : 335–335

Clarke, Jennie T. 2011. "Recognizing and Managing Reticular Erythematous Mucinosis." Archives of Dermatology 147(6):715
Clarke, Jennie T. 2011. "Recognizing and Managing Reticular Erythematous Mucinosis." Archives of Dermatology 147 (6) : 715

Chalmers, Alan, and Richard Nicholas. 1983. "Galileo on the Dissipative Effect of a Rotating Earth." Studies in History and Philosophy of Science Part A 14(4):315–40
Chalmers, Alan, and Richard Nicholas. 1983. "Galileo on the Dissipative Effect of a Rotating Earth." Studies in History and Philosophy of Science Part A 14 (4) : 315–340

It seems that the styles are indeed very similar. The strings look almost identical, apart from spacing, which is not included in any way in our feature representation. No wonder that the classifier confuses these two styles a lot.

A more detailed analysis of the classifier can be found here.

What if I told you that bibliographic references can be structured?

Dominika Tkaczyk — Mon, 08 Jul 2019 00:00:00 +0000

Last year I spent several weeks studying how to automatically match unstructured references to DOIs (you can read about these experiments in my previous blog posts). But what about references that are not in the form of an unstructured string, but rather a structured collection of metadata fields? Are we matching them, and how? Let’s find out.

TL;DR

43% of open/limited references deposited with Crossref have no publisher-asserted DOI and no unstructured string. This means they need a matching approach suitable for structured references. [EDIT 6th June 2022 - all references are now open by default].
I adapted our new matching algorithms: Search-Based Matching (SBM) and Search-Based Matching with Validation (SMBV) to work with both structured and unstructured references.
I compared three matching algorithms: Crossref’s current (legacy) algorithm, SBM and SBMV, using a dataset of 2,000 structured references randomly chosen from Crossref’s references.
SBMV and the legacy algorithm performed almost the same. SBMV’s F1 was slightly better (0.9660 vs. 0.9593).
Similarly as in the case of unstructured references, SBMV achieved slightly lower precision and better recall than the legacy algorithm.

Introduction

Those of you who often read scientific papers are probably used to bibliographic references in the form of unstructured strings, as they appear in the bibliography, for example:

[5] Elizabeth Lundberg, “Humanism on Gallifrey,” Science Fiction Studies, vol. 40, no. 2, p. 382, 2013.

This form, however, is not the only way we can store the information about the referenced paper. An alternative is a structured, more machine-readable form, for example using BibTeX format:

@article{Elizabeth_Lundberg_2013,
year = 2013,
publisher = {{SF}-{TH}, Inc.},
volume = {40},
number = {2},
pages = {382},
author = {Elizabeth Lundberg},
title = {Humanism on Gallifrey},
journal = {Science Fiction Studies}
}

Probably the most concise way to provide the information about the referenced document is to use its identifier, for example (🥁drum roll…) the DOI:

<https://doi-org.pluma.sjfc.edu/10.5621/sciefictstud.40.2.0382>

It is important to understand that these three representations (DOI, structured reference and unstructured reference) are not equivalent. The amount of information they carry varies:

The DOI, by definition, provides the full information about the referenced document, because it identifies it without a doubt. Even though the metadata and content are not directly present in the DOI string, they can be easily and deterministically accessed. It is by far the preferred representation of the referenced document.
The structured reference contains the metadata of the referenced object, but it doesn’t identify the referenced object without a doubt. In our example, we know that the paper was published in 2013 by Elizabeth Lundberg, but we might not know exactly which paper it is, especially if there are more than one document with the same or similar metadata.
The unstructured reference contains the metadata field values, but without the names of the fields. This also doesn’t identify the referenced document, and even its metadata is not known without a doubt. In our example, we know that the word “Science” appears somewhere in the metadata, but we don’t know for sure whether it is a part of the title, journal title, or maybe the author’s (very cool) name.

The diagram presents the relationships between all these three forms:

The arrows show actions that Crossref has to perform to transform one form to another.

Green transformations are in general easy and can be done without introducing any errors. The reason is that green arrows go from more information to less information. We all know how easy it is to forget important stuff!

Green transformations are typically performed when the publication is being created. At the beginning the author can access the DOI of the referenced document, because they know exactly which document it is. Then, they can extract the bibliographic metadata (the structured form) of the document based on the DOI, for example by following the DOI to the document’s webpage or retrieving the metadata from Crossref’s REST API. Finally, the structured form can be formatted into an unstructured string using, for example, the CiteProc tool.

We’ve also automated it further and these two green transformation (getting the document’s metadata based on the DOI and formatting it into a string) can be done in one go using Crossref’s content negotiation.

Red transformations are often done in systems that store bibliographic metadata (like our own metadata collection), often at a large scale. In these systems, we typically want to have DOIs (or other unique identifiers) of the referenced documents, but in practise we often have only structured and/or unstructured form. To fix this, we match references. Some systems also perform reference parsing (thankfully, we discovered we do not need to do this in our case).

In general, red transformations are difficult, because we have to go from less information to more information, effectively recreating the information that has been lost during paper writing. This requires a bit of reasoning, educated guessing, and juggling probabilities. Data errors, noise, and sparsity make the situation even more dire. As a result, we do not expect any matching or parsing algorithm to be always correct. Instead, we perform evaluations (like in this blog post) to capture how well they perform on average.

My previous blog post focused on matching unstructured references to DOIs (long red “matching” arrow). In this one, I analyse how well we can match structured references to DOIs (short red “matching” arrow).

References in Crossref

You might be asking yourself how important it is to have the matching algorithm working for both structured and unstructured references. Let’s look more closely at the references our matching algorithm has to deal with.

29% of open/limited references deposited with Crossref already have the DOI provided by the publisher member. At Crossref, when we come across those references, we start dancing on a rainbow to the tunes of Linkin Park, while the references holding their DOIs sprinkle from the sky. Some of us sing along. We live for those moments, so if you care about us, please provide as many DOIs in your references as possible!

You might be wondering how we are sure these publisher-provided DOIs are correct. The short answer is that we are not. After all, the publisher might have used an automated matcher to insert the DOIs before depositing the metadata. Nevertheless, our current workflow assumes these publisher-provided DOIs are correct and we simply accept them as they are.

Unfortunately, the remaining 71% of references are deposited without a DOI. Those are the references we try to match ourselves.

Here is the distribution of all the open/limited references:

17% of the references are deposited with no DOI and both structured and unstructured form. 11% have no DOI and only an unstructured form, and 43% have no DOI and only a structured form. These 43% cannot be directly processed by the unstructured matching algorithm.

This distribution clearly shows that we need a matching algorithm able to process both structured and unstructured references. If our algorithm worked only with one type, we would miss a large percentage of the input references, and the quality of our citation metadata would be questionable.

The analysis

Let’s get to the point. I evaluated and compared three matching algorithms, focusing on the structured references.

The first algorithm is one of the legacy algorithms currently used in Crossref. It uses fuzzy querying in a relational database to find the best matching DOI for the given structured reference. It can be accessed through a Crossref OpenURL query.

The second algorithm is an adaptation of the Search-Based Matching (SBM) algorithm for structured references. In this algorithm, we concatenate all metadata fields of the reference and use it to search in the Crossref’s REST API. The first hit is returned as the target DOI if its relevance score exceeds the predefined threshold.

The third algorithm is an adaptation of the Search-Based Matching with Validation (SBMV) for structured references. Similarly as in the case of SBM, we also concatenate all metadata fields of the input reference and use it to search in the Crossref’s REST API. Next, a number of top hits are considered as candidates and their similarity score with the input reference is calculated. The candidate with the highest similarity score is returned as the target DOI if its score exceeds the predefined threshold. The similarity score is based on fuzzy comparison of the metadata field values between the candidate and the input reference.

I compared these three algorithms on a test set composed of 2,000 structured bibliographic references randomly chosen from Crossref’s metadata. For each reference, I manually checked the output of all matching algorithms, and in some cases performed additional manual searching. This resulted in the true target DOI (or null) assigned to each reference.

The metrics are the same as in the previous evaluations: precision, recall and F1 calculated over the set of input references.

The thresholds for SBM and SBMV algorithms were chosen on a separate validation dataset. The validation dataset also contains 2,000 structured references with manually-verified target DOIs.

The results

The plot shows the results of the evaluation of all three algorithms:

The vertical black lines on top of the bars represent the confidence intervals.

As we can see, SBMV and the legacy approach achieved very similar results. SBMV slightly outperforms the legacy approach in F1: 0.9660 vs. 0.9593.

SBMV is slightly worse that the legacy approach in precision (0.9831 vs. 0.9929) and better in recall (0.9495 vs. 0.9280).

The SBM algorithm performs the worst, especially in precision. Why is there such a huge difference between SBM and SBMV? The algorithms differ in the post-processing validation stage. SBM relies on the ability of the search engine to select the best target DOI, while SBMV re-scores a number of candidates obtained from the search engine using custom similarity. The results here suggest that in the case of structured references, the right target DOI is usually somewhere close to the top of the search results, but often it is not in the first position. One of the reasons might be missing titles in 76% of the structured references, which can confuse the search engine.

Let’s look more closely at a few interesting cases in our test set:

first-page = 1000
article-title = Sequence capture using PCR-generated probes: a cost-effective method of targeted high-throughput sequencing for nonmodel organisms
volume = 14
author = Peñalba
year = 2014
journal-title = Molecular Ecology Resources

The reference above was successfully matched by SBMV to https://doi-org.pluma.sjfc.edu/10.1111/1755-0998.12249, even though the document’s volume and pages are missing from Crossref’s metadata.

issue = 2
first-page = 101
volume = 6
author = Abraham
year = 1987
journal-title = Promoter: An Automated Promotion Evaluation System

Here the structure incorrectly labels article title as journal title. Despite this, the reference was correctly matched by our brave SBMV to https://doi-org.pluma.sjfc.edu/10.1287/mksc.6.2.101.

author = Marshall Day C.
volume = 39
first-page = 572
year = 1949
journal-title = India. J. A. D. A.

Above we have most likely a parsing error. A part of the article title appears in the journal name, and the main journal name is abbreviated. ‘I see what you did there, my old friend Parsing Algorithm! Only a minor obstacle!’ said SBMV, and matched the reference to https://doi-org.pluma.sjfc.edu/10.14219/jada.archive.1949.0114.

volume = 5
year = 2015
article-title = A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels
journal-title = BMJ Open

Here the the page number and author are not in the structure, but our invincible SBMV jumped over the holes left by the missing metadata and gracefully grabbed the right DOI https://doi-org.pluma.sjfc.edu/10.1136/bmjopen-2015-009138.

issue = 2
first-page = 533
volume = 30
author = Uthman BM
year = 1989
journal-title = Epilepsia

In this case we have a mismatch in the page number (“533” vs. “S33”). But did SBMV give up and burst into tears? I think we already know the answer! Of course, it conquered the nasty typo with the sword made of fuzzy comparisons (yes, it’s a thing!) and brought us back the correct DOI https://doi-org.pluma.sjfc.edu/10.1111/j.1528-1157.1989.tb05823.x.

Structured vs. unstructured

How does matching structured references compare to matching unstructured references?

The general trends are the same. For both structured and unstructured references, SBMV outperforms the legacy approach in F1, achieving worse precision and better recall. This tells us that our legacy algorithms are more strict and as a result they miss some links.

Structured reference matching seems easier than unstructured reference matching. The reason is that when we have the structure, we can compare the input reference to the candidate field by field, which is more precise than using the unstructured string.

Structured matching, however, in practise brings new challenges. One big problem is data sparsity. 15% of structured references without DOIs have fewer than four metadata fields. This is not always enough to identify the DOI. Also, 76% of the structured references without DOIs do not contain the article title, which poses a problem for candidate selection using the search engine.

What’s next?

So far, I have focused on evaluating SBMV for unstructured and structured references separately. 17% of the open/limited references at Crossref, however, have both unstructured and structured form. In those cases, it might be beneficial to use the information from both forms. I plan to perform some experiments on this soon.

The data and code for this evaluation can be found at https://github.com/CrossRef/reference-matching-evaluation. The Java version of SBMV (for both structured and unstructured references) can be found at https://gitlab.com/crossref/search-based-reference-matcher.

Underreporting of matched references in Crossref metadata

Geoffrey Bilder — Tue, 05 Feb 2019 00:00:00 +0000

TL;DR

About 11% of available references in records in our OAI-PMH & REST API don’t have DOIs when they should. We have deployed a fix, but it is running on billions of records, and so we don’t expect it to be complete until mid-April.

Note that the Cited-by API that our members use appears to be unaffected by this problem.

The gory details

When a Crossref member registers metadata for a publication, they often include references. Sometimes the member will also include DOIs in the references, but often they don’t. When they don’t include a DOI in the reference, Crossref tries to match the reference to metadata in the Crossref system. If we succeed, we add the DOI of the matched record to the reference metadata. If we fail, we append the reference to an ever-growing list which we re-process on an ongoing basis.

You may have seen that the R&D team has been doing work to improve our reference matching system. We will soon be rolling out a new reference matching process that will increase recall significantly.

But while testing our new reference matching approach, we started to see inconsistent results with our existing legacy reference matching system. When we implemented new regression tests, we noticed that, even when using our legacy system, we were consistently getting better results than were reflected in the metadata we exposed via our APIs. For example, we would pick a random Crossref DOI record that included 3 matched references, and when we tried matching all the references in the record again using our existing technology, we would get more matched references than were reported in the metadata.

At first, we thought this might have something to do with sequencing issues. For example, that article A might cite article B, but somehow article A would get its DOI registered with Crossref prior to article B. In this theoretical case, we would initially fail to match the reference, but it would eventually get matched as we continued to reprocess our unmatched references. But this wasn’t the issue. And the problem was not with the matching technology we are using. Instead, we discovered a problem with the way we process references on deposit.

When a member deposits references with Crossref, each reference has to include a member-defined key that is unique to each reference they are depositing in the DOI record. When we match a reference- we report to the members that we matched the reference with key X to DOI Y. The problem is that sometimes members would deposit references with an empty key. If there was only one such reference, then, technically, it would pass our test for making sure the key was unique within the record. So we would process the reference, and match it, and report it via our Cited-by service, but later in the process, when we went to include the matched DOI in the reference section of our API metadata, we’d skip including DOIs for references that had blank keys. The reference itself would be included in the metadata, it would just appear that we hadn’t matched it to a DOI when we actually had.

Again, we estimate this to have resulted in about 11% of the references in our metadata to be missing matched DOIs. We are processing our references again and inserting the correctly matched DOIs in the metadata. We expect the process to complete in mid-April. We will keep everybody up-to-date on the progress of this fix.

We will also be integrating the new matching system that we’ve developed. As mentioned at the start of this post, this matching system will also increase the recall rate of our reference matching and so, the two changes combined, should result in users seeing a significant increase in the number of matched references included in Crossref metadata.

And finally, as part of the work that we are doing to improve our reference matching, we are putting a comprehensive testing framework that will make it easier for us to detect inconsistencies and/or regressions in our reference matching.

Please contact Crossref support with any questions or concerns.

Improved processes, and more via Metadata Manager

Shayn Smulyan — Thu, 17 Jan 2019 00:00:00 +0000

Hi, Crossref blog-readers. I’m Shayn, from Crossref’s support team. I’ve been fielding member questions about how to effectively deposit metadata and register content (among other things) for the past three years. In this post, I’ll take you through some of the improvements that Metadata Manager provides to those who currently use the Web Deposit form.

We recently announced the launch of Metadata Manager, a new tool from Crossref that makes it easier for you to submit robust, accurate, and thorough metadata for the content you register. Metadata Manager already covers journals and articles; more record types will be supported soon. It offers some extra features that will make your experience less stressful, make your metadata better, and ultimately make your content more discoverable.

Metadata Manager has the potential to improve your metadata registration experience in a number of ways:

by correcting one-off errors in previously registered metadata
by directly allowing you to add references, license data, funder information, or any other ancillary metadata to items that have previously been registered
by updating Crossmark data, in the case of a retraction or withdrawal

With the Web Deposit form, you finish entering your metadata for a new issue of your journal, and then get asked for your password, and of course that’s when you realize you’ve forgotten it (it happens a lot!). With Metadata Manager, the very first step is to log in, so you know your login credentials are accurate before you get down to the task of entering your metadata.

Easily import journals, or add new ones

When you switch to Metadata Manager, you can import the journals already associated with your account. Simply go to the search bar on the Home screen, search for your journal by title, then click ’Add’. If you are registering your first article for a journal that you’ve not registered before, you can add the journal information on the Home screen, by clicking “New Publication”.

Adding a Journal DOI

In the Web Deposit form, the Journal DOI is optional, as long as you include a valid ISSN. However, with Metadata Manager, a Journal DOI must be created for each journal you register. So, you need to enter a Journal DOI and a Journal URL for each of your journals before your deposits can be submitted. The Journal DOI won’t become active until you submit your first successful deposit for an article within that journal.

If you’ve never registered a Journal DOI before and are unsure what to use for your Journal DOI’s suffix, take a look at our suggested best practice for constructing DOI suffixes.

Adding new articles

Once your journal is added, the process of adding articles in Metadata Manager should be familiar, as it’s similar to the Web Deposit form process. You type in or paste as plain text (without formatting) all your relevant, accurate, and thorough metadata into the appropriate fields in the form.

Save your work as you go

In Metadata Manager there is no need to complete a full issue’s worth of articles at once. And, you don’t need to worry about losing your progress if you accidentally close your browser window, or your laptop runs out of battery while you’re in the middle of a deposit. You can simply and easily ‘save-as-you-go’, one article at a time, until you’re ready to submit them all. You can even review your saved metadata to make sure there aren’t any errors before the deposit is finalized.

Other metadata fields you didn’t know you needed (but you do!)

Have you ever wanted to add an abstract to your content’s metadata? How about license information, so that other organisations know what they can and can’t do with the work? Does your journal use article ID numbers instead of page numbers? These are all elements that can be added to Metadata Manager that were not available in the Web Deposit form. Additionally, you can add funding data, Similarity Check links, and relationships between your articles and other content. These types of metadata are hugely valuable for building a robust, interconnected web of scholarly communication.

Adding references

Unlike the Web Deposit form, Metadata Manager allows you to easily add references to your article’s metadata—this is an important requirement for participating in our Cited-by service.

To add references to an article’s metadata, you can copy and paste its reference list into the references field on the same screen as the rest of the article metadata (as per the image below).

Metadata Manager will match DOIs to those references (where available), and include the full list in your record. So, if you’ve been putting off participating in Cited-by because the reference deposit requirement was too much of a hassle, we hope this will help ease the way! The more references everyone registers, the more robust our Cited-by counts and Cited-by data become.

Edit mistakes without having to re-enter all your metadata

Mistakes happen. Sometimes you put an author’s first name in the last name field. Sometimes you copy and paste some stray HTML tags into your abstract. You might break a link by leaving a space in the middle of a URL, or enter the first-page number as 3170 instead of 317.

With Metadata Manager you can fix any errors quickly and easily right in the interface, then just click to redeposit the article with its metadata corrected. You won’t need to re-enter all the metadata or worry about editing the XML files directly.

We’ll have another blog post coming soon that will be devoted entirely to updating, correcting, or otherwise editing metadata for already-registered DOIs in Metadata Manager.

Find out immediately if your registration was successful

When you have finished adding the metadata for your articles, navigate to the “To deposit” section and click ‘Deposit’ to submit them. Instead of having to wait for your content to go through our processing queue, you’ll get immediate feedback. The number of Accepted and Failed deposits show immediately. Any articles which have failed are clearly marked with a red triangle icon and an explanation for the error. If you don’t understand an error message or how to correct the metadata, please contact us at support@crossref.org.

To get started with Metadata Manager take a look at our full help documentation.

Reference matching: for real this time

Dominika Tkaczyk — Tue, 18 Dec 2018 00:00:00 +0000

In my previous blog post, Matchmaker, matchmaker, make me a match, I compared four approaches for reference matching. The comparison was done using a dataset composed of automatically-generated reference strings. Now it’s time for the matching algorithms to face the real enemy: the unstructured reference strings deposited with Crossref by some members. Are the matching algorithms ready for this challenge? Which algorithm will prove worthy of becoming the guardian of the mighty citation network? Buckle up and enjoy our second matching battle!

TL;DR

I evaluated and compared four reference matching approaches: the legacy approach based on reference parsing, and three variants of search-based matching.
The dataset comprises 2,000 unstructured reference strings from the Crossref metadata.
The metrics are precision and recall calculated over the citation links. I also use F1 as a standard single-number metric that combines precision and recall, weighing them equally.
The best variant of search-based matching outperforms the legacy approach in F1 (96.3% vs. 92.5%), with the precision worse by only 0.9% (98.09% vs. 98.95%), and the recall better by 8.9% (94.56% vs. 86.85%).
Common causes of SBMV’s errors are: incomplete/erroneous metadata of the target documents, and noise in the reference strings.
The results reported here generalize to the subset of references in Crossref that are deposited without the target DOI and are present in the form of unstructured strings.

Introduction

In reference matching, we try to find the DOI of the document referenced by a given input reference. The input reference can have a structured form (a collection of metadata fields) and/or an unstructured form (a string formatted in a certain citation style).

In my previous blog post, I used reference strings generated automatically to compare four matching algorithms: Crossref’s legacy approach based on reference parsing and three variations of search-based matching. The best algorithm turned out to be Search-Based Matching with Validation (SBMV). SBMV uses our REST API’s bibliographic search function to select the candidate target documents, and a separate validation-scoring procedure to choose the final target document. The legacy approach and SBMV achieved very similar average precision, and SBMV was much better in average recall.

This comparison had important limitations, which affect the interpretation of these results.

First of all, the reference strings in the dataset might be too perfect. Since they were generated automatically from the Crossref metadata records, any piece of information present in the string, such as the title or the name of the author, will exactly match the information in Crossref’s metadata. In such a case, a matcher comparing the string against the record can simply apply exact matching and everything should be fine.

In real life, however, we should expect all sorts of errors and noise in the reference strings. For example, a string might have been manually typed by a human, so it can have typos. The string might have been scraped from the PDF file, in which case it could have unusual unicode characters, ligatures or missing and extra spaces. A string can also have typical OCR errors, if it was extracted from a scan.

These problems are typical for messy real-life data, and our matching algorithms should be robust enough to handle them. However, when we evaluate and compare approaches using the perfect reference strings, the results won’t tell us how well the algorithms handle harder, noisy cases. After all, even if you repeatedly win chess games against your father, it doesn’t mean you will likely defeat Garry Kasparov (unless, of course, you are Garry Kasparov’s child, in which case, please pass on our regards to your dad!).

Even though I attempted to make the data more similar to the noisy real-life data by simulating some of the possible errors (typos, missing/extra spaces) in two styles, this might not be enough. We simply don’t know the typical distribution of the errors, or even what all the possible errors are, so our data was probably still far from the real, noisy reference strings.

The differences in the distributions are a second major issue with the previous experiment. To build the dataset, I used a random sample from Crossref metadata, so the distribution of the cited item types (journal paper, conference proceeding, book chapter, etc.) reflects the overall distribution in our collection. However, the distribution in real life might be different if, for example, journal papers are on average cited more often than conference proceedings.

Similarly, the distribution of the citation styles is most likely different. To generate the reference strings, I used 11 styles distributed uniformly, while the real distribution most likely contains more styles and is skewed.

All these issues can be summarized as: the data used in my previous experiment is different from the data our matching algorithms have to deal with in the production system. Why is this important? Because in such a case, the evaluation results do not reflect the real performance in our system, just like the child’s score on the math exam says nothing about their score on the history test. We can hope my previous results accurately showed the strengths and weaknesses of each algorithm, but the estimations could be far off.

So, can we do better? Sure!

This time, instead of automatically-generated reference strings, I will use real reference strings found in the Crossref metadata. This will give us a much better picture of the matching algorithms and their real-life performance.

Evaluation

This time the evaluation dataset is composed of 2,000 unstructured reference strings from the Crossref metadata, along with the target true DOIs. The dataset was prepared mostly manually:

First, I drew a random sample of 100,000 metadata records from the system.
Second, I iterated over all sampled items, and extracted those unstructured reference strings, that do not have the DOI provided by the member.
Next, I randomly sampled 2,000 reference strings.
Finally, I assigned a target DOI (or null) to each reference string. This was done by verifying DOIs returned by the algorithms and/or manual searching.

The metrics this time are based on the citation links. A citation link points from the reference (or the document containing the reference) to the referenced (target) document.

When we apply a matching algorithm to a set of reference strings in our collection, we get a set of citation links between our documents. I will call those citation links returned links.

On the other hand, in our collection we have real, true links between the documents. In the best-case scenario, the set of true links and the set of returned links are identical. But we don’t live in a perfect world and our matching algorithms make mistakes.

To measure how close the returned links are to the true links, I used precision, recall and F1. This time they are calculated over all citation links in the dataset. More specifically:

Precision is the fraction of the returned links that are correct. Precision answers the question: if I see a citation link A->B in the output of a matcher, how certain can I be that paper A actually cites paper B?
Recall is the percentage of true links that were returned by the algorithm. Recall answers the question: if paper A cites paper B and B is in the collection, how certain can I be that the matcher’s output contains the citation link A->B?
F1 is the harmonic mean of precision and recall.

In the previous experiment, I also used precision, recall and F1, but they were calculated for each target document and then averaged. This time precision, recall and F1 are not averaged but simply calculated over all citation links. This is a more natural approach, since now the dataset comprises isolated reference strings rather than target documents, and in practice each target document has at most one incoming reference.

I tested the same four approaches as before:

the legacy approach, based on reference parsing
SBM with a simple threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score exceeds the predefined threshold
SBM with a normalized threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score divided by the string length exceeds the predefined threshold
SBMV, which first applies SBM with a normalized threshold to select a number of candidate items, and a separate validation procedure is used to select the final target item

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in the experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Results

The plot shows the overall results of all tested approaches:

The exact values are also given in the table (the best result for each metric is bolded):

	precision	recall	F1
legacy approach	0.9895	0.8685	0.9251
SBM (simple threshold)	0.8686	0.8191	0.8431
SBM (normalized threshold)	0.7712	0.9121	0.8358
SBMV	0.9809	0.9456	0.9629

As we can see, the legacy approach is the best in precision, slightly outperforming SBMV. In recall, SBMV is clearly the best, which also decided about its victory over the legacy approach in F1.

How do these results compare to the results from my previous blog post? The overall trends (the legacy approach slightly outperforms SBMV in precision, and SBMV outperforms the legacy approach in recall and F1) are the same. The most important differences are: 1) on the real dataset SBM without validation is worse than the legacy approach, and 2) this time the algorithms achieved much higher recall. These differences are most likely related to the difference in data distributions explained before.

SBMV’s strengths and weaknesses

Let’s look at a few example cases where SBMV successfully returned the correct DOI, while the legacy approach failed.

Lundqvist D, Flykt A, Ohman A: The Karolinska Directed Emotional Faces - KDEF, CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet. 1998

matched to https://doi-org.pluma.sjfc.edu/10.1037/t27732-000

The target item is a dataset, which means unusual metadata fields and an unusual reference string.

Schminck, A. , ‘The Beginnings and Origins of the “Macedonian” Dynasty’ in J. Burke and R. Scott , eds., Byzantine Macedonia: Identity, Image and History (Melbourne, 2000), 61–8.

matched to https://doi-org.pluma.sjfc.edu/10.1163/9789004344730_006

This is an example of a book chapter. The reference string contains special quotes and dash characters.

R. Schneider,On the Aleksandrov-Fenchel inequality, inDiscrete Geometry and Convexity (J. E. Goodman, E. Lutwak, J. Malkevitch and R. Pollack, eds.), Annals of the New York Academy of Sciences440 (1985), 132–141.

matched to https://doi-org.pluma.sjfc.edu/10.1111/j.1749-6632.1985.tb14547.x

In this case, spaces are missing in the reference string, which might be problematic for the parsing.

R. B. Husar andE. M. Sparrow, Int. J. Heat Mass Transfer11, 1206 (1968).

matched to https://doi-org.pluma.sjfc.edu/10.1016/0017-9310(68)90036-7

This is another example of a reference string with missing spaces.

F. Cappello, A. Geist, W. Gropp, S. Kale, B. Kramer, and M. Snir. Toward exascale resilience: 2014 update. Supercomputing frontiers and innovations, 1(1), 2014.

matched to https://doi-org.pluma.sjfc.edu/10.14529/jsfi140101

In this case authors are missing in the Crossref metadata.

Li KZ, Shen XT, Li HJ, Zhang SY, Feng T, Zhang LL. Ablation of the Carbon/carbon Composite Nozzle-throats in a Small Solid Rocket Motor[J]. Carbon, 2011, 49: 1 208–1 215

matched to https://doi-org.pluma.sjfc.edu/10.1016/j.carbon.2010.11.037

Here we have unexpected spaces inside page numbers.

N. Kaloper, A. Lawrence and L. Sorbo, An Ignoble Approach to Large Field Inflation, JCAP 03 (2011) 023 [ arXiv:1101.0026 ] [ INSPIRE ].

matched to https://doi-org.pluma.sjfc.edu/10.1088/1475-7516/2011/03/023

In this case we have an acronym of the journal name and additional arXiv id.

KrönerE. ?Stress space and strain space continuum mechanics?, Phys. Stat. Sol. (b), 144 (1987) 39?44.

matched to https://doi-org.pluma.sjfc.edu/10.1002/pssb.2221440104

This reference string has a missing space, a missing word in the title, and incorrectly encoded special characters.

Suyemoto K. L., (1998) The functions of self-mutilationClinical Psychology Review 18(5): 531–554

matched to https://doi-org.pluma.sjfc.edu/10.1016/s0272-7358(97)00105-0

In this case the space is missing between the title and the journal name.

Ono , N. 2011 Stable and fast update rules for independent vector analysis based on auxiliary function technique Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics 189 192

matched to https://doi-org.pluma.sjfc.edu/10.1109/aspaa.2011.6082320

The parsing can also have problems with missing punctuation, like in this case.

Hybertsen M.S., Witzigmann B., Alam M.A., Smith R.K. (2002) 1 113

matched to https://doi-org.pluma.sjfc.edu/10.1023/a:1020732215449

In this case both title and journal name are missing from the reference string.

We can see from these examples that SBMV is fairly robust and able to deal with a small amount of noise in the metadata and reference strings.

What about the errors SBMV made? From the perspective of citation links, we have two types of errors:

False positives: incorrect links returned by the algorithm.
False negatives: links that should have been returned but weren’t.

When we apply SBMV instead of the legacy approach, the fraction of false positives within the returned links increases from 1.05% to 1.91%, and the fraction of false negatives within the true links decreases from 13.15% to 5.44%. This means with SBMV:

1.91% of the links in the algorithm’s output are incorrect
5.44% of the true links are not returned by the algorithm

We can also classify all the references in the dataset into several categories, based on the values of true and returned DOIs:

We have the following categories:

References matched to correct DOIs (1129 cases, returned and true blue)
References correctly not matched to anything (791 cases, returned and true white)
References not matched to anything, when they should be (58 cases, returned white, true grey)
References matched to wrong DOIs (7 cases, returned red, true yellow)
References matched to something, when they shouldn’t be matched to anything (15 cases, returned black, true white)

Note that in terms of these categories, precision is equal to:

And recall is equal to:

What are the most common causes of SBMV’s errors?

Incomplete or incorrect Crossref metadata. Even a perfect reference string formatted in the most popular citation style will not be matched, if the target record in the Crossref collection has many missing or incorrect fields.
Similarly, missing or incorrect information in the reference string is very problematic for the matchers.
Errors/noise in the reference string, such as:
- HTML/XML markup not stripped from the string
- multiple references mixed in one string
- spacing issues and typos
In a few cases a document related to the real target was matched, such as the book instead of its chapter, or the conference proceedings paper instead of the thesis.

Limitations

The most important limitation is the size of the dataset. Every item had to be verified manually, which significantly limited the possibility of creating a large set and also using a lot of independent sets.

Finally, the numbers reported here still don’t reflect the overall precision and recall of the current links in the Crossref metadata. This is because:

we still use the legacy approach for matching,
some references are deposited along with the target DOIs and are not matched by Crossref, these links are not analyzed here, and
in Crossref we have both unstructured and structured references, and in this experiment only the unstructured ones were tested.

What’s next?

The next experiment will be related to the structured references. Similarly as here, I will try to estimate the performance of the search-based matching approach and compare it to the performance of the legacy approach.

The evaluation framework, evaluation data and experiments related to the reference matching are available in the repository https://github.com/CrossRef/reference-matching-evaluation. Future experiments will be added there as well.

https://github.com/CrossRef/reference-matching-evaluation also contains the Python implementation of the SBMV algorithm. The Java implementation of SBMV is available in the repository https://gitlab.com/crossref/search_based_reference_matcher.

Data Citation: what and how for publishers

Rachael Lammey — Fri, 23 Nov 2018 00:00:00 +0000

We’ve mentioned why data citation is important to the research community. Now it’s time to roll up our sleeves and get into the ‘how’. This part is important, as citing data in a standard way helps those citations be recognised, tracked, and used in a host of different services.

This week A Data Citation Roadmap for Scientific Publishers was published in Scientific Data. This roadmap is the outcome of a collaboration between different publishers that worked on identifying all steps you need to take as a publisher to implement data citation. If you want to know more about establishing a data policy, capturing data citations at the point of submission, or tagging data citations in your XML, we recommend you take a look at this article!

In this blog post, we’ll discuss the steps you need to take after you’ve implemented this roadmap. The steps in the roadmap describe how you can track & tag data citation yourself. Here we describe how Crossref can help you make these available to the rest of the community.

The ‘what’

Here’s the recap! From the Crossref perspective, there are two ways to add data citation links into the metadata that you register:

1. Metadata deposits using the references section of the schema

This is where ‘citations’ are normally recorded. Publishers include the data citation into the deposit of bibliographic references for each publication.

Publishers can deposit the full data or software citation as a unstructured reference. For guidance here, we recommend that authors cite the dataset or software based on community best practice (Joint Declaration of Data Citation Principles, FORCE11 citation placement, FORCE11 Software Citation Principles).

<citation key="ref=3">
<unstructured_citation>Morinha F, Dávila JA, Estela B, Cabral JA, Frías Ó, González JL, Travassos P, Carvalho D, Milá B, Blanco G (2017) Data from: Extreme genetic structure in a social bird species despite high dispersal capacity. Dryad Digital Repository. http://dx.doi.org.pluma.sjfc.edu/10.5061/dryad.684v0</unstructured_citation\>
</citation>
</citation_list>

Or they can employ any number of reference tags currently accepted by Crossref.

<citation key="ref2">
<doi>10.5061/dryad.684v0</doi>
<cYear>2017</cYear>
<author>Morinha F, Dávila JA, Estela B, Cabral JA, Frías Ó, González JL, Travassos P, Carvalho D, Milá B, Blanco G</author>
</citation>

We are exploring JATS4R recommendations to expand the current collection and better support these citations - more on this soon. We also encourage additional suggestions from the community.

2. Metadata deposits using the relations section of the schema

This is where other relationships can be recorded. Publishers assert the data link in the relationship section of the metadata deposit. Here, publishers can identify data which are direct outputs of the research results if this is known. This level of specificity is optional, but we’d recommend it as it can support scientific validation and research funding management.

Data and software citations via relation type enables precise tagging of the dataset and its specific relationship to the research results published. To tag the data & software citation in the metadata deposit, we ask for the description of the dataset & software (optional), dataset & software identifier and identifier type (DOI, PMID, PMCID, PURL, ARK, Handle, UUID, ECLI, and URI), and relationship type.

<program xmlns="http://www.crossref.org.pluma.sjfc.edu/relations.xsd">
<related_item>
<description>Data from: Extreme genetic structure in a social bird species despite high dispersal capacity</description>
<inter_work_relation relationship-type="references" identifier-type="doi">10.5061/dryad.684v0</inter_work_relation>
</related_item>
</program>
</doi_relations>

In general, use the relation type references for data and software resources.

Publishers who wish to specify that the data or software resource was generated as part of the research results can use the isSupplementedBy relation type.

The ‘how’

I create my own XML and register it with Crossref

Add links to datasets into your reference lists, including their DOIs if available as shown above and deposit them with Crossref. We’ll do the rest. If you want to add references to existing metadata records, you don’t need to redeposit the full article metadata, you can send us a resource-only deposit that just contains the reference metadata to append that to the existing metadata for the article. You can also use this method if you prefer to deposit references in a separate workflow to registering your content (we know some members prefer to work this way).

I’ve started using Metadata Manager for journal article deposits

Article<->Data relationships in Crossref

You can deposit data citations using either method using our new Metadata Manager tool. When entering journal article metadata, you can use the ‘Related Items’ section to enter the DOI (or other identifier) for the dataset, the type of identifier, a description of the relation type e.g. ‘Data from: Extreme genetic structure in a social bird species despite high dispersal capacity’, and the relation type - ‘references’ or ‘is supplemented by’ depending on the relationship between the data and the article as described above. When you make the deposit, this relationship information will be registered in Crossref along with the rest of the article metadata.

Metadata Manager also has a section where you can enter and match your references, and then deposit these with Crossref. If you choose this method, enter any data citations into the references section before depositing the article metadata with Crossref.

If you want to add this information to deposits you have already made using Metadata Manager, you can search for the journals and articles in the interface, bring up the existing metadata and add in the additional information before redepositing.

I use “simple text query” to search for and deposit references

Make sure you include any citations to data in the references you add into Simple Text Query. When you use simple text query to deposit these references, they will then be added into the article metadata in the Crossref database.

If you use OJS, they’re working on functionality (due for release soon) that will make it easier to deposit reference metadata with Crossref, so you can include citations to data in that.

All of this metadata—registered with Crossref—make it possible to build up pictures of data citations, linking, and relationships. Whether the citations come from the authors in the reference list or they are extracted by the publisher and then deposited, Crossref collects them across publishers. We then make the aggregate set freely available via Crossref’s APIs in multiple interfaces (REST, OAI-PMH, OpenURL) and formats (XML and JSON). DataCite does the same for data repositories and so this provides an easy way for publishers and data repositories to exchange information about data citations. As mentioned previously, this all feeds in Event Data. Data is made openly available to a wide host of parties across the extended research ecosystem including funders, research organisations, technology and service providers, indexers, research data frameworks such as Scholix, etc.

Do you have questions about how to add these links to your Crossref or DataCite metadata? We’ll be running a series of webinars in early 2019 to give you a chance to join us live and ask any questions you have. Eager to get started in the meantime? Let us know and we’ll start to coordinate.

Matchmaker, matchmaker, make me a match

Dominika Tkaczyk — Mon, 12 Nov 2018 00:00:00 +0000

Matching (or resolving) bibliographic references to target records in the collection is a crucial algorithm in the Crossref ecosystem. Automatic reference matching lets us discover citation relations in large document collections, calculate citation counts, H-indexes, impact factors, etc. At Crossref, we currently use a matching approach based on reference string parsing. Some time ago we realized there is a much simpler approach. And now it is finally battle time: which of the two approaches is better?

TL;DR

I evaluated and compared four approaches to reference matching: the legacy approach based on reference parsing, and three variants of the new idea called search-based matching.
A large automatically generated dataset was used for the experiments. It is composed of 7,374 metadata records from the Crossref collection, each of which was formatted automatically into reference strings using 11 citation styles.
The main metrics used for the evaluation are precision and recall. I also use F1 as a standard metric that combines precision and recall into a single number, weighing them equally. All values are calculated for each metadata record separately and averaged over the dataset.
In general, search-based matching is better than the legacy approach in F1 and recall, but worse in precision.
The best variant of search-based matching outperforms the legacy approach in average F1 (84.5% vs. 52.9%), with the average precision worse by only 0.1% (99.2% vs 99.3%), and the average recall better by 88% (79.0% vs. 42.0%).
The best variant of search-based matching also outperforms the legacy approach in average F1 for each one of the 11 styles.
A weak spot of the parsing-based approach is degraded/noisy reference strings, which do not appear to use any of the known citation styles.
A weak spot of search-based approach is short reference strings, and in particular citation styles that do not include the title in the reference string.

Introduction

In reference matching, on the input we have a bibliographic reference. It can have the form of an unstructured string, such as:

(1) Adamo, S. H.; Cain, M. S.; Mitroff, S. R. Psychological Science 2013, 24, 2569–2574.

The input can also have the form of a structured reference, such as (BibTex format):

 @article{adamo2013,
 author = {Stephen H. Adamo and Matthew S. Cain and Stephen R. Mitroff},
 title = {Self-Induced Attentional Blink: A Cause of Errors in Multiple-Target Search},
 journal = {Psychological Science},
 volume = {24},
 number = {12},
 pages = {2569-2574},
 year = {2013}
}

The goal of matching is to find the document, which the input reference points to.

Matching algorithms

Matching references is not a trivial task even for a human, not to mention the machines, which are still a bit less intelligent than us (or so they want us to believe…). A typical meta-approach to reference matching might be to score the similarity between the input reference and the candidate target documents. The document most similar to the input is then returned as the target.

Of course, still a lot can go wrong here. We can have more than one potential target record with the same score (which one do we choose?). We can have only documents with low to medium scores (is the actual target even present in our collection?). We can also have errors in the input string (are the similarity scores robust enough?). Life’s tough!

The main difference between various matching algorithms is in fact how the similarity is calculated. For example, one idea might be to compare the records field by field (how similar is the title/author/journal in the input reference to the title/author/journal of our candidate target record?). This is roughly how the matching works currently at Crossref.

The main problem with this approach is that it requires a structured reference, and in practise, often all we have is a plain reference string. In such a case we need to extract the metadata fields from the reference string (this is called parsing). Parsing introduces errors, since no parser is omniscient. The errors propagate further and affect the scoring… you get the picture.

Luckily, as we have known for some time now, this is not the only approach. Instead of comparing structured objects, we could calculate the similarity between them using their unstructured textual form. This effectively eliminates the need for parsing, since the unstructured form is either already available on the input or can be easily generated from the structured form.

What about the similarity scores? We already know a powerful method for scoring the similarities between texts. Those are (you guessed it!) scoring algorithms used by search engines. Most of them, including Crossref’s, do not need a structured representation of the object, they are perfectly happy with just a plain text query.

So all we need to do is to pass the original reference string (or some concatenation of the reference fields, if only a structured reference is available) to the search engine and let it score the similarity for us. It will also conveniently sort the results so that it is easy to find the top hit.

Evaluation

So far so good. But which strategy is better? Is it better to develop an accurate parser, or just rely on the search engine? I don’t feel like guessing. Let’s try to answer this using (data) science. But first, we need to decompose our question into smaller pieces.

Question 1. How can I measure the quality of a reference matcher?

Generally speaking, this can be done by checking the resulting citation links. Simply put, the better the links, the better the matching approach must have been.

A few standard metrics can be applied here, including accuracy, precision, recall and F1. We decided to calculate precision, recall and F1 separately for each document in the dataset, and then average those numbers over the entire dataset.

When I say “documents”, I really mean “target documents”:

precision for a document X tells us, what percentage of links to X in the system are correct
recall for a document X tells us, what percentage of true links to X are present in the system
F1 is the harmonic mean of precision and recall

F1 is a single-number metric combining precision and recall. In F1 precision and recall are weighted equally. It is also possible to combine precision and recall using different weights, to place more emphasis on one of those metrics.

We decided to look at links from the target document’s perspective, because this is what the academic world cares about (i.e. how accurate the citation counts of academic papers are).

Calculating separate numbers for individual documents and averaging them within a dataset is the best way to have reliable confidence intervals (which makes the whole analysis look much smarter!).

Question 2. Which approaches should be compared?

In total we tested four reference matching approaches.

The first approach, called the legacy approach, is the approach currently used in Crossref ecosystem. It uses a parser and matches the extracted metadata fields against the records in the collection.

The second approach is the search-based matching (SBM) with a simple threshold. It queries the search engine using the reference string and returns the top hit from the results, if its relevance score exceeds the threshold.

The third approach is the search-based matching (SBM) with a normalized threshold. Similarly as in the simplest SBM, in this approach we query the search engine using the reference string. In this case the first hit is returned if its normalized score (the score divided by the reference length) exceeds the threshold.

Finally, the fourth approach is a variation of the search based matching, called search-based matching with validation (SBMV). In this algorithm we use additional validation procedure on top of SBM. First, SBM with a normalized threshold is applied and the search results with the scores exceeding the normalized threshold are selected as candidate target documents. Second, we calculate validation similarity between the input string and each of the candidates. This validation similarity is based on the presence of the candidate record’s metadata fields (year, volume, issue, pages, the last name of the first author, etc.) in the input reference string, as well as the relevance score returned by the search engine. Finally, the most similar candidate is returned as the final target document, if its validation similarity exceeds the validation threshold.

By adding the validation stage to the search-based matching we make sure that the same bibliographic numbers (year, volume, etc.) are present in both the input reference and the returned document. We also don’t simply take the first result, but rather use this validation similarity to choose from results scored similarly by the search engine.

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in these experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Question 3. How to create the dataset?

Results

We could try to calculate our metrics for every single document in the system. Since we currently have over 100M of them, this would take a while, and we already felt impatient…

A faster strategy was to use sampling with all the tools statistics was so generous to provide. And this is exactly what we did. We used a random sample of 2500 items from our system, which is big enough to give reliable results and, as we will see later, produces quite narrow confidence intervals.

Apart from the sample, we needed some input reference strings. We generated those automatically by formatting the metadata of the chosen items using various citation styles. (Similarly to what happens when you automatically format the bibliography section for your article. Or at least we hope you don’t produce those reference strings manually…)

For each record in our sample, we generated 11 citation strings, using the following styles:

Well known citation styles from various disciplines:
- american-chemical-society (acs)
- american-institute-of-physics (aip)
- elsevier-without-titles (ewt)
- apa
- chicago-author-date
- modern-language-association (mla)
Known styles + random noise. To simulate not-so-clean data, we randomly added noise (additional spaces, deleted spaces, typos) to the generated strings of the following styles:
- american-institute-of-physics
- apa
Custom degraded “styles”:
- degraded: a simple concatenation of authors’ names, title, container title, year, volume, issue and pages
- one author: a simple concatenation of the first author’s name, title, container title, year, volume, issue and pages
- title scrambled: same as degraded, but with title words randomly shuffled

Some styles include the DOI in the reference string. In such cases we stripped the DOI from the string, to make the matching problem non-trivial.

An ideal matching algorithm will match every generated string to the record it was generated from. In practise, some of the expected matches will be missing, which will lower the recall of the tested matching approach. On the other hand, it is very probable that we will get the precision of 100%. To have the precision lower than 100%, we would have to have some unexpected matches to our sampled documents, which is unlikely. This is obviously not great, because we are missing a very important piece of information.

What can we do to “encourage” such mismatches to our sampled documents? We could generate additional reference strings of documents that are not in our sample, but are similar to the documents in our sample. Hopefully, we will see some incorrect links from those similar strings to our sampled documents.

For each sampled document I added up to 2 similar documents (I used, surprise surprise, our search engine to find the most similar documents). I ended up with 7,374 items in total (2,500 originally sampled and 4,874 similar items). For each item, 11 different reference strings were generated. Each reference string was then matched using the tested approaches and I could finally look at some results.

Results

First, let’s compare the overall results averaged over the entire dataset:

The small vertical black lines at the top of the boxes show the confidence intervals at the confidence level 95%. The table gives the exact values and the same confidence intervals. The best result for each metric is bolded.

	average precision	average recall	average F1
legacy approach	0.9933 (0.9910 - 0.9956)	0.4203 (0.4095 - 0.4312)	0.5289 (0.5164 - 0.5413)
SBM (simple threshold)	0.9890 (0.9863 - 0.9917)	0.7127 (0.7021 - 0.7233)	0.7866 (0.7763 - 0.7968)
SBM (normalized threshold)	0.9872 (0.9844 - 0.9901)	0.7905 (0.7796 - 0.8015)	0.8354 (0.8249 - 0.8458)
SBMV	0.9923 (0.9902 - 0.9945)	0.7902 (0.7802 - 0.8002)	0.8448 (0.8352 - 0.8544)

The confidence intervals given in the table are the ranges, in which it is 95% likely to have the real average precision, recall and F1. For example, we are 95% sure that the real F1 for SBMV in our entire collection is within the range 0.8352 - 0.8544.

As we can see, each metric has a different winner.

The legacy approach is the best in precision. This suggests the legacy approach is quite conservative and outputs a match only if it is very sure about it. This might also result in missing a number of true matches (false negatives).

According to the paired Student’s t-test, the difference between the average precision of the legacy approach and the average precision of the second best SBMV is not statistically significant. This means we cannot rule out that this difference is simply the effect of the randomness in sampling, and not the sign of the true difference.

SBM with a normalized threshold is the best in recall. This suggests that it is fairly tolerant and returns a lot of matches, which might also result in returning more incorrect matches (false positives). Also in this case the difference between the winner and the second best (SBMV) is not statistically significant.

SBMV is the best in F1. This shows that this approach balances precision and recall the best, despite being only the second best in both of those metrics. According to the paired Student’s t-test, the difference between SBMV and the second best approach (SBM with a normalized threshold) is statistically significant.

All variants of the search-based matching outperform the parsing-based approach in terms of F1, with statistically significant differences. This shows that in search based-matching it is possible to keep precision almost as good as in the legacy approach, and still include many more true positives.

Let’s also look at the same results split by the citation style:

For all styles the precision values are very high, and the legacy approach is slightly better than all variations of the search-based approach.

In terms of recall and F1 SBM with a simple threshold is better than the legacy approach in 8 out of 11 styles. The three styles for which the legacy approach outperforms SBM with a simple threshold are styles that do not include the title in the reference strings (acs, aip and ewt). The reason for this is that the simple threshold cannot be well calibrated for shorter and longer reference strings at the same time.

SBM with a normalized threshold and SBMV is better than the legacy approach in recall and F1 for all 11 styles.

The weak spot of the legacy approach is degraded and noisy reference strings, which do not appear to use any of the known citation styles.

The weak spot of the search-based matching is short reference strings, and in particular citation styles that do not include the title in the string.

Limitations

The limitations are related mostly to the method of building the dataset.

All the numbers reported here are estimates, since they were calculated on a sample.
The numbers show strengths and weaknesses of each approach, but they do not reflect the real precision and recall in the system:
- Since we included only 2 similar documents for each document in the sample, precision is most likely lower in the real data.
- We used a number of styles distributed uniformly. Of course in the real system the styles and their distribution might be different, which affects all the calculated numbers.

What does the sample say?

Dominika Tkaczyk — Fri, 09 Nov 2018 00:00:00 +0000

At Crossref Labs, we often come across interesting research questions and try to answer them by analyzing our data. Depending on the nature of the experiment, processing over 100M records might be time-consuming or even impossible. In those dark moments we turn to sampling and statistical tools. But what can we infer from only a sample of the data?

Imagine you are cooking soup. You just put some salt in it and now you are wondering if it is salty enough. What do you do next?

Option #1: Since you carefully measured 1/7 of a teaspoon of salt per 0.13 litres of soup (as always), you already know the soup is fine. Everyone else better stop asking silly questions and eat their soup.
Option #2: You stir everything carefully and taste a tablespoon. If it is not salty enough, you put more salt in the soup and repeat the tasting procedure.
Option #3: You eat a tablespoon of soup and it tastes fine. But wait, there’s more soup in the pot, what if the sip you’ve just tasted was somehow different than the rest? You decide it’s better to eat another spoon of soup (which tastes fine). Still, a lot of soup left, who knows what that tastes like? It might be safer to eat an entire bowl of soup. Hmm, still not sure, you’ve eaten such a small fraction of the soup, who can guarantee the rest tastes the same? You have no choice but to eat another bowl, and then some more… Ooops, now you have eaten the entire pot of soup! At least you can be 100% sure now that the soup was indeed salty enough. The problem is, there is no soup left, and also, you don’t feel so good. But people are getting hungry, so you start cooking a new batch…

If your answer was option #3, read on. Your life is going to get easier!

TL;DR

Sampling and confidence intervals can be used to estimate the mean of a certain feature, or the proportion of items passing a certain test, by calculating it only for a random sample of items, instead of the entire large set of items. Note that estimating =/= guessing.
Confidence intervals are a way of controlling the amount of uncertainty related to randomness in sampling.
The confidence interval has a form (estimated value - something, estimated value + something). Confidence interval at the confidence level 95% is interpreted as follows: we are 95% sure that the real value that we are estimating is within our calculated confidence interval.
The higher the confidence level (i.e. the more certain we want to be about the interval), the wider the interval has to be.
The larger the sample, the narrower the confidence interval.
We are never 100% sure that the value we are estimating is actually within our calculated confidence interval. By setting the confidence level high, we only make sure this is a very likely event.

The problem

Sampling and estimating drew my attention while I was working on the evaluation of the reference matching algorithms. In Crossref’s case, reference matching is the task of finding the target document DOI for the given input reference string, such as:

(1) Adamo, S. H.; Cain, M. S.; Mitroff, S. R. Psychological Science 2013, 24, 2569–2574.

Accurate reference matching is very important for the scientific community. Thanks to automatic reference matching we are able to find citing relations in large document sets, calculate citation counts, H-indexes, impact factors, etc.

For several weeks now I have been investigating a simple reference matching algorithm based on the search engine. In this algorithm, we use the input reference string as the query in the search engine, and we return the first item from the results as the target document. Luckily, at Crossref we already have a good search engine in place, so all the pieces are there.

I was interested in how well this simple algorithm works, i.e. how often the correct target document is found. For example, let’s say we have a reference string in APA citation style generated for a specific record in Crossref system. How certain can I be that it will be correctly matched to the record’s DOI?

I could calculate this directly by generating the APA reference string for every record in the system and trying to match those strings to DOIs. Since we already have over 100M records, this would take a while and I was getting impatient. So instead of eating the whole pot of soup, I decided to stir and taste just a little bit of it, or, academically speaking, use sampling and confidence intervals.

These statistical tools are useful in situations, where we have a large set of items, and we want to know the average of a certain feature of an item in our set, or the proportion of items passing a certain test, but calculating it directly is impossible or difficult. For example, we might want to know the average height of all women living in USA, the average salary of a Java programmer in London, or the proportion of book records in the Crossref collection. The entire set we are interested in is called a population and the value we are interested in is called a population average or a population proportion. Sampling and confidence intervals let us estimate the population average or proportion using only a sample of items, in a reliable and controlled way.

Experiments

In general I wanted to see, how well I can estimate the population proportion of records passing a certain test, using only a sample.

In the following experiments, the population is 1 million metadata records from the Crossref collection. I didn’t use the entire collection as the population, because I wanted to be able to calculate the real proportion and compare it to the estimates.

The test for a single record is: whether the APA reference string generated from said record is correctly matched to the record’s original DOI. In other words: if I generate the APA reference string from my record and use it as the query in Crossref’s search, will the record be the first element in the result list? Note that this proportion can also be interpreted as the probability that the APA reference string will be correctly matched to the target DOI.

Estimating from a sample

I took a random sample of size 100 from my population and calculated the proportion of the records correctly matched - this is called a sample proportion. In my case, the sample proportion is 0.92. This means that in my sample 92 reference strings were successfully matched to the right DOIs. Not too bad.

I could now treat this number as the estimate and assume that 0.92 is close to the population proportion. On the other hand, this is only a sample, and a rather small one, which raises doubts. What if our 92 correct matches happen to be the only correct matches in the entire 1M population? In such a case, our estimate of 0.92 would be very far from the population proportion. This uncertainty related to sampling randomness can be captured by the confidence interval.

Confidence interval

The confidence interval for my 100-point sample, at the confidence level 95%, is 0.8668-0.9732. This is interpreted as follows: we are 95% sure that the real population proportion is within the range 0.8668-0.9732. Note that the sample average (0.92) is exactly in the middle of this range.

100 items is not a big sample. Let’s calculate the confidence interval for a sample 10 times larger. From a sample of size 1000 I got the estimate 0.932, and the confidence interval 0.9164-0.9476. Based on this, we can be 95% sure that the real population proportion is within the range 0.9164-0.9476.

It seems the our interval got smaller when we increased the sample size. Let’s plot the intervals for a variety of sample sizes:

The blue line represents the estimated proportion for samples of different sizes, and the grey vertical lines are confidence intervals. The estimated proportion varies, because for each size a different sample was drawn.

We can see that increasing the sample size decreases the interval. This should make intuitive sense: if we have more data to estimate from, we can expect our estimate to be more reliable (i.e. closer to the population proportion).

What about the confidence level? By setting the confidence level we specify, how certain we want to be about our confidence interval. So far I used 95%. What happens if I calculate the confidence intervals for my original sample of 100 records, but with varying confidence level?

In this case the average is always the same, because only one sample was used.

As we can see, increasing the confidence level widens the interval. In other words, the more certain we want to be about the interval containing the real population average, the wider the interval has to be.

Sampling distribution

So far so good, but where does this magic confidence interval actually come from? It is calculated by the theoretical analysis of the sampling distribution (not to be confused with sample distribution):

Sample distribution is when we collect one sample of size k and calculate a certain feature for every element in the sample. It is a distribution of k values of the feature in one sample.
Sampling distribution is when we independently collect n samples, each of size k, and calculate the sample proportion for each sample. It is the distribution of n sample proportions.

Imagine I collect all samples of size 100 from my population and I calculate the sample proportion for each sample. This is the sampling distribution. Now I randomly choose one number from this sampling distribution. Note that this is equivalent to what I did before: choosing one random sample of size 100 and calculating its sample proportion.

According to Central Limit Theorem, sampling distribution is approximately normal with the mean equal to the population proportion. Here is the visualisation of the sampling distribution:

The black vertical line shows the mean of the sampling distribution. This is also the real population proportion. The grey area covers the middle 95% of the distribution mass (within 2 standard deviations from the mean).

When we choose one sample and calculate the sample proportion, there are two possibilities:

With 95% probability, we were lucky and the sample proportion is within the grey area. In that case, the real population proportion is not further than 2 standard deviations from our estimate.
With 5% probability, we were unlucky and the sample proportion is outside the grey area. In that case, the real population proportion is further than 2 standard deviations from our estimate.

So with the confidence of 95% we can say that the real population proportion is within 2 standard deviations from our sample proportion. We can see now that these 2 standard deviations of the sampling distribution define our confidence interval at the confidence level of 95%.

Smaller confidence level would make the grey area narrower, and the confidence interval would shrink as well. Larger confidence level makes the grey area, and the confidence interval, larger.

To look more closely at the sampling distribution, I generated sampling distributions for all combinations of “n samples of size k”, where n and k are the elements of the set {25, 50, 100, 200, 400, 800, 1600, 3200}. This is only an approximation, since the real sampling distributions would contain many more samples.

Here is the heatmap showing the mean of each sampling distribution (this should be approximately the same as the real population proportion):

We can see that there is some variability in the top left part of the heatmap, which corresponds to small sample sizes and small numbers of samples. The bottom right part of the heatmap shows much less variability. As we increase the sample size and number of samples, the mean of the sampling distribution approaches numbers around 0.933.

Here is the heatmap showing the standard deviation for each sampling distribution:

We can clearly see how the standard deviation decreases when we increase the sample size. This is consistent with the previous observation, that the confidence interval decreases when the sample size is increased.

Let’s also see the histograms of all the sampling distributions:

Here we can see the following patterns:

All histograms indeed seem to be centered around approximately the same number.
The more samples we include, the more normal the sampling distribution appears. This happens because with more samples the real sampling distribution is better approximated.
The larger the sample size, the narrower the sampling distribution (i.e. smaller standard deviation).

The estimation vs. the real value

Let’s go back to my original question. What is the proportion of reference strings in APA style, that are successfully matched to the original DOIs of the records they were generated from? So far we observed the following:

A small sample of 100 gave the estimate 0.92 (confidence interval 0.8668-0.9732)
A larger samples of 1000 gave the estimate 0.932 (confidence interval 0.9164-0.9476)
The means of sampling distributions seem to slowly approach 0.933

So what is the real population proportion in my case? It is 0.933005. As we can see, the estimations were fairly close, and the intervals indeed contain the real value.

Now I can also calculate the confidence interval for each sample in my sampling distributions, and then the fraction of the intervals that contain the real population proportion (I expect these numbers to be close to the confidence level 95%). Here is the heatmap:

We can see that for larger sample sizes indeed the fractions are high. The fraction is not always above 95%, as we would expect, especially for smaller sample sizes. One of the reasons is that when we calculate the confidence interval, we approximate the standard deviation of the population with the standard deviation of the sample. This is not always a reliable estimate, especially for small samples. This suggests that sample sizes of at least 1000-2000 should be used.

Be careful

Some important things to remember:

Aggregate functions. As mentioned before, apart from estimating the proportion, a similar procedure can be applied for estimating the average of a certain numeric feature.
(Lack of) certainty. Remember that the confidence level < 1. This means that we are never sure that our confidence interval contains the true population proportion. If for any reason you need to be 100% sure, just process the entire dataset.
Randomness, a.k.a. “stirring before tasting”. The sample has to be chosen randomly. Beware of assuming that the dataset is shuffled and taking the first 1000 rows!
Sample size. We know already that the larger the sample, the better. As a rule of thumb, using sample sizes < 30 makes the estimates, including the interval, rather unreliable.
Skewness. In general, the more skewed the original feature distribution, the larger sample we need. In case of the proportion, the sample should contain at least 5 data points of each value of the feature (passes/doesn’t pass the test).
Generalization. The sample average/proportion can be used as an estimate for the population average/proportion, but only the population it was drawn from. This means that if we applied any filters before sampling (which is equivalent to sampling from a subset passing the filter), we can reason only about the filtered subset of the data.
Reproducibility. This is more of an engineering concern. In short, all the analyses we do should be reproducible. In the context of sampling it means, at the very least, that we should record the samples we use.

Why Data Citation matters to publishers and data repositories

Helena Cousijn — Thu, 08 Nov 2018 00:00:00 +0000

A couple of weeks ago we shared with you that data citation is here, and that you can start doing data citation today. But why would you want to? There are always so many priorities, why should this be at the top of the list?

I’m sure you heard this before—data sharing and data citation are important for scientific progress. The three key reasons for this are:

1) Transparency and reproducibility

Most scientific results that are shared today are just a summary of what researchers did and found. The underlying data are not available, making it difficult to verify and replicate results. If data would always be made available with publications, transparency of research would be greatly improved.

2) Reuse

The availability of raw data allows other researchers to reuse the data. Not just for replication purposes, but to answer new research questions.

3) Credit

When researchers cite the data they used, this forms the basis for a data credit system. Right now researchers are not really incentivized to share their data, because nobody is looking at data metrics and measuring their impact. Data citation is a first step towards changing that.

The benefits described above are all quite long-term, so why, as a publisher or data repository, should you put your resources towards implementing data citation workflows now? During our pre-conference workshop at FORCE2018 we asked repositories and publishers this question. Below you’ll find some of the answers.

Data repositories

For data repositories, data citation leads to increased visibility of both the repository and the datasets. The workshop revealed that many repositories do a lot of work to establish links between articles and datasets, thereby significantly contributing to transparency in research. Some of the repositories explained that they hire curators that text mine articles to find associations and manually curate datasets to ensure information about links is part of the metadata. This is reflected in Event Data, where 99% of links between articles and datasets comes from data repository metadata. This downstream enrichment of metadata is useful, but it would be more effective if all stakeholders strive to establish these links at a much earlier stage in the research communication process.

ICPSR, the Inter-university Consortium for Political and Social Research, shared:

ICPSR views data citation as vital. As a large social science data archive, ICPSR curates, preserves, and distributes data for the research community to re-use over time. Data citation makes data visible to the research community. Without it, data cannot be accessed for re-use or reproduced for transparency. Its use cannot be tracked and counted to reveal its impact and potential for new uses by investigators in new fields or in combination with new types of data. Data creators cannot receive adequate credit for their intellectual output. And the original investment by funders and scientists to create those data stops producing dividends. Therefore, data citation plays an essential role in the data sharing lifecycle.

Proper data citation, with a unique identifier, makes it much easier to measure impact. When data use is not cited or cited obliquely, it is rendered virtually invisible. Hence, much data use is still not easily detected. The ICPSR Bibliography of Data-related Literature represents ICPSR’s efforts to identify publications that analyze data distributed at ICPSR and link them directly to the data in the ICPSR catalog. As of 2018, ICPSR has a searchable database that contains nearly 80,000 citations of published and unpublished works resulting from analyses of data held in the archive. ICPSR also makes the case for data citation in its brief new video, “ICPSR 101: Why Should I Cite Data?”

GBIF, the Global Biodiversity Information Facility, explained:

The work required to collect, clean, compile and publish biodiversity datasets is significant and deserves recognition. Researchers publish studies based on data made available through GBIF.org at a rate of about 2 papers every single day. It is crucial for GBIF to link these scientific uses to the underlying data as one measure of demonstrating the value and impact of sharing free and open biodiversity data. At the moment, however, only about 10 percent of authors cite or acknowledge the datasets used in research papers properly. As a result, data publishers efforts often risk going unnoticed, and the true impact of sharing data remains invisible. GBIF will continue to work with publishers and researchers to provide guidance and input for how to best cite the use of GBIF-mediated data in scientific journals to ensure proper attribution and reproducible research and to demonstrate the true value of free and open access to biodiversity data.

Publishers

By ensuring data is cited in a consistent way, publishers help provide transparency and context for the content they publish. Depositing that information as part of the Crossref metadata helps that work go further by uncovering how data is being used across multiple publications and publishers This means patterns can be explored and researchers can gain more comprehensive recognition and credit for the work they have done.

Melissa Harrison, Head of Production Operations at eLife says:

eLife is committed to ensuring researchers get credit for all their outputs, and data is a major component of this. We’re working with Crossref and JATS4R to enable publishers to tag their JATS data content consistently and thus create an easy crosswalk to their Crossref deposits. The JATS4R guidance on Data Availability Statements, linked to and incorporating data citations, will be updated soon, please watch that space!

It will be really interesting to see how much re-use of previously published data is happening, look for patterns in re-use, and see links and hopefully building up of data by different research groups. Ultimately, this will incentivize researchers and publishers to ensure it is correctly accredited at source and in publications, improving the cycle further.’

Anita de Waard, VP of Research Collaborations at Elsevier, says:

One of the key recommendations of the Force11 Manifesto was to “3.3 Add data, software, and workflows into the publication as first-class research objects”, which will allow greater reproducibility and rigor to experimental research, and allow the reuse of all digital artefacts in the scholarly lifecycle. By following the data citation principles, we achieve two things: the author presents a richer representation of their work, and the data producer receives credit for the hard work of curating and publishing citable datasets.

Mendeley Data and Elsevier are active contributors to the Scholix framework that as a collaborative and open standard, allows the open mining of relationships between articles and datasets. We are also active participants in the new Enabling FAIR Data Project, and next to supporting the TOP Guidelines in all domains, require all authors in the earth and space sciences to deposit their data before publication.

Next week at Crossref LIVE18, Patricia Cruse from DataCite will talk about Data Citations and why they matter. If you’re in Toronto next week, do not hesitate to ask her or anyone from Crossref anything you want to know about data citation!

Metadata Manager: Members, represent!

Jennifer Lin — Mon, 15 Oct 2018 00:00:00 +0000

Over 100 Million unique scholarly works are distributed into systems across the research enterprise 24/7 via our APIs at a rate of around 633 Million queries a month. Crossref is broadcasting descriptions of these works (metadata) to all corners of the digital universe.

Whether you’re a publisher, institution, governmental agency, data repository, standards body, etc.: when you register and update your metadata with Crossref, you’re relaying it to the entire research enterprise. So make sure your publications are fully and accurately represented.

Metadata Manager is here to help

This year, we’ve released a new tool aimed to make this easier and give you, members, full control over your metadata. Presenting: Metadata Manager. It helps to:

Simplify and streamline the Content Registration service, with a user-friendly interface
Give you greater flexibility and control of metadata deposits
Support users who are less familiar with XML
Boost metadata quality, encourage cleaner and more complete metadata records

Metadata Manager is available to all our members and the service providers they work with, providing assistance with a wide range of metadata-related tasks:

Regular Content Registration conducted by journal staff, editors and service providers
Registering corrections, retractions, or other editorial expressions of concern
Matching references to their DOIs and registering them with the publication
Adding metadata to existing records such as license and funding information, abstracts, or data citations
Late-arriving editorial updates/corrections after initial publication
Unexpected corrections to production hiccups
Emergency editorial changes that affect publication record
Accelerated registration for special pieces published outside of regular workflow
Securely and efficiently transfer titles to another publisher as the authorized owner

Issues arise all the time in the dynamic and challenging work of scholarly communications. Metadata Manager provides a fast and easy way to meet these head-on when broadcasting new content or updating existing content. Submissions through this tool are processed immediately upon submission (i.e., no queues!).

This new tool empowers our members to “represent” in the exhilarating thrum of data reaching our API users. At this moment in time, it only supports journals, but our development team is currently working hard to include the remaining record types.

Features

Here’s a smattering of highlights from the Metadata Manager feature list:

All metadata: easily adds any and all metadata, allowing publishers to add richness and depth to their records.
Prevents rejected submissions: it ensures you have satisfied all the basic Content Registration requirements and points out any input errors.
Expedited deposit: the Content Registration system processes each submission immediately, bypassing the deposit queue.
Historic log: easy to read archive of all previous submissions.
Effortless review: provides a clean, condensed view of metadata (invariably complicated and lengthy) to support human review of the content before submission.
Aids members to follow best practices: checks for completeness and reminds users of the full breadth of metadata available for the article, volume/issue, and the journal itself.
Full control over title transfers: no need to make these requests through our support channels. Complete the transfer at your convenience, directly through the system.

For those of you that have looked at your own metadata contribution with the use of our new Participation Reports, you’ll find using Metadata Manager a quick and useful way to help you level-up your records.

Members, represent!

We invite you to register and update your publications with Metadata Manager, relay the metadata fully and accurately to the entire research enterprise. Check out the comprehensive help documentation to find out how to set up your workspace and get started right away with your usual Content Registration login details.

As mentioned, we are continuing development, adding support for all remaining record types as well as enhancing existing features. The webDeposit form will remain available throughout this time. For journal publishers, give us a whirl and let us know if you see something missing or there’s a function that would improve your Content Registration experience!

Data citation: let’s do this

Rachael Lammey — Thu, 04 Oct 2018 00:00:00 +0000

Data citation is seen as one of the most important ways to establish data as a first-class scientific output. At Crossref and DataCite we are seeing growth in journal articles and other record types citing data, and datasets making the link the other way. Our organisations are committed to working together to help realize the data citation community’s ambition, so we’re embarking on a dedicated effort to get things moving.

Efforts regarding data citation are not a new thing. One of the first large-scale initiatives to establish data citation as a standard academic practice was the FORCE11 Joint Declaration of Data Citation Principles (JDDCP) in 2014. This declaration was endorsed by over 100 organisations in the scholarly community as well as many individuals.

Following this agreement on how data citation should be done, many projects followed. Within FORCE11, the Data Citation Implementation Pilot brought together publishers and repositories to put data citation into practice and work on the implementation of the JDDCP. Within the context of the Research Data Alliance, a data-literature linking group started under the name of Scholix to establish a framework for exchanging information about the relationships between articles and datasets. The infrastructure building blocks now feed into projects such as Make Data Count and Enabling FAIR Data.

Projects aside, if datasets are cited consistently and in a standard way, it will make it much easier for the research community to see links between different research outputs and work with these outputs. It also makes it much easier to count these citations, so that researchers can get credit for their data and the sharing of that data.

The underlying work has been done to create an infrastructure that will effectively support and disseminate information on data citation. Data citation is here today!

Different organisations know how to handle data citations, and are starting to count these and make that information available in turn. This means that the only thing that’s needed is for people to actually cite data, and this information be captured and passed on. Some Crossref and DataCite members have already made great progress on this already (see Melissa Harrison’s blog on what eLife is doing).

The goals of all the data citation projects can only be realized if you start doing data citation, and we know you’ll have questions about it…

In the coming months, we’ll be posting several blogs and organizing sessions to tell you how you can start doing data citation - if you’re attending FORCE2018 you can catch our joint workshop there. So stay tuned and please get in touch if you can’t wait, we’d love to help you get started!

Event Data is production ready

Christine Buske — Wed, 12 Sep 2018 00:00:00 +0000

We’ve been working on Event Data for some time now, and in the spirit of openness, much of that story has already been shared with the community. In fact, when I recently joined as Crossref’s Product Manager for Event Data, I jumped onto an already fast moving train—headed for a bright horizon.

What’s on the horizon? Well, the reality is you never really reach the horizon. Good product development—in my opinion—is like that train. You keep aiming for the horizon and passing all the stations (milestones) along the way, but the horizon keeps moving as you add features, improve the service, and maybe even review where you are headed. However, for Event Data we are pleased to say we have now arrived at a rather important station.

Technical readiness

Thank you to all the beta testers who have journeyed with us this far—we’ve listened and learned, refined and rebuilt with the help of your feedback. We are now thrilled to say that we are service production ready. We’ve reached the station called ‘technical readiness’, and are eager to see more users board our train!

During this time of building and refining, Event Data has grown to include at least 66,7 million events from sources like (in order of magnitude): Wikipedia, Cambia Lens, Twitter, Datacite, F1000, Newfeeds, Reddit links, Wordpress.com, Crossref, Reddit, Hypothesis, and Stackexchange. Wikipedia alone accounts for 50 million events (and counting).

What does this mean?

Event Data is production ready.

Being production ready means we are not going to make any breaking changes to the code, and we are excited to see more people jump on board to explore where you can go with Event Data, and what product or service you might want to build with it.

Getting started

Having a look at Event Data, and using it, is easy. While the user guide outlines everything you need to know to get fully engrossed, you can get your feet wet with a few sample queries:

Above I mentioned Event Data has about 50 million Wikipedia events, you can check if that has grown by looking at a query that lists all distinct events by source (your browser will need a JSON viewer extension):

https://api-eventdata-crossref-org.pluma.sjfc.edu/v1/events/distinct?facet=source/:*&rows=0

You can also see a live stream of events going through Event Data.

For all events registered for a specific content item, you simply query http://api.eventdata.crossref.org.pluma.sjfc.edu/v1/events?obj-id=https://doi-org.pluma.sjfc.edu/XXX, where XXX is replaced with the DOI.

What next?

We are now focusing on the final stretch towards the official roll-out. Beyond this, we will continue to add sources and features and have a healthy roadmap to keep us on track. We value any feedback you have for us about your own journey with Event Data. Your feedback may help shape the direction we take in the future. Most of all, we are all excited to see what people build with it!

We look forward to continuing on our Event Data journey and we welcome you all aboard the train! Please contact me with your ideas.

Preprints growth rate ten times higher than journal articles

Jennifer Lin — Thu, 31 May 2018 00:00:00 +0000

The Crossref graph of the research enterprise is growing at an impressive rate of 2.5 million records a month - scholarly communications of all stripes and sizes. Preprints are one of the fastest growing types of content. While preprints may not be new, the growth may well be: ~30% for the past 2 years (compared to article growth of 2-3% for the same period). We began supporting preprints in November 2016 at the behest of our members. When members register them, we ensure that: links to these publications persist over time; they are connected to the full history of the shared research results; and the citation record is clear and up-to-date.

Summary

As of May 24, 2018 we have 44,388 works (see API query https://api-crossref-org.pluma.sjfc.edu/types/posted-content/works with a json viewer) registered as posted content. Today that number is over 150k. Preprints are part of this record type category, which is meant to house scholarly outputs that have been posted online and intended for publication in the future.

For a more granular view, see the monthly stats captured by Jordan Anaya in PrePubMed. This data is based on a slightly different set of preprint repositories, though both show the same trends.

The figure below shows the preprints registered with Crossref, broken down by repository.

We eagerly await our newest preprints member, Center for Open Science, who will soon be registering the preprints from their 18 community archives with us (~9k preprints total to date).

Metadata coverage

We accept a range of metadata for the preprints registered with us, including:

Repository name & hosting platform
Contributor names & ORCID iDs
Title
Dates (posted, accepted)
License
Funding
Abstract
Relations
References

As with all resource/record types, certain metadata is required, though others are optional. We encourage full coverage of metadata in the record where applicable and possible. So what are publishers including in their posted content records? The summary view is as follows:

License: 9926 (json), 22% (PeerJ Preprints, ChemRxiv)
Funder: 0 (json), 0%
ORCID: 19309 (json), 44% (bioRxiv, PeerJ Preprints, Preprints.org, ChemRxiv)
Abstracts: 35874 (json), 81% (bioRxiv, PeerJ Preprints, ChemRxiv)
References: 1921 (json):, 4% (JMIR)

Compared to all the published content registered with us over time, preprints have above average coverage of ORCID iDs deposited and show well above average with abstract metadata. However, they are significantly lagging behind with depositing references, license, and funding metadata. (See a summary of the full corpus stats taken two months ago in the blog post, A Lustrum over the Weekend).

Preprint-article pairs

Members registering preprints have an obligation to update the metadata record when a journal article is subsequently published, to clearly identify this work. This pairing is passed on to our metadata users: indexing platforms; recommendations engines; platforms; tools, etc. which pull from our APIs. (The preprint landing page also must link to the article.) As such, the preprint-article pairings are amassing as each week passes. We currently have a total of 12983 (json) preprints connected to articles. The figure below provides the counts based on repository.

Citations

We can see from preprint Cited-by counts that researchers are indeed citing preprints in their articles. This practice is an extension of the common citation behavior to provide evidence for and credit to previous work, a natural consequence of work shared with their peers. The most highly cited preprint papers (json) as of May 24, 2018 are as follows. In some cases, a subsequent paper was published from the results shared in the preprint. These have also accrued citations in their own right and these are also indicated in the table below.

No.	Cited-by	Preprint DOI	Preprint title	Date	Subsequent journal article	Citations of journal article
1	Cited-by 72	https://doi-org.pluma.sjfc.edu/10.1101/005165	qqman: an R package for visualizing GWAS results using Q-Q and manhattan plots	May 14, 2014.	n/a	n/a
2	Cited-by 63	https://doi-org.pluma.sjfc.edu/10.1101/002824	HTSeq - A Python framework to work with high-throughput sequencing data	August 19, 2014	Bioinformatics, https://doi-org.pluma.sjfc.edu/10.1093/bioinformatics/btu638	2372
3	Cited-by 43	https://doi-org.pluma.sjfc.edu/10.1101/030338	Analysis of protein-coding genetic variation in 60,706 humans	May 10, 2016	Nature, https://doi-org.pluma.sjfc.edu/10.1038/nature19057	1598
4	Cited-by 38	https://doi-org.pluma.sjfc.edu/10.1101/002832	Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2	November 17, 2014	Genome Biology, https://doi-org.pluma.sjfc.edu/10.1186/s13059-014-0550-8	3284
5	Cited-by 32	https://doi-org.pluma.sjfc.edu/10.1101/021592	Salmon provides accurate, fast, and bias-aware transcript expression estimates using dual-phase inference	August 30, 2016	Nature Methods, https://doi-org.pluma.sjfc.edu/10.1038/nmeth.4197	112
6	Cited-by 22	https://doi-org.pluma.sjfc.edu/10.1101/012401	DensiTree 2: Seeing Trees Through the Forest	December 8, 2014	n/a	n/a
7	Cited-by 21	https://doi-org.pluma.sjfc.edu/10.1101/011650	FusionCatcher - a tool for finding somatic fusion genes in paired-end RNA-sequencing data	November 19, 2014	n/a	n/a
8	Cited-by 19	https://doi-org.pluma.sjfc.edu/10.1101/048991	Analysis of shared heritability in common disorders of the brain	September 6, 2017	n/a	n/a
9	Cited-by 18	https://doi-org.pluma.sjfc.edu/10.1101/006395	Error correction and assembly complexity of single molecule sequencing reads	June 18, 2014	n/a	n/a
10	Cited-by 18	https://doi-org.pluma.sjfc.edu/10.1101/032839	Spread of the pandemic Zika virus lineage is associated with NS1 codon usage adaptation in humans	November 25, 2015	n/a	n/a

The relationship between preprints and the proceeding publication is an interesting area that is not yet well understood. We invite the community to analyze the Crossref metadata using the REST API in concert with other datasets. For example, the citation lifecycle for these two research products has been one of speculation so far without a systematic investigation into patterns and timeframes of preprint citations and those of its succeeding article across the corpus. Here, submission dates would be critical data to this research question as publication windows vary significantly by publisher and by paper.

Linking references is different from registering references

Anna Tolwinska — Wed, 30 May 2018 00:00:00 +0000

From time to time we get questions from members asking what the difference is between reference linking and registering references as part the Content Registration process.

Here’s the distinction:

Linking out to other articles from your reference lists is a key part of being a Crossref members - it’s an obligation in the membership agreement and it levels the playing field when all members link their references to one another.

Registering references when you register your content is completely different. It’s enriching the metadata record that describes your content, and it allows Crossref and others—including non-members—to use them.

Reference Linking

A research article usually includes a reference list of citations to other works that helped inform it. The original function of Crossref was to provide a central service for publishers that enabled them to link to each others’ content from these reference lists—using a DOI as a persistent link. This meant that members of all sizes and in all disciplines could easily link to one another without having to sign hundreds of bilateral agreements.

We made Reference Linking obligatory for Crossref members because it’s fundamental to making content discoverable, and because when everyone links their references, research travels further and benefits everyone.

Registering references

Every single day hundreds of members register and update their metadata with us—and every single day hundreds of organisations search for, extract and use it. To make sure your content is discovered in this process, it’s important to make the metadata you register with us as rich as possible. Rich metadata includes information such as journal title, article author, publication date, page numbers, ISSN, abstracts, ORCID iDs, funding information, clinical trials numbers, license information, and of course—references.

Additionally, registering references is ~~a prerequisite~~ recommended for participating in our Cited-by service—which provides citation counts and lists, and ultimately makes your content more discoverable. [EDIT 7th February 2024 - it is no longer required but highly recommended.]

We know it’s not easy for smaller publishers to deposit references. Read more on how to here. Our upcoming Metadata Manager tool will allow you to register your references at the same time as the rest of your content. This service is currently in development but let us know if you want to try it out. [EDIT 7th February 2024 - Metadata Manager has been deprecated. More info about it here.]

Reference Linking

Reference Linking means adding Crossref DOI links to the reference list for journal articles on your article pages as per this example: https://doi-org.pluma.sjfc.edu/10.1088/1367-2630/1/1/006.

How it works

First retrieve DOIs for all available references either through our human or machine interfaces. Then make sure you use the DOI link in your references and on your article landing page using the Crossref DOI display guidelines.

Why it’s useful

Reference Linking:

Enables you to link to more than 10,000 publishers without having to sign multiple agreements
Helps with discoverability, because DOIs don’t break if implemented correctly
Displays your DOIs as URLs so that anyone can copy and share them
Makes your content more useful to readers
Drives traffic to your website from other publishers.

Is it obligatory?

Yes, within a short time after becoming a member you should be including references.

Registering References

Registering references means submitting them as part of your Crossref metadata deposit as per this example: https://www-crossref-org.pluma.sjfc.edu/xml-samples/article_with_references.xml.

How it works

Whenever you register content with us, make sure you include your references in the submission. You can also add references to your existing content via a metadata redeposit, or our resource-only deposit, or our Simple Text Query form.

Why it’s useful

References registered as part of your metadata:

Make your content more discoverable
Make your content richer and more useful
Are required to participate in our Cited-by service (this service shows what articles cite your article)
Enables discovery of research
Enables evaluation of research
Highlights your contents’ provenance
Helps with citation counts.

Is it obligatory?

No, it’s optional, but strongly encouraged. It is ~~required~~ recommended if you are participating in our Cited-by service. [EDIT 7th February 2024 - it is no longer required but highly recommended].

If you have any questions about reference linking or registering your references please get in touch.

Hello, meet Event Data Version 1, and new Product Manager

Christine Buske — Thu, 29 Mar 2018 00:00:00 +0000

I joined Crossref only a few weeks ago, and have happily thrown myself into the world of Event Data as the service’s new product manager. In my first week, a lot of time was spent discussing the ins and outs of Event Data. This learning process made me very much feel like you might when you’ve just bought a house, and you’re studying the blueprints while also planning the house-warming party.

If Event Data is like a house, it’s been built and we’ve recently been putting on a last coat of paint. We’re very happy to announce version 1 of the API today. This is bringing us closer to the launch (house warming party), which will officially present Event Data to the world. Further to that analogy, while I bought into the house, I wasn’t around to see it being built. That’s both incredibly exciting and a little daunting.

Version 1 contains fixes for some challenges we came up against. Like scalability, data modeling for Wikipedia, and polishing. Version 1 is a new release of the data, but it is the same data set you already know and love. It should solve some of the recent stability issues, for which we apologize.

Moving forward, we expect the data model in V1 to persist and are not planning to make further large scale, fundamental changes to the Event Data API. As such, the version 1 release of the API is exceptional and a big step forward. It is important that we address these fixes before we go into production as it affects everyone who uses the service.

Same Event Data, new address

In setting up for the upcoming production service rollout, we have updated the Event Data API domain so that it is in line with Crossref’s suite of APIs. The Query API can now be found at a new URL. Here is an example query: https://api-eventdata-crossref-org.pluma.sjfc.edu/v1/events?rows=1

We have also simplified the standard query parameters in favor of a cleaner filter syntax.

Lastly, we have added a new “Mailto” parameter, just like in our REST API. It is encouraged but optional, so you are not obliged to supply it. We’ll only use it to contact you if there’s a problem.

Changes to the Wikipedia data structure

We’ve done a lot of work to use the canonical URLs for web pages to represent content as consistently as possible. This has entailed updating previously collected Events across data sources. As such, we’ve updated our Wikipedia data model to align with this. Because this update has impacted every Wikipedia Event in the system, we recommend those who have used or saved existing data from the deprecated Query API version to pull a new copy of the data. Read more about the rationale for changing the Wikipedia data model.

Updated data

This then brings me to how we now handle updated data. Sometimes we edit Events to add new features, or we may edit Events if there is an issue processing and/or representing the data when we provision it to the community. And sometimes we must remove Events to comply with a particular data source’s terms and conditions (ex: deleted Tweets). You can read about how updates work in the user guide.

To make life easier moving forward, we’ve split updated Events into two API endpoints. If you are already using Event Data, you will need to make some small updates to your client(s) to align with this. The new endpoints are further described in the documentation.

Event Data beta group

With the version 1 release we are making solid progress towards an official launch (the house-warming party!), we are quite excited to hear how you are using Event Data. Please consider [joining our beta group] (https://groups.google.com/forum/#!forum/crossref-event-data-beta-testers), if you are using the Event Data API or want to hear about updates.

This is also where you can read about these updates in more detail.

For more information and to get started with Crossref Event Data, please refer to the user guide.

I am looking forward to seeing how Event Data is being used, and working with the community to continuously improve what we can offer through this service. Feedback is always welcome, feel free to get in touch with me at eventdata@crossref.org.

Publishers, help us capture Events for your content

Madeleine Watson — Mon, 02 Oct 2017 00:00:00 +0000

The day I received my learner driver permit, I remember being handed three things: a plastic thermosealed reminder that age sixteen was not a good look on me; a yellow L-plate sign as flimsy as my driving ability; and a weighty ‘how to drive’ guide listing all the things that I absolutely must not, under any circumstances, even-if-it-seems-like-a-really-swell-idea-at-the-time, never, ever do.

The margin space dedicated to finger-wagging left little room for championing any driving-do’s. And as each page delivered a fresh new warning, my enthusiasm for hitting the road sunk to levels usually reserved for activities like trigonometry and visits to my orthodontist.

Many years (and an excellent driving record) later, I’m reminded of this again now when thinking about our own Event Data User Guide. Because it contains a chapter with some really important don’ts for our members. Really good, we’d-love-you-to-consider-not-doing-these-things type of advice. But despite our intent to encourage, I feel the ghost of finger-waggers past. So in the spirit of championing enthusiasm over ennui, I thought I’d attempt to contextualise our Event Data Best Practices Guide for Publishers and show you why there’s a lot of good reasons for publishers to be enthusiastic about these rules.

So if you’re a publisher, I encourage you to read on to learn more about how you can help us have the best chance possible of capturing Events for your content.

What’s in it for you? Well, collecting this data helps to give everyone (Crossref, yourself, and others) a better picture of how your content is being used, including for altmetrics.

1. Please let us in

Please do open the door when we come knocking, we promise not to stay long. You can do this by allowing the User Agent CrossrefEventDataBot to visit your site, and whitelisting it if necessary. The bot is how we visit URLs to confirm if they are for an item of content registered with us. The reason why we’re visiting your site could include:

someone tweeted an article landing page
someone discussed it on Reddit
it was linked to from a blog post

The Bot has only one job: to work out the DOI. No information beyond this is stored. Whenever we become aware of a link that we think points to a DOI or an Article Landing Page, we follow it so we can collect the required metadata. Everything in Crossref Event Data is linked via its DOI, so it’s important that we can collect this information.

The bot will identify itself using the standard method. It sets two headers:

Referer: https://eventdata-crossref-org.pluma.sjfc.edu
User-Agent: CrossrefEventDataBot (eventdata@crossref.org)

Once we confirm that a link points to registered content, we then log an Event for the DOI. You should expect our bot to visit no more than once or twice per second, although if there is a period of activity around your articles, you may see higher rates. The bot also takes a sample of DOIs and visits them to work out which domain names belong to our members, so it can maintain a list. This can happen every few weeks. You may see a small number of requests from the bot, but limited to one per second.

If we can’t enter your site to look for metadata though, then we won’t be able to collect Events for your DOIs. So by allowing our bot, you will be helping us to collect Event Data for your registered content.

If you’re worried about traffic on your site, consider sending us your mapping of article landing pages to DOIs. Because Resource URLs aren’t the same as article landing pages, we need more information than the DOI Resource URLs that you already send us.

If you’re running a blog or website (and you’re not a member of Crossref), you may also see our bot visiting, to look for links that comprise Events. Please allow us to visit, so we can record in our Event Data service the fact that your website links to registered content.

2. We ❤️ robots.txt

Robots.txt files are important and we ensure our Event Data Bot respects yours. If we are instructed not to visit a site, we won’t. So if you want us to visit your site in order to check the metadata of your article landing page, please ensure you provide an exception for our Bot, or make sure that you’re not blocking it. Check the restrictions in your file to see if we’re allowed to visit. This is just another way you can help us work for you.

3. Include the DC Identifier

Including good metadata is general best practice for scholarly publishing. When we visit a publisher’s site, we look for metadata embedded in the HTML document (such as DC.Identifier tags that, amongst other things, enable Crossmark to work).

By ensuring you include a Dublin Core identifier meta tag in each of your articles pages, our system can match your landing pages back to DOIs.

Here’s an example:

4. Let us in, even if we don’t bring cookies

We’re like that friend who turns up for dinner without bringing a bottle of wine. And we hope that you’ll be ok with that. Some Publisher sites don’t allow browsers to visit unless cookies are enabled and they block visitors that don’t accept them. If your site does this, we will be unable to collect Events for your DOIs. Allowing your site to be accessed without cookies will help give us the best chance of successfully reading your metadata.

5. We may not speak your language

Sometimes we come across a publisher’s site that won’t render unless JavaScript is enabled. This means that the site won’t show any content to browsers that don’t execute JavaScript. The Event Data Bot does not execute JavaScript when looking for a DOI. This means that if your site requires JavaScript, then we will be unable to collect DOIs for your Events. Consider allowing your site to be accessed without JavaScript. And if this is not possible, then if you ensure you include the tag in the HTML header, then we’ll do our best to collect Events for your registered content.

If you want to pass this on to your friendly system administrator, the best practice is documented in full here: https://www-eventdata-crossref-org.pluma.sjfc.edu/guide/best-practice/publishers-best-practice/. And sorry about all the don’ts you’ll find on that page…. don’t let them curb your enthusiasm for taking Event Data out for a spin!

More metadata for machines-citations, relations, and preprints arrive in the REST API

Kirsty Meddings — Mon, 11 Sep 2017 00:00:00 +0000

Over the past few months we have been adding to the metadata and functionality of our REST API, Crossref’s public machine interface for the metadata of all 90 million+ registered content items. Much of the work focused on a review and upgrade of the API’s code and architecture in order to better support its rapidly growing usage. But we have also extended the types of metadata that the API can deliver.

One of the biggest changes is that references are now available if the publisher has made them public (a simple email instruction to us). Currently 45% of all publications with deposited references are now accessible. For example:

This article studying fluid ejection from animals has 55 references and they are all in the metadata here. You can also see that the article has an is-referenced-by count of 6.
This article exploring whether people bitten by their cat are more likely to develop depression has 142 references and is referenced by 12.

We recently announced that we would be accepting preprints, and the metadata for 15,000 preprints registered to date is now in the API, labelled as posted-content. Over 4,000 have been subsequently published in a journal, and the Crossref metadata now links these preprints to their respective articles (and vice versa). For example this article in Biorxiv has since been published in a journal, and this relationship is recorded in its metadata as is-preprint-of.

Also new to the API:

Cited-by counts - the number of times each work has been referenced by other content registered with us. Look for is-referenced-by-count within a record.
This article from 1953 about a fairly notable discovery has been cited 4832 times, but the two most cited articles both have over 100,000 citations and thousands have been cited more than Watson and Crick.
Abstracts for over 1 million works.
Similarity Check URLs–the ones that Turnitin crawl to add content to the database–are now showing so that participating publishers can check that they are including them in their metadata deposits.
Subject categories have been added for an additional 7000 journal titles, taking the total number of classified titles to ~45,000.

Are you already using our Metadata APIs for your system or project? We’re always keen to hear new use cases and happy to answer any questions.

You may need to install a JSON viewer extension in your browser to render API queries in a human-friendly way.

Event Data enters Beta

Jennifer Kemp — Wed, 05 Jul 2017 00:00:00 +0000

We’ve been talking about it at events, blogging about it on our site, living it, breathing it, and even sometimes dreaming about it, and now we are delighted to announce that Crossref Event Data has entered Beta.

A collaborative initiative by Crossref and DataCite, Event Data offers transparency around the way interactions with scholarly research occur online, allowing you to discover where it’s bookmarked, linked, liked, shared, referenced, commented on etc., across the web, and beyond publisher platforms.

The name Event Data reflects the nature of the service, as it collects and stores digital actions that occur on the web, from the quick and simple, such as bookmarking and referencing, through to deeper interconnectivity such as exposing the links between research artifacts. Each individual action is timestamped and recorded in our system as an Event, and made available to the community via an API.

Event Data will be available for absolutely anyone to use; publishers, third party vendors, editors, bibliometricans, researchers, authors, funders etc., and with tens of thousands of events occurring every day, there’s a wealth of insight to be gained for those interested in analyzing and interpreting the data.

It’s important to note that Event Data does not provide metrics. What is does provide is the raw data to help you facilitate your own analysis, giving you the freedom to integrate the data into your own systems.

We are currently working very closely with a few organisations with specific use cases who are helping us to test and refine Beta before we launch our production service later this year. If you decide to take a look at Beta yourself, all the data you collect from Event Data is licensed for public sharing and reuse according to our Terms of Use.

Until Event Data is in production mode, we do not recommend building any commercial or customer-based tools off the data. If you are not in the Beta test group but are interested in participating, please contact me below. For more information about Event Data, please see our user guide.

Please contact me, Jennifer Kemp—Outreach Manager for Event Data—with any questions.

Data citations and the eLife story so far

Melissa Harrison — Thu, 18 May 2017 00:00:00 +0000

When we set up the eLife journal in 2012, we knew datasets were an important component of research content and decided to give them prominence in a section entitled ‘Major datasets’ (see images below). Within this section, major previously published and generated datasets are listed. We also strongly encourage data citations in the reference list.

Major Datasets for “Structural basis of protein translocation by the Vps4-Vta1 AAA ATPase” by N. Monroe, H. Han, P. Shen, et. al.

Almost five years on and I feel we have still not cracked it! We have signed up to the Force11 data citation principles, which were published three years back; we have been actively involved in working groups of Force11 and others, for example the Data Citation Roadmap for Scientific Publishers and the JATS XML data citation recommendation of JATS4R. I am also currently working with other publishers to come up with recommended JATS XML tagging for data availability statements, which is easier said than done considering the nuances of dataset uses and also how different publishers approach this.

Added to this, there is still significant push-back from authors about putting all dataset citations in the reference list (for example, authors are concerned about self-citing by citing a dataset created as part of the research article; “dataset citations” that are in effect a link to a search results page on a database; and the necessitation of hundreds of reference entries if an author has used a large base for the research).

While eLife is very active in this space, and aims to arrange and mark up the datasets and citations produced by our authors in line with recommendations, the recommendations still have some gaps and the complete picture is not yet clear.

In late 2014, we brought in-house the process of depositing Crossref metadata (previously our online host did this for us). It gave us control of our processes and, at the time, we sent all the information we could to Crossref and have ensured our references are open and available in the Crossref public API. The code for this conversion process is all open-source and available for reuse. It can be found on GitHub. Since then, besides small improvements to the code and troubleshooting problems, we’ve not updated the code. I have been keeping a list of Crossref features and new deposit metadata we can add to our deposits, and now is the time for us to start working on this again.

One of the items we’ll be addressing is data citations.

The Crossref reference schema does not cater well for non-book or -journal content, and if an item does not have a DOI, the “reference” is not very useful because of the few tags available in the Crossref schema.

However, Crossref have introduced the relationship type to their schema, so data references can be well linked and mineable. As I see Crossref as a potential broker between publishers and data repositories in the future, using the relationship-type deposit for all datasets will assist this and also allow these data points to more easily be seen within the article Nexus framework (see the recent blog post, How do you deposit data citations?).

At eLife, we already distinguish between Dataset generated as part of research results (relationship type in the Crossref schema: “isSupplementedBy”) and Dataset produced by a different set of researchers or previously published (relationship type: “references”). Therefore, it will not be hard for us to convert all the information about data referencing that is within the dataset section into a relationship-type deposit in the conversion to Crossref XML.

We have also recently gone through an exercise of defining a set of rules for all our references and, of the 12 allowed types, one is data. The rules for Schematron (a rule-based validation language for making assertions about the presence or absence of patterns in XML trees; see also this useful article about Schematron on the JATS4R learning centre) have been written for the eLife ‘business’ rules. Subject to final testing, these will be integrated into our workflow (the Schematron is open source and available for reuse on GitHub, and we will also build an API for people to use the Schematron direct). This will allow us to easily identify all data references and convert them into relationship types in the XML delivered to Crossref. This way, they will not be lost in the references section of our deposits, but properly identified.

However, we do appreciate this will become harder for us as authors become more familiar with datasets as references, because we will not be able to identify the difference between generated and analysed datasets so easily.

The code developed and used to complete these conversions will, again, be on Github and open source, and we actively encourage the reuse of this.

While the industry is still working on the best way to deal with data and ensuring it is given the prominence it requires, we feel this is the best approach we can take. Nothing is forever and we can still change what we do in the future. The beauty of open-source code also means that if there is an alternative approach now or in the future, the code we wrote at eLife can be developed by someone else in the future and we can all benefit.

If you have any questions, please do not hesitate to contact us.

How do you deposit data citations?

Jennifer Lin — Thu, 02 Mar 2017 00:00:00 +0000

Please visit Crossref’s official Data & Software Citations Deposit Guide for deposit details.

Very carefully, one at a time? However you wish.

Last year, we introduced linking publication metadata to associated data and software when registering publisher content with Crossref Linking Publications to Data and Software. This blog post follows the “whats” and “whys” with the all-important “how(s)” for depositing data and software citations. We have made the process simple and fairly straightforward: publishers deposit data & software links by adding them directly into the standard metadata deposit via relation type and/or references. This is part of the **existing Content Registration ** process and requires no new workflows.

Relationships

Data & software citations are a valuable part of the “research article nexus”, comprised of the publication linked to a variety of associated research objects, including data and software, supporting information, protocols, videos, published peer reviews, a preprint, conference papers, etc. For all of these resources, we use relation types in the metadata deposit to “anchor” the article in the article nexus and link to it.

For data & software, we ask for:

identifier of the dataset/software
identifier type: “DOI”, “Accession”, “PURL”, “ARK”, “URI”, “Other” *
relationship type: “isSupplementedBy” or “references”
description of dataset or software.
*Additional identifier types beyond those used for data or software are also accepted, including ARXIV, ECLI, Handle, ISSN, ISBN, PMID, PMCID, and UUID.

Crossref maintains an expansive set of relationship types to support the various resources linked in the research article nexus. For data and software, we recommend “isSupplementedBy” and “references” as relationship types in the metadata. Use the former if it was generated de novo as part of the research results. For those generated by another project and then reused, we recommend applying “references” in the relationship type. These were selected in consultation with DataCite and data working groups. They will provide the level of specificity requested by the community.

To illustrate how to represent the link within the metadata deposit, we offer two examples from two popular dataset identifiers, one for each of the relationship types.

Dataset	Snippet of deposit XML containing link
Dataset with DOI: Data from: Extreme genetic structure in a social bird species despite high dispersal capacity. Database: Dryad Digital Repository DOI: https://doi-org.pluma.sjfc.edu/10.5061/dryad.684v0	`<program xmlns="http://www.crossref.org.pluma.sjfc.edu/relations.xsd">` `<related_item>` `<description>Data from: Extreme genetic structure in a social bird species despite high dispersal capacity</description>` `<inter_work_relation relationship-type="isSupplementedBy" identifier-type="doi">10.5061/dryad.684v0</inter_work_relation>` `</related_item>` `</program>`
Dataset with accession number: NKX2-5 mutations causative for congenital heart disease retain functionality and are directed to hundreds of targets Database: Gene Expression Omnibus (GEO) Accession number: GSE44902 URL: https://www-ncbi-nlm-nih-gov.pluma.sjfc.edu/geo/query/acc.cgi?acc=GSE44902	`<program xmlns="http://www.crossref.org.pluma.sjfc.edu/relations.xsd">` `<related_item>` `<description>NKX2-5 mutations causative for congenital heart disease retain and are directed to hundreds of targets</description>` `<inter_work_relation relationship-type="references" identifier-type="Accession">GSE44902</inter_work_relation>` `</related_item>` `</program>`

In the examples above, the Dryad dataset was generated as part of the research published in an article. Hence, it contains the “isSupplementedBy” relationship type. The GEO dataset was reused by and referenced in a scholarly article published separate from the project that generated this dataset. Hence, it contains the “references” relationship type.

Both Crossref and DataCite employ this method of linking. Data repositories who register their content with DataCite follow the same process and apply the same metadata tags. This means that we achieve direct data interoperability with links in the reverse direction (data and software repositories to journal articles).

References

Another mechanism for depositing data and software citations is to insert it into the manuscript’s references. Publishers then deposit it as part of the article’s references. To do so, publishers follow the general process for depositing references. (Visit Crossref’s Support page for step-by-step instructions.)

Publishers can deposit the full data or software citation as a unstructured reference.
<citation key="ref=3">
<unstructured_citation>Morinha F, Dávila JA, Estela B, Cabral JA, Frías Ó, González JL, Travassos P, Carvalho D, Milá B, Blanco G (2017) Data from: Extreme genetic structure in a social bird species despite high dispersal capacity. Dryad Digital Repository. http://dx.doi.org.pluma.sjfc.edu/10.5061/dryad.684v0</unstructured_citation\>
</citation>
</citation_list>

Or they can employ any number of reference tags currently accepted by Crossref. Most do not readily suit datasets and software as the suite was originally established to match article and book references. This leaves out substantial metadata needed to identify and describe the dataset, however, if the resource does not have a DOI.
<citation key="ref2">
<doi>10.5061/dryad.684v0</doi>
<cYear>2017</cYear>
<author>Morinha F, Dávila JA, Estela B, Cabral JA, Frías Ó, González JL, Travassos P, Carvalho D, Milá B, Blanco G</author>
</citation>
We are exploring the JATS4R recommendations while we consider expanding the current collection. We welcome additional suggestions from the community.

Precise, accessible links

Crossref’s infrastructure is setup to facilitate the flow of information about scholarly works across the research network. We maintain a fair degree of flexibility both in the structure and completeness of metadata deposited. The aim, though, is to make the links rich in metadata, accurate in associating literature to corresponding resource, and available to both human and machine consumers as per Principle #5 and #7 in the Joint Declaration of Data Citation Principles.

As with the other associated resources in the article nexus, we recommend depositing data/software links in the publication metadata via relationships. Publishers are free to do this on top of or independent of references. Relationship metadata offer a high degree of precision. References are a hodgepodge of various resources cited by the publication, including articles, books, media, blogs, reference materials, etc. and data citations are hard to isolate. Furthermore, the unstructured, “spaghetti string” text is difficult for systems to parse and extract specific information.

With relationship metadata, data and software resources are expressly designated. We obtain a more accurate link that specifies identifier type and explicitly identifies data generated as part of the research shared in the paper or as reuse of existing data). The richer metadata contained here enables consumers to conduct powerful queries based on different attributes (identifier type, description, relationship), taking data discovery and mining to the next level.

Furthermore, relationships are important for achieving full accessibility of data and software citations. Access to references is based on publisher permission so not all data citations can be shared (excluding DataCite DOIs). In contrast, all links deposited via relationships are publicly available.

Publishers play an important role in supporting research validation and reproducibility. Data & software citation is a basic part of of this practice, and instrumental in enabling the reuse and verification of these research outputs, tracking their impact, and creating a scholarly structure that recognizes and rewards those involved in producing them. For the full scoop of how to deposit (i.e., technical details and more), we encourage you to reference the Crossref Data & Software Citations Deposit Guide and contact us (support@crossref.org) with questions or feedback.

Linking Publications to Data and Software

Jennifer Lin — Wed, 07 Sep 2016 00:00:00 +0000

TL;DR

Crossref and Datacite provide a service to link publications and data. The easiest way for Crossref members to participate in this is to cite data using DataCite DOIs and to include them in the references within the metadata deposit. These data citations are automatically detected. Alternatively and/or additionally, Crossref members can deposit data citations (regardless of identifier) as a relation type in the metadata. Data & software citations from both methods are freely propagated. This blog post also describes how to retrieve the links collected between publication and data & software.

Data & software citation is good research practice (DataCite-STM Joint Statement and FORCE11 Joint Declaration of Data Citation Principles) and is part of the scholarly ecosystem supporting research validation and reproducibility. Data & software citation is also instrumental in enabling the reuse and verification of these research outputs, tracking their impact, and creating a scholarly structure that recognises and rewards those involved in producing them.

Crossref supports the propagation of data & software citations alongside a publisher’s standard bibliographic metadata. members deposit the data citation link as part of the overall publication metadata when registering their content. Crossref partners with DataCite and together, we jointly provide a clearinghouse for the citations collected. These are all made freely available to the community as open data.

Citation practices are evolving across different communities of practice. Crossref’s offering is flexible and easily accommodates variations and changes, since it does not rely on a specific set of citation metadata elements, citation format, nor manner of credit and attribution. Publishers deposit data & software citations in their metadata deposit via a) references and/or b) relation type.

Method A: Bibliographic references

Crossref and DataCite have partnered to provide automatic linking between publications registered with Crossref and datasets bearing DataCite DOIs. This is the most efficient and effective way to ensure that data citations are fully integrated into the scholarly research information network with full and accurate metadata.

All data & software citations that include datasets bearing a DataCite DOI are eligible for auto-update linking with Crossref. In this method: authors cite the dataset or software containing the DataCite DOI per journal article submission guidelines and add it to the article citation list (c.f. FORCE11 citation placement, FORCE11 Software Citation Principles). Publishers then deposit references as part of their standard practice when registering content. Crossref checks every reference deposited for a DOI. If the DOI is identified as DataCite’s, we automatically link it to the article. With this method, no additional action is needed when publishers register their content with Crossref.

Data citation links to non-DataCite DOIs can only be exposed in the references if the publisher makes references openly available. Even in the event that the data citation is shared, it remains undifferentiated from other references. Method B described below offers another approach.

Method B: Relation type

Publishers can link their publication to a variety of associated research objects as part of the article metadata directly in the metadata deposited to Crossref, including data & software, protocols, videos, published peer reviews, preprints, conference papers, etc. Doing so not only groups digital objects together, but formally associates them with the publication. Each link is a relationship and the sum of all these relationships constitutes a ‘research article nexus.’ Data & software citations are a valuable part of this.

To tag the citation in the metadata deposit, we ask for:

description of dataset or software (optional)

dataset or software identifier

identifier type

relationship type.

Crossref can accommodate research outputs with any identifier, though we currently only validate DOI relationships during metadata processing. Technical details are documented in the [Data & Software Citations Deposit Guide][4].

Combining methods increases total available citations

The two methods are independent and can be used exclusively or jointly. Each caters to a different set of conditions and their practical considerations. See the comparison of benefits and limitations for each method in the deposit guide. We recommend that publishers use both methods where possible at this time for optimum specificity and coverage.

How to access data & software citations

Crossref and DataCite make the data & software citations deposited by Crossref members and DataCite data repositories openly available to a wide host of parties, including both Crossref and DataCite communities as well as the extended research ecosystem (funders, research organisations, technology and service providers, research data frameworks such as Scholix, etc.).

Data & software citations from references can be accessed via the Crossref Event Data API Citations included directly into the metadata by relation type can be accessed via Crossref’s APIs in a number of formats (REST, OAI-PMH, OpenURL). (A single channel containing data & software citations across interfaces is in development and will be released next year.)

Publishers, visit our detailed guide on how to deposit data and software citations. We welcome your questions and concerns at feedback@crossref.org.

Special thanks to the following who provided valuable feedback in developing the guide: Martin Fenner (DataCite), Amye Kenall (Springer Nature), Brooks Hanson (AGU), Shelley Stall (AGU), and the FORCE11 Data Citation Implementation Pilot publisher’s subgroup.

The article nexus: linking publications to associated research outputs

Jennifer Lin — Thu, 25 Aug 2016 00:00:00 +0000

Crossref began its service by linking publications to other publications via references. Today, this extends to relationships with associated entities. People (authors, reviewers, editors, other collaborators), funders, and research affiliations are important players in this story. Other metadata also figure prominently in it as well: references, licenses and access indicators, publication history (updates, revisions, corrections, retractions, publication dates), clinical trial and study information, etc. The list goes on.

What is lesser known (and utilized) is that Crossref is increasingly linking publications to associated scholarly artifacts. At the bottom of it all, these links can help researchers better understand, reproduce, and build off of the results in the paper. But associated research objects can enormously bolster the research enterprise in many ways (e.g., discovery, reporting, evaluation, etc.).

With all the relationships declared across all 80+ million Crossref metadata records, Crossref creates a global metadata graph across subject areas and disciplines that can be used by all.

Research article nexus

As research increasingly goes digital, more research artifacts associated with the formal publication are stored or shared online. We see a plethora of materials closely connected to publications, including: versions, peer reviews, datasets generated or analysed in the research, software packages used in the analysis, protocols and related materials, preprints, conference posters, language translations, comments, etc. Occasionally, these resources are linked from the publication. But very rarely are these relationships made available beyond the publisher platform.

Crossref will make these relationships available to the broader research ecosystem. When publishers register content for a publication, they can identify the associated scholarly artifacts directly in the article metadata. Doing so not only groups digital objects together, but formally associates with the publication. Each link is a relationship and the sum of all these relationships constitutes a “research article nexus.”

An assortment of connections already abound in the wild today. Examples include:

F1000Research article http://doi.org.pluma.sjfc.edu/10.12688/f1000research.2-198.v3 connected to initial version http://doi.org.pluma.sjfc.edu/10.12688/f1000research.2-198.v1

OECD publication http://dx.doi.org.pluma.sjfc.edu/10.1787/empl_outlook-2014-en and its German translation http://dx.doi.org.pluma.sjfc.edu/10.1787/empl_outlook-2014-de

PeerJ article http://doi.org.pluma.sjfc.edu/10.7717/peerj.1135 and its peer review http://doi.org.pluma.sjfc.edu/10.7287/peerj.1135v0.1/reviews/3

eLife article http://doi.org.pluma.sjfc.edu/10.7554/eLife.09771 and its BioArXiv preprint http://doi.org.pluma.sjfc.edu/10.1101/018317

PLOS ONE article http://doi.org.pluma.sjfc.edu/10.1371/journal.pone.0161541 with underlying data in Dryad http://doi.org.pluma.sjfc.edu/10.5061/dryad.d2vf8

Frontiers article http://doi.org.pluma.sjfc.edu/10.3389/fevo.2015.00015 with a figshare http://doi.org.pluma.sjfc.edu/10.6084/m9.figshare.1305089.v1 video

Journal of Chemical Theory and Computation article http://doi.org.pluma.sjfc.edu/10.1021/ct400399x with software archived in Zenodo http://doi.org.pluma.sjfc.edu/10.5281/zenodo.60678

Nature Biotech article http://doi.org.pluma.sjfc.edu/10.1038/nbt.3481 with a Protocols.io protocol http://doi.org.pluma.sjfc.edu/10.17504/protocols.io.dm649d

To date, almost all these relationships are not directly recorded in the article metadata (great job, PeerJ!). And as a result, they are more than likely “invisible” to the broader scholarly research ecosystem. Publishers can remedy these gaps by depositing associations when registering content with Crossref or updating the records after registration. That is how the article nexus is formed.

(Associated datasets can also be identified in the reference list as per Joint Declaration of Data Citation Principles as with the FORCE11 Software Citation Principles. Stay tuned next week for a follow up blog post on Crossref’s support for publisher data and software citations through its metadata.)

Forming the nexus

The mechanism of declaring these relationships is straightforward and a longstanding part of the standard deposit process. For each associated research object, simply provide the identifier and identifier type for the object, an optional description of it, as well as name the relationship into the metadata record. For the latter, Crossref and DataCite share a closed list of relationship types, which ensures interoperability between mappings. See Crossref technical documentation for more details.

We maintain a list of the recommended relation types for a host of associated research objects to promote standardization across publishers. If you have relationships not specified, please contact us at feedback@crossref.org to identify a suitable one considered best practice. Common adoption of relation types will make relationship metadata useful to tool builders and systems. For example, programmatic queries on supporting materials require proper tagging of their respective relationship types.

This approach is highly extensible and accommodates the introduction of new research object forms as they emerge. It also supports associated research objects regardless of identifier type. When an associated entity has a DOI, however, we can validate the relationship during metadata processing as well as provide a more reliable representation of the article nexus.

Article nexus: a far richer scholarly map

Bibliographic metadata is like a ship’s manifest that catalogs each item of cargo in a ship’s hold - crate, drum, sack, and barrel. It identifies the components that have an internal relation to the publication (contributor, funder, article update, license, etc.), each of which are well-understood points on the scholarly map. But when we integrate the article nexus into the graph, new territories become visible - not isolated islands, but places with highways connecting them to addresses already known.

When a publication has its relationships clearly identified, the connections both go out as well as lead back to it. The more connections, the more visibility on the scholarly map, as the Art of Cartography goes. Numerous systems tap into this map: publishing, funders, research institutions, research councils, indexers & repositories, indexers, research information systems, lab & diagnostics systems, reference management and literature discovery, other PID suppliers. So publishers, you can provide the fullest value to your own publishing operation, your authors, their research communities, and the overall research enterprise by ensuring that all publications are fully linked both inside and out.

Crossref Event Data: early preview now available

Madeleine Watson — Mon, 18 Apr 2016 00:00:00 +0000

Test out the early preview of Event Data while we continue to develop it. Share your thoughts. And be warned: we may break a few eggs from time to time!

Chicken by anbileru adaleru from the The Noun Project

Want to discover which research works are being shared, liked and commented on? What about the number of times a scholarly item is referenced? Starting today, you can whet your appetite with an early preview of the forthcoming Crossref Event Data service. We invite you to start exploring the activity of DOIs as they permeate and interact with the world after publication.

But first, a bit of background

Discussion around scholarly research increasingly occurs online after publication, for example on blogs, sharing services, social media, and wikis. These ‘events’ occur across the web on numerous platforms and are a critical part of the scholarly enterprise. We are developing an infrastructure service (Crossref Event Data) that collects, stores, and delivers raw data of the events occurring with Crossref DOIs. We will store the data in an open, auditable and portable form for the community to access. Publishers, platforms, funders, bibliometricians and service providers may benefit from access to this raw data, and it can be used to feed into research records or proprietary tools and services that offer aggregation and analysis.

For more information, see our pilot blog post and description of potential use cases.

Collaborative, transparent development

Developers Martin Fenner (DataCite) and Joe Wass (Crossref) enjoy a tofu break

Lagotto, the software originally developed at PLOS, has been extended and improved in a joint effort between DataCite and Crossref. The two DOI Registration Agencies have partnered to envision, build and release the service. On the 13th of April, after a year of collaboration, we jointly released Lagotto 5.0. You can read about the collaboration on the DataCite blog post.

Crossref and DataCite will continue to work closely together to develop Lagotto and the Event Data service. Although Crossref Event Data has mostly Crossref DOIs at launch, you will be able to find DataCite DOIs if they are cited in Crossref or Wikipedia.

All of the software that runs Event Data, including Lagotto, is developed in the open and is open source. Please refer to the Crossref Event Data Technical User Guide for full details.

Preview the data

This service is currently under development with a full launch expected the second half of 2016. Before it is launched however, we invite you to take a look around and preview a subset of the data sources we plan to include. You may experience occasional hiccups while we continue building the service.

At this stage, we are working with data from three sources although we will greatly expand the variety of platforms from which we collect data as development progresses. At this stage, you can view Mendeley bookmarks, Wikipedia references, and Crossref to DataCite links.

Mendeley

Mendeley is a reference manager and academic social network for scholars. View the number of social bookmarks from scholars or groups on Mendeley.

For example, doi.org/10.1016/J.JIP.2016.03.007 currently has 8 readers on Mendeley to date.

Wikipedia

Wikipedia is an online encyclopaedia, the Internet’s largest and most popular general reference work. View references in Wikipedia of Crossref publications in Wikipedia articles in all languages.

For example, doi.org/10.3897/ZOOKEYS.565.7185 was referenced in the Russian Wikipedia page on Oxyscelio

Crossref to DataCite links

DataCite is a global consortium that assigns DOIs to research data. This enables people to find, share, use, and cite data. You can view all the data citations to DataCite research outputs found in Crossref publications (work is underway to make the links found in DataCite metadata available in Event Data).

For example, Global, Regional, and National Fossil-Fuel CO2 Emissions (doi.org/10.3334/CDIAC/00001) dataset has been referenced by six Crossref publications to date. Software links are also included. Another example is PGOPHER (doi.org/10.5523/bris.huflggvpcuc1zvliqed497r2), a general purpose software for simulating and fitting rotational, vibrational and electronic spectra, which has been referenced by seven Crossref publications to date.

Ready to take a spin?

You can explore the Crossref Event Data early preview by visiting http://eventdata.crossref.org.pluma.sjfc.edu and following the links to featured examples within our interim application for inspecting the data, technical documentation, and our Quick Start guide.

This service is currently under development and as such we welcome your thoughts and feedback on the data we are collecting currently from our three active sources. As a reminder, we expect to include the following sources as part of our full service launch later this year (pending confirmation):

[table id=1 /]

We’re also on the lookout for new data sources to investigate for future inclusion in the Event Data service so please do get in touch with requests and recommendations. As we continue to build the service throughout 2016, we will be committing to a model of continuous development so that we can make new sources available as they are completed.

Watch this blog for regular updates on our progress, or subscribe to receive new blog posts by email (just add your details to the upper right side of this page).

Rehashing PIDs without stabbing myself in the eyeball

Geoffrey Bilder — Thu, 11 Jun 2015 00:00:00 +0000

Anybody who knows me or reads this blog is probably aware that I don’t exactly hold back when discussing problems with the DOI system. But just occasionally I find myself actually defending the thing…

About once a year somebody suggests that we could replace existing persistent citation identifiers (e.g. DOIs) with some new technology that would fix some of the weaknesses of the current systems. Usually said person is unhappy that current systems like

DOI, Handle, Ark, perma.cc, etc. depend largely on a social element to update the pointers between the identifier and the current location of the resource being identified. It just seems manifestly old-fashioned and ridiculous that we should still depend on bags of meat to keep our digital linking infrastructure from falling apart.

In the past, I’ve threatened to stab myself in the eyeball if I was forced to have the discussion again. But the dirty little secret is that I play this game myself sometimes. After all, the best thing a mission-driven membership organisation could do for its members would be to fulfil its mission and put itself out of business. If we could come up with a technical fix that didn’t require the social component, it would save our members a lot of money and effort.

When one of these ideas is posed, there is a brief flurry of activity as another generation goes through the same thought processes and (so far) comes to the same conclusions.

The proposals I’ve seen generally fall into one of the following groups:

Replace persistent identifiers (PIDs) with hashes, checksums, etc.
Just use search (often, but not always coupled with 1 above)
Automagically create PIDs out of metadata.
Automagically redirect broken citations to archived versions of the content identified
And more recently… use the blockchain

I thought it might help advance the discussion and avoid a bunch of dead ends if I summarised (rehashed?) some of the issues that should be considered when exploring these options.

Warning: Refers to FRBR terminology. Those of a sensitive disposition might want to turn away now.

DOIs, PMIDs, etc. and other persistent identifiers are primarily used by our community as “citation identifiers”. We generally cite at the “expression” level.
Consider the difference between how a “citation identifier” a “work identifier” and a “content verification identifier” might function.
How do you deal with “equivalent manifestations” of the same expression. For example the ePub, PDF and HTML representations of the same article are intellectually equivalent and interchangeable when citing. The same applies to csv & tsv representations of the same dataset. So, for example, how do hashes work here as a citation identifier?
Content can be changed in ways that typically doesn’t effect the interpretation or crediting of the work. For example, by reformatting, correcting spelling, etc. In these cases the copies should share the same citation identifier, but the hashes will be different.
Content that is virtually identical (and shares the same hash) might be republished in different venues (e.g. a normal issue and a thematic issue). Context in citation is important. How do you point somebody at the copy in the correct context?
Some copies of an article or dataset are stewarded by publishers. That is, if there is an update, errata, corrigenda, retraction/withdrawal, they can reflect that on the stewarded copy, not on copies they don’t host or control. Location is, in fact, important here.
Some copies of content will be nearly identical, but will differ in ways that would affect the interpretation and/or crediting of the work. A corrected number in a table for example. How would you create a citation form a search that would differentiate the correct version from the incorrect version?
Some content might be restricted, private or under embargo. For example private patient data, sensitive data about archaeological finds or the migratory patterns of endangered animals.
Some content is behind paywalls (cue jeremiads)
Content is increasingly composed of static and dynamic elements. How do you identify the parts that can be hashed?
How do you create an identifier out of metadata and not have them look like this?

This list is a starting point that should allow people to avoid a lot of blind alleys.

In the mean time, good luck to those seeking alternatives to the current crop of persistent citation identifier systems. I’m not convinced it is possible to replace them, but if it is- I hope I beat you to it. 🙂 And I hope I can avoid stabbing myself in the eye.

Real-time Stream of DOIs being cited in Wikipedia

Joe Wass — Tue, 03 Mar 2015 00:00:00 +0000

TL;DR

Watch a real-time stream of DOIs being cited (and “un-cited!” ) in Wikipedia articles across the world: https://live-eventdata-crossref-org.pluma.sjfc.edu/live.html

Background

For years we’ve known that the Wikipedia was a major referrer of Crossref DOIs and about a year ago we confirmed that, in fact, the Wikipedia is the 8th largest refer of Crossref DOIs. We know that people follow the DOIs, too. This despite a fraction of Wikipedia citations to the scholarly literature even using DOIs. So back in August we decided to create a Wikimedia Ambassador programme. The goal of the programme was to promote the use of persistent identifiers in citation and attribution in Wikipedia articles. We would do this through outreach and through the development of better citation-related tools.

Remember when we originally wrote about our experiments with the PLOS ALM code and how that has transitioned into the DOI Event Tracking Pilot? In those posts we mentioned that one of the hurdles in gathering information about DOI events is the actual process of polling third party APIs for activity related to millions of DOIs. Most parties simply wouldn’t be willing handle the load of a 100K API calls an hour. Besides, polling is a tremendously inefficient process, only a fraction of DOIs are ever going to generate events, but we’d have to poll for each of them, repeatedly, forever, to get an accurate picture of DOI activity. We needed a better way. We needed to see if we could reverse this process and convince some parties to instead “push” us information whenever they saw DOI related events (e.g. citations, downloads, shares, etc). If only we could convince somebody to try this…

Wikipedia DOI Events

In December 2014 we took the opportunity of the 2014 PLOS/Crossref ALM Workshop in San Francisco too meet with Max Klein and Anthony Di Franco where we kicked off a very exciting project.

There’s always someone editing a Wikipedia somewhere in the world. In fact, you can see a dizzying live stream of edits. We thought that given that there are so many DOIs in Wikipedia, that live stream may contain some diamonds (DOIs are made of diamond, that’s how they can be persistent). Max and Anthony went away and came back with a demo that contains a surprising amount of DOI activity.

That demo is evolving into a concrete service, called Cocytus. It is running at Wikimedia Labs monitoring live edits as you read this.

For now we’re feeding that data into the DOI Events Collection app (which is an off-shoot of the Chronograph project). We are in the process of modifying the Lagotto code so that we can instead push those events into the DOI Event Tracking Instance.

The first DOI event we noticed was delightfully prosaic: The DOI for “The polymath project” is cited by the Wikipedia page for “Polymath Project”. Prosaic perhaps, but the authors of that paper probably want to know. Maybe they can help edit the page.

Or how about this. Someone wrote a a paper about why people edit Wikipedia and then it was cited by Wikipedia. And then the citation was removed. The plot thickens…

We’re interested in seeing how DOIs are used outside of the formal scholarly literature. What does that mean? We don’t fully know, that’s the point. We have retractions in scholarly literature (and our Crossmark metadata and service allow publishers to record that), but it’s a bit different on Wikipedia. Edit wars are fought over … well you can see for yourself.

Citations can slip in and out of articles. We saw the DOI 10.1001/archpediatrics.2011.832 deleted from “Bipolar disorder in children”. If we’d not been monitoring the live feed (we had considered analysing snapshots of the Wikipedia in bulk) we might never have seen that. This is part of what non-traditional citations means, and it wasn’t obvious until we’d seen it.

You can see this activity on the Chronograph’s stream. Or check your favourite DOI. Please be aware that we’re only collecting newly added citations as of today. We do intend to go back and back-fill, but that may take some time- as it * cough * requires polling again.

Some Technical Things

A few interesting things that happened as a result of all this:

Secure URLs

SSL and HTTPS were invented so you could do things like banking on the web without fear of interception or tampering. As the web becomes a more important part of life, many sites are upgrading from HTTP to HTTPS, the secure version. This is not only because your confidential details may be tampered with, but because certain governments might not like you reading certain materials.

Because of this, some time ago, Wikipedia decided to embark on an upgrade to HTTPS last year, and they are a certain way along the path. The IDF, who are responsible for running the DOI system, upgraded to HTTPS this Summer, although most DOIs are referred to by HTTP still.

We met with Dario Taraborelli at the ALM workshop and discussed the DOI referral data that is fed into the Chronograph. We put two and two together and realised that Wikipedia was linking to DOIs (which are mostly HTTP) from pages which might be served over HTTPS. New policies in HTML5 specify that referrer URL headers shouldn’t be sent from HTTPS to HTTP (in case there was something secret in them). The upshot of this is, if someone’s browsing Wikipedia via HTTPS and click on a normal DOI, we won’t know that the user came from Wikipedia. Not a huge problem today, but as Wikipedia switches over to entirely secure, we’re going to miss out on very useful information.

Fortunately, the HTML5 specification includes a way to fix this (without leaking sensitive information). We discussed this with Dario, and he did some research, and came up with a suggestion, which got discussed. It’s fascinating to watch a democratic process like this take place and take part in it.

We’re waiting to see how the discussion turns out, and hope that it all works out so we can continue to report on how amazing Wikipedia is at sending people to scholarly literature.

How shall I cite thee?

Another discussion grew out of that process, and we started talking to a Wikipedian called Nemo (note to Latin scholars: we weren’t just talking to ourselves). Nemo (real name Federico Leva) had a few suggestions of his own. Another way to solve the referrer problem is by using HTTPS URLs (HTML5 allows browsers to send the referrer domain when going from HTTPS to HTTPS).

This means going back to all the articles that use DOIs and change them from HTTP to HTTPS. Not as simple as it sounds, and it doesn’t sound simple. We started looking into how DOIs were cited on Wikipedia.

After some research we found that there are more ways that we expected to cite DOIs.

First, there’s the URL. You can see it in action in this article. URLs can take various forms.

Second there’s the official template tag, seen in action here:

<ref name="SCI-20140731">{{cite journal |title=Sustained miniaturization and anatomical innovation in the dinosaurian ancestors of birds |url=http://www.sciencemag.org.pluma.sjfc.edu/content/345/6196/562 |date=1 August 2014 |journal=[[Science (journal)|Science]] |volume=345 |issue=6196 |pages=562–566 |doi=10.1126/science.1252243 |accessdate=2 August 2014 |last1=Lee |first1=Michael S. Y. |first2=Andrea|last2=Cau |first3=Darren|last3=Naish|first4=Gareth J.|last4=Dyke}}</ref>

There’s a DOI in there somewhere. This is the best way to cite DOIs, firstly as it’s actually a proper traditional citation and there’s nothing magic about DOIs, secondly because it’s a template tag and can be re-rendered to look slightly different if needed.

Third there’s the old official DOI template tag that’s now discouraged:

<ref name="Example2006">{{Cite doi|10.1146/annurev.earth.33.092203.122621}}</ref>

And then there’s another one.

{{doi|10.5555/123456789}}

Knowing all this helps us find DOIs. But if we want to convert DOIs links in Wikipedia to use HTTPS, it means that there are more template tags to modify and more pages to re-render.

Nemo also put DOIs on the Interwiki Map which should make automatically changing some of the URLs a lot easier.

We’re very grateful to Nemo for his suggestions and work on this. We’ll report back!

The elephant in the room

Those of you who know how DOIs work will have spotted an unsecured elephant in the room. When you visit a DOI, you visit the URL, which hits the DOI resolver proxy server, which returns a message to your browser to redirect to the landing page on the publisher’s site.

Securely talking to the DOI resolver by using HTTPS instead of HTTP means that no-one can eavesdrop and see which DOI you are visiting, or tamper with the result and send you off to a different page. But the page you are sent to will be, in nearly all cases, still HTTP. Upgrading infrastructure isn’t trivial, and, with over 4000 members (mostly publishers), most Crossref DOIs will still redirect to standard HTTP pages for the foreseeable future.

You can keep as secure as possible by using HTTPS Everywhere.

Fin

There’s lots going on, watch this space to see developments. Thanks for reading this, and all the links. We’d love to know what you think.

Bootnote

Not long after this blog post was published we saw something very interesting.

That’s no DOI. We like interesting things, but they can panic us. This turned out to be a great example of why this kind of thing can be useful. A minute’s digging and we found the article edit:

It turns out that this was a typo: someone put a title when they should have put in a DOI. And, as the event shows, this was removed from the Wikipedia article.

Crossref’s DOI Event Tracker Pilot

Geoffrey Bilder — Mon, 02 Mar 2015 00:00:00 +0000

TL;DR

Crossref’s “DOI Event Tracker Pilot”- 11 million+ DOIs & 64 million+ events. You can play with it at: http://goo.gl/OxImJa

Tracking DOI Events

So have you been wondering what we’ve been doing since we posted about the experiments we were conducting using PLOS’s open source ALM code? A lot, it turns out. About a week after our post, we were contacted by a group of our members from OASPA who expressed an interest in working with the system. Apparently they were all about to conduct similar experiments using the ALM code, and they thought that it might be more efficient and interesting if they did so together using our installation. Yippee. Publishers working together. That’s what we’re all about.

So we convened the interested parties and had a meeting to discuss what problems they were trying to solve and how Crossref might be able to help them. That early meeting came to a consensus on a number of issues:

The group was interested in exploring the role Crossref could play in providing an open, common infrastructure to track activities around DOIs, they were not interested in having Crossref play a role in the value-add services of reporting on an interpreting the meaning of said activities.
The working group needed representatives from multiple stakeholders in the industry. Not just open access publishers from OASPA, but from subscription based publishers, funders, researchers and third party service providers as well.
That it was desirable to conduct a pilot to see if the proposed approach was both technically feasible and financially sustainable.

And so after that meeting, the “experiment” graduated to becoming a “pilot.” This Crossref pilot is based on the premise that the infrastructure involved in tracking common information about “DOI events” can be usefully separated from the value-added services of analysing and presenting these events in the form of qualitative indicators. There are many forms of events and interactions which may be of interest. Service providers will wish to analyse, aggregate and present those in a range of different ways depending on the customer and their problem. The capture of the underlying events can be kept separate from those services.

In order to ensure that the Crossref pilot is not mistaken for some sub rosa attempt to establish new metrics for evaluating scholarly output, we also decided eschew any moniker that includes the word “metrics” or synonyms. So the “ALM Experiment” is dead. Long live the “”DOI Event Tracker” (DET) pilot. Similarly PLOS’s open source “ALM software” has been resurrected under the name “Lagotto.”

The Technical Issues

Crossref members are interested in knowing about “events” relating to the DOIs that identify their content. But our members face a now-classic problem. There are a large number of sources for scholarly publications (3k+ Crossref members) and that list is still growing. Similarly, there are an unbounded number of potential sources for usage information. For example:

Supplemental and grey literature (e.g. data, software, working papers)
Orthogonal professional literature (e.g. patents, legal documents, governmental/NGO/IGO reports, consultation reports, professional trade literature).
Scholarly tools (e.g. citation management systems, text and data mining applications).
Secondary outlets for scholarly literature (institutional and disciplinary repositories, A&I services).
Mainstream media (e.g. BBC, New York Times).
Social media (e.g. Wikipedia, Twitter, Facebook, Blogs, Yo).

Finally, there is a broad and growing audience of stakeholders who are interested in seeing how the literature is being used. The audience includes publishers themselves as well as funders, researchers, institutions, policy makers and citizens.

Publishers (or other stakeholders) could conceivably each choose to run their own system to collect this information and redistribute it to interested parties. Or they can work with a vendor to do the same. But either case, they would face the following problems:

The N sources will change. New ones will emerge. Old ones will vanish.
The N audiences will change. New ones will emerge. Old ones will vanish.
Each publisher/vendor will need to deal with N source’s different APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the publishers/vendors and for the sources.
Each audience will need to deal with N publisher/vendor APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the audiences and for the publishers.
If publishers/vendors use different systems which in turn look at different sources, it will be difficult to compare or audit results across publishers/vendors.
If a journal moves from one publisher to another, then how are the metrics for that journal’s articles going to follow the journal?

And then there is the simple issue of scale. Most parties will be interested in comparing the data that they collect for their own content, with data about their competitors. Hence, if they all run their own system, they will each be querying much more than their own data. If, for example, just the commercial third-party providers were interested in collecting data covering the formal scholarly literature, they would each find themselves querying the same sources for the same 80 million DOIs. To put this into perspective, to refresh the data for 10 million DOIs once a month, would require sources to support ~ 14K API calls an hour. 60 million DOIs would require 100K API calls an hour. Current standard API caps for many of the sources that people are interested in querying hover around 2K per hour. We may see these sources lift that cap for exceptional cases, but they are unlikely to do so for many different clients all of whom are querying essentially the same thing.

These issues typify the “multiple bilateral relationships” problem that Crossref was founded to try and ameliorate. When we have many organisations trying to access the exact same APIs to process the exact same data (albeit to different ends), then it seems likely that Crossref could help make the process more efficient.

Piloting A Proposed Solution

The Crossref DET pilot aims to show the feasibility of providing a hub for the collection, storage and propagation of DOI events from multiple sources to multiple audiences.

Data Collection

Pull: DET will collect DOI event data from sources that are of common interest to the membership, but which are unlikely to make special efforts to accommodate the scholarly communications industry. Examples of this class of source include large, broadly popular services like FaceBook, Twitter, VK, Sina Weibo, etc.
Push: DET will allow sources to send DOI event data directly to Crossref in one of three ways:
- Standard Linkback: Using standards that are widely used on the web. This will automatically enable linkback-aware systems like WordPress, Moveable Type, etc. to alert DET to DOI events.
- Scholarly Linkback: A to-be-defined augmented linkback-style API which will be optimized to work with scholarly resources and which will allow for more sophisticated payloads including other identifiers (e.g. ORCIDs, FundRefs), metadata, provenance information and authorization information. This system could be used by tools designed for scholarly communications. So, for example, it could be used by publisher platforms to distribute events related to downloads or comments within their discussion forums. It could also be used by third party scholarly apps like Zotero, Mendeley, Papers, Authorea, IRUS-UK, etc. in order to alert interested parties in events related to specific DOIs.
- Redirect: DET will also be able to serve as a service discovery layer that will allow sources to push DOI event data directly to an appropriate publisher-controlled endpoint using the above scholarly linkback mechanism. This can be used by sources like repositories in order to send sensitive usage data directly to the relevant publishers.

Data Propagation

Parties may want to use the DET in order to propagate information about DOI events. The system will support two broad data propagation patterns:

one-to-many: DOI events that are commonly harvested (pulled) by the DET system from a single source will be distributed freely to anybody who queries the DET API. Similarly, sources that push DOI events via the standard or scholarly linkback mechanisms, will also propagate their DOI events openly to anybody who queries the DET API. DOI events that are propagated in either of these cases will be kept and logged by the DET system along with appropriate provenance information. This will be the most common, default propagation model for the DET system.
one-to-one: Sources of DOI events can also report (push) DOI event data directly to owner of the relevant DOI if the DOI owner provides & registers a suitable end-point with the DET system. In these cases, data sources seeking to report information relating to a DOI, will be redirected (with a suitable 30X HTTP status and relevant headers) to the end-point specified by the DOI owner. The DET system will not keep the request or provenance information. One-to-one propagation model is designed to handle use cases where the source of the DOI event has put restrictions on the data and will only share the DOI events with the owner (registrant) of the DOI. This use case may be used, for example, by aggregators or A&I services that want to report confidential data directly back to a publisher. The advantage of the redirect mechanism is that Crossref is not put into the position of having to secure sensitive data as said data will never reside on Crossref systems.

Note that the two patterns can be combined. So, for example, a publisher might want to have public social media events reported to the DET and propagated accordingly, but to also to private third parties report confidential information directly to the publisher.

So Where Are We?

So to start with, the DET Working Group has grown substantially since the early days and we have representatives from a wide variety of stakeholders. The group includes:

Cameron Neylon, PLOS
Chris Shillum, Elsevier
Dom Mitchell, Co-action Publishing
Euan Adie, Altmetric
Jennifer Lin, PLOS
Juan Pablo Alperin, PKP
Kevin Dolby, Wellcome Trust
Liz Ferguson, Wiley
Maciej Rymarz, Mendeley
Mark Patterson, eLife
Martin Fenner, PLOS
Mike Thelwell, U Wolverhampton
Rachel Craven, BMC
Richard O’Beirne, OUP
Ruth Ivimey-Cook, eLife
Victoria Rao, Elsevier

As well as the usual contingent of Crossref cat-herders including: Geoffrey Bilder, Rachael Lammey & Joe Wass.

When we announced the then-DET experiment, we said that one of the biggest challenges would be to create something that scaled to industry levels. At launch, we only loaded in about 317,500+ Crossref DOIs representing publications from 2014 and we could see the system was going to struggle. Since then Martin Fenner and Jennifer Lin at PLOS have been focusing on making sure that the Lagotto code scales appropriately and now it is currently humming along with just over 11.5 million DOIs for which we’ve gathered over 64 million “events.” We aren’t worried about scalability on that front any more.

We’ve also shown that third parties should be able to access the API to provide value added reporting and metrics. As a demonstration of this, PLOS configured a copy of its reporting software “Parascope” to point at the Crossref DET instance. The next step we’re taking is to start testing the “push” API mechanism and the “point-to-point redirect” API mechanism. For the push API, we should have a really exciting demo available to show within the next few days. And on the point-to-point redirect, we have a sub-group exploring how the point-to-point redirect mechanism could potentially be used for reporting COUNTER stats as a compliment to the Sushi initiative.

The other major outstanding task we have before us is to calculate what the costs will be of running the DET system as a production service. In this case we expect to have some pretty accurate data to go on as we will have had close to half a year of running the pilot with a non-trivial number of DOIs and sources. Note that the work group is concerned to ensure that the underlying data from the system remains open to all. Keeping this raw data open as seen as critical to establishing trust in the metrics and reporting systems that third parties build on the data. The group has also committed to leaving the creation of value-add services to third parties. As such we have been focusing on exploring business models based around service-level-agreement backed versions of the API to complement the free version of the same API. The free API will come with no guarantees of uptime, performance characteristics or support. For those users that depend on the API in order to deliver their services, we will offer paid-for SLA-backed versions of the free APIs. We can then configure our systems so that we can independently scale these SLA-backed APIs in order to meet SLA agreements.

Our goal is to have these calculations complete in time for the working group to make a recommendation to the Crossref board meeting in July 2015.

Until then, we’ll use CrossTech as a venue for notifying people when we’ve hit new milestones or added new capabilities to the DET Pilot system.

Introducing the Crossref Labs DOI Chronograph

Joe Wass — Mon, 12 Jan 2015 00:00:00 +0000

tl;dr http://chronograph.labs.crossref.org.pluma.sjfc.edu

At Crossref we mint DOIs for publications and send them out into the world, but we like to hear how they’re getting on out there. Obviously, DOIs are used heavily within the formal scholarly literature and for citations, but they’re increasingly being used outside of formal publications in places we didn’t expect. With our DOI Event Tracking / ALM pilot project we’re collecting information about how DOIs are mentioned on the open web to try and build a picture about new methods of citation.

As part of the preparation for collaborating with Wikipedia, we looked at our statistics about when DOIs are clicked and discovered that Wikipedia was, over a two year period from 2012, the eighth largest referrer of DOIs. This means that not only does Wikipedia have a lot of DOIs, but people click them too. This bit of one-off data analysis (which surprised us) gave us enough of a prod to kickstart our collaboration with Wikipedia.

At the ALM Workshop 2014 in San Francisco we talked to some Wikipedians and bibliometricians and realised that we were sitting on a really interesting data-set and that it would be churlish not to share it. At the hackathon (read the report here) we started work on a service to gather information about DOIs and, a month later, we’re ready to unveil the DOI Chronograph.

Show me the goods

You can see:

Daily referrals (clicks) from top level domains, e.g. Wikipedia.org: http://chronograph.labs.crossref.org.pluma.sjfc.edu/domain.html?domain=wikipedia.org

Daily referrals from specific subdomains, e.g. fr.wikipedia.org: http://chronograph.labs.crossref.org.pluma.sjfc.edu/domain.html?domain=fr.wikipedia.org

Daily resolutions per DOI: http://chronograph.labs.crossref.org.pluma.sjfc.edu/doi.html?doi=10.1787%2F20752288

And, the chart that kicked this all off: DOI referring domains league tables. This shows that Wikipedia is the 3rd or 4th non-traditional referrer of DOIs (i.e. excluding referrals from Publishers’ domains): http://chronograph.labs.crossref.org.pluma.sjfc.edu/top.html

Try it out

Visit the Chronograph and give it a try chronograph.labs.crossref.org on your favourite DOI (everyone has one).

More data

Talking to a bibliometrician we also realised we can correlate other data for DOIs. We’re getting the issue date (approximately the publication date) from our own metadata, as well as the date that the Crossref metadata was updated. This gives interesting results, like the resolutions for 10.1038/ncomms2953, which peak after publication and then tails off. We are attempting to collect the following information:

daily resolution counts
day on which resolution was first successful
day on which it’s possible to resolve the DOI (we’ve got a bot running for new publications)
day on which the publisher says the article was published
day on which the metadata was most recently deposited with us
day on which the metadata was first deposited with us

We’re not there yet, but we’ve made a start and we’ve already got some pretty interesting data!

Weasel words

It’s a labs project so the usual weasel words apply. Specifically, we currently have the logs for 2012 to 2014 (we’re working at digging out the rest), and the referral information for 50 million DOIs (out of 71 million). That number will be higher by the time you read this. If your page is slow to load, be patient, as it’s currently working hard crunching numbers.

This project is focused on exploring the use of DOIs outside of the formal literature. As such, we are only looking at referrals from domains that do not appear to belong to primary publishers (i.e. our members). If you try a domain and it doesn’t work, it could be that the domain belongs to one of our members. If you’ve notice any mistakes, please email us at labs@crossref.org .

Finally, these numbers contain all DOI resolutions. That’s human clicks but also content negotiation to retrieve metadata, robots etc. We might try to filter them in future, but for now be aware that not every visitor is a human.

I’ll detail some of the the technical stuff (it’s very interesting) and what happened next with Wikipedia in a future post. Watch this space.

Citation needed

Geoffrey Bilder — Thu, 07 Aug 2014 00:00:00 +0000

Remember when I said that the Wikipedia was the 8th largest referrer of DOI links to published research? This despite only a fraction of eligible references in the free encyclopaedia using DOIs.

We aim to fix that. Crossref and Wikimedia are launching a new initiative to better integrate scholarly literature in the world’s largest public knowledge space, Wikipedia.

This work will help promote standard links to scholarly references within Wikipedia, which persist over time by ensuring consistent use of DOIs and other citation identifiers in Wikipedia references. Crossref will support the development and maintenance of Wikipedia’s citation tools on Wikipedia. This work will include bug fixes and performance improvements for existing tools, extending the tools to enable Wikipedia contributors to more easily look up and insert DOIs, and providing a “linkback” mechanism that alerts relevant parties when a persistent identifier is used in a Wikipedia reference.

In addition, Crossref is creating the role of Wikimedia Ambassador (modeled after Wikimedian-in-Residence) to act as liaison with the Wikimedia community, promote use of scholarly references on Wikipedia, and educate about DOIs and other scholarly identifiers (ORCIDs, PubMed IDs, DataCite DOIs, etc) across Wikimedia projects.

Starting today, Crossref will be working with Daniel Mietchen to coordinate Crossref’s Wikimedia-related activities. Daniel’s team will be composed of Max Klein and Matt Senate, who will work to enhance Wikimedia citation tools, and will share the role of Wikipedia ambassador with Dorothy Howard.

Since the beginnings of Wikipedia, Daniel Mietchen has worked to integrate scholarly content into Wikimedia projects. He is part of an impressive community of active Wikipedians and developers who have worked extensively on linking Wikipedia articles to the formal literature and other scholarly resources. We’ve been talking to him about this project for nearly a year, and are happy to finally get it off the ground.

-G

]7 Matt, Max and Daniel at #wikimania2014. Photo by Dorothy.

wikimania2014

Many Metrics. Such Data. Wow.

Geoffrey Bilder — Mon, 24 Feb 2014 00:00:00 +0000

[

Crossref Labs loves to be the last to jump on an internet trend, so what better than than to combine the Doge meme with altmetrics?

Note: The API calls below have been superceeded with the development of the Event Data project. See the latest API documentation for equivalent functionality

Want to know how many times a Crossref DOI is cited by the Wikipedia?

http://det.labs.crossref.org.pluma.sjfc.edu/works/doi/10.1371/journal.pone.0086859

Or how many times one has been mentioned in Europe PubMed Central?

http://det.labs.crossref.org.pluma.sjfc.edu/works/doi/10.1016/j.neuropsychologia.2013.10.021

Or DataCite?

http://det.labs.crossref.org.pluma.sjfc.edu/works/doi/10.1111/jeb.12289

Background

Back in 2011 PLOS released its awesome ALM system as open source software (OSS). At Crossref Labs, we thought it might be interesting to see what would happen if we ran our own instance of the system and loaded it up with a few Crossref DOIs. So we did. And the code fell over. Oops. Somehow it didn’t like dealing with 10 million DOIs. Funny that.

But the beauty of OSS is that we were able to work with PLOS to scale the code to handle our volume of data. Crossref contracted with Cottage Labs and we both worked with PLOS to make changes to the system. These eventually got fed back into the main ALM source on Github. Now everybody benefits from our work. Yay for OSS.

So if you want to know technical details, skip to Details for Propellerheads. But if you want to know why we did this, and what we plan to do with it, read on.

Why?

There are (cough) some problems in our industry that we can best solve with shared infrastructure. When publishers first put scholarly content online, they used to make bilateral reference linking agreements. These agreements allowed them to link citations using each other’s proprietary reference linking APIs. But this system didn’t scale. It was too time-consuming to negotiate all the agreements needed to link to other publishers. And linking through many proprietary citation APIs was too complex and too fragile. So the industry founded Crossref to create a common, cross-publisher citation linking API. Crossref has since obviated the need for bilateral linking arrangements.

So-called altmetrics look like they might have similar characteristics. You have ~4000 Crossref member publishers and N sources (e.g. Twitter, Mendeley, Facebook, CiteULike, etc.) where people use (e.g. discuss, bookmark, annotate, etc.) scholarly publications. Publishers could conceivably each choose to run their own system to collect this information. But if they did, they would face the following problems:

The N sources will be volatile. New ones will emerge. Old ones will vanish.
Each publisher will need to deal with each source’s different APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the publishers and for the sources.
If publishers use different systems which in turn look at different sources, it will be difficult to compare results across publishers.
If a journal moves from one publisher to another, then how are the metrics for that journal’s articles going to follow the journal? This isn’t a complete list, but it shows that there might be some virtue in publishers sharing an infrastructure for collecting this data. But what about commercial providers? Couldn’t they provide these ALM services? Of course - and some of them currently do. But normally they look on the actual collection of this data as a means to an end. The real value they provide is in the analysis, reporting and tools that they build on top of the data. Crossref has no interest in building front-ends to this data. If there is a role for us to play here, it is simply in the collection and distribution of the data.

No, really, WHY?

Aren’t these altmetrics an ill-conceived and meretricious idea? By providing this kind of information, isn’t Crossref just encouraging feckless, neoliberal university administrators to hasten academia’s slide into a Stakhanovite dystopia? Can’t these systems be gamed?

FOR THE LOVE OF FSM, WHY IS CROSSREF DABBLING IN SOMETHING OF SUCH QUESTIONABLE VALUE?

takes deep breath. wipes spittle from beard

These are all serious concerns. Goodhart’s Law and all that… If a university’s appointments and promotion committee is largely swayed by Impact Factor, it won’t improve a thing if they substitute or supplement Impact Factor with altmetrics. Amy Brand has repeatedly pointed out, the best institutions simply don’t use metrics this way at all (PowerPoint presentation). They know better.

But yes, it is still likely that some powerful people will come to lazy conclusions based on altmetrics. And following that, other lazy, unscrupulous and opportunistic people will attempt to game said metrics. We may even see an industry emerge to exploit this mess and provide the scholarly equivalent of SEO. Feh. Now I’m depressed and I need a drink.

So again, why is Crossref doing this? Though we have our doubts about how effective altmetrics will be in evaluating the quality of content, we do believe that they are a useful tool for understanding how scholarly content is used and interpreted. The most eloquent arguments against altmetrics for measuring quality, inadvertently make the case for altmetrics as a tool for monitoring attention.

Critics of altmetrics point out that much of the attention that research receives outside of formal scholarly communications channels can be ascribed to:

Puffery. Researchers and/or university/publisher “PR wonks” over-promoting research results.
Innocent misinterpretation. A lay audience simply doesn’t understand the research results.
Deliberate misinterpretation. Ideologues misrepresent research results to support their agendas.
Salaciousness. The research appears to be about sex, drugs, crime, video games or other popular bogeymen.
Neurobollocks. A category unto itself these days.

In short, scholarly research might be misinterpreted. Shock horror. Ban all metrics. Whew. That won’t happen again.

Scholarly research has always been discussed outside of formal scholarly venues. Both by scholars themselves and by interested laity. Sometimes these discussions advance the scientific cause. Sometimes they undermine it. The University of Utah didn’t depend on widespread Internet access or social networks to promote yet-to-be peer-reviewed claims about cold fusion. That was just old-fashioned analogue puffery. And the Internet played no role in the Laetrile or DMSO crazes of the 1980s. You see, there were once these things called “newspapers.” And another thing called “television.” And a sophisticated meatspace-based social network called a “town square.”

But there are critical differences between then and now. As citizens get more access to the scholarly literature, it is far more likely that research is going to be discussed outside of formal scholarly venues. Now we can build tools to help researchers track these discussions. Now researchers can, if they need to, engage in the conversations as well. One would think that conscientious researchers would see it as their responsibility to remain engaged, to know how their research is being used. And especially to know when it is being misused.

That isn’t to say that we expect researchers will welcome this task. We are no Pollyannas. Researchers are already famously overstretched. They barely have time to keep up with the formally published literature. It seems cruel to expect them to keep up with the firehose of the Internet as well.

Which gets us back to the value of altmetrics tools. Our hope is that, as altmetrics tools evolve, they will provide publishers and researchers with an efficient mechanism for monitoring the use of their content in non-traditional venues. Just in the way that citations were used before they were distorted into proxies for credit and kudos.

We don’t think altmetrics are there yet. Partly because some parties are still tantalized by the prospect of usurping one metric for another. But mostly because the entire field is still nascent. People don’t yet know how the information can be combined and used effectively. So we still make naive assumptions such as “link=like” and “more=better.” Surely it will eventually occur to somebody that, instead, there may be a connection between repeated headline-grabbing research and academic fraud. A neuroscientist might be interested in a tool that alerts them if the MRI scans in their research paper are being misinterpreted on the web to promote neurobollocks. An immunologist may want to know if their research is being misused by the anti-vaccination movement. Perhaps the real value in gathering this data will be seen when somebody builds tools to help researchers DETECT puffery, social-citation cabals, and misinterpretation of research results?

But Crossref won’t be building those tools. What we might be able to do is help others overcome another hurdle that blocks the development of more sophisticated tools; getting hold of the needed data in the first place. This is why we are dabbling in altmetrics.

Wikipedia is already the 8th largest referrer of Crossref DOIs. Note that this doesn’t just mean that the Wikipedia cites lots of Crossref DOIs, it means that people actually click on and follow those DOIs to the scholarly literature. As scholarly communication transcends traditional outlets and as the audience for scholarly research broadens, we think that it will be more important for publishers and researcher to be aware of how their research is being discussed and used. They may even need to engage more with non-scholarly audiences. In order to do this, they need to be aware of the conversations. Crossref is providing this experimental data source in the hope that we can spur the development of more sophisticated tools for detecting and analyzing these conversations. Thankfully, this is an inexpensive experiment to conduct - largely thanks to the decision on the part of PLOS to open source its ALM code.

What Now?

Crossref’s instance of PLOS’s ALM code is an experiment. We mentioned that we had encountered scalability problems and that we had resolved some of them. But there are still big scalability issues to address. For example, assuming a response time of 1 second, if we wanted to poll the English-language version of the Wikipedia to see what had cited each of the 65 million DOIs held in Crossref, the process would take years to complete. But this is how the system is designed to work at the moment. It polls various source APIs to see if a particular DOI is “mentioned”. Parallelizing the queries might reduce the amount of time it takes to poll the Wikipedia, but it doesn’t reduce the work. Another obvious way in which we could improve the scalability of the system is to add a push mechanism to supplement the pull mechanism. Instead of going out and polling the Wikipedia 65 million times, we could establish a “scholarly linkback” mechanism that would allow third parties to alert us when DOIs and other scholarly identifiers are referenced (e.g. cited, bookmarked, shared). If the Wikipedia used this, then even in an extreme case scenario (i.e. everything in Wikipedia cites at least one Crossref DOI), this would mean that we would only need to process ~ 4 million trackbacks.

The other significant advantage of adding a push API is that it would take the burden off of Crossref to know what sources we want to poll. At the moment, if a new source comes online, we’d need to know about it and build a custom plugin to poll their data. This needlessly disadvantages new tools and services as it means that their data will not be gathered until they are big enough for us to pay attention to. If the service in question addresses a niche of the scholarly ecosystem, they may never become big enough. But if we allow sources to push data to us using a common infrastructure, then new sources do not need to wait for us to take notice before they can participate in the system.

Supporting (potentially) many new sources will raise another technical issue- tracking and maintaining the provenance of the data that we gather. The current ALM system does a pretty good job of keeping data, but if we ever want third parties to be able to rely on the system, we probably need to extend the provenance information so that the data is cheaply and easily auditable.

Perhaps the most important thing we want to learn from running this experimental ALM instance is: what it would take to run the system as a production service? What technical resources would it require? How could they be supported? And from this we hope to gain enough information to decide whether the service is worth running and, if so, by whom. Crossref is just one of several organisations that could run such a service, but it is not clear if it would be the best one. We hope that as we work with PLOS, our members and the rest of the scholarly community, we’ll get a better idea of how such a service should be governed and sustained.

Details for Propellerheads

Warning, Caveats and Weasel Words

The Crossref ALM instance is a Crossref Labs project. It is running on R&D equipment in a non-production environment administered by an orangutang on a diet of Redbulls and vodka.

So what is working?

The system has been initially loaded with 317,500+ Crossref DOIs representing publications from 2014. We will load more DOIs in reverse chronological order until we get bored or until the system falls over again.

We have activated the following sources:

PubMed

DataCite

PubMedCentral Europe Citations and Usage

We have data from the following sources but will need some work to achieve stability:

Facebook

Wikipedia

CiteULike

Twitter

Some of them are faster than others. Some are more temperamental than others. WordPress, for example, seems to go into a sulk and shut itself off after approximately 1,300 API calls.

In any case, we will be monitoring and tweaking the sources as we gather data. We will also add new sources as we get requested API keys. We will probably even create one or two new sources ourselves. Watch this blog and we’ll update you as we add/tweak sources.

Dammit, shut up already and tell me how to query stuff.

You can login to the Crossref ALM instance simply using a Mozilla Persona (yes, we’d eventually like to support ORCID too). Once logged-in, your account page will list an API key. Using the API key, you can do things like:

http://det.labs.crossref.org.pluma.sjfc.edu/api/v5/articles?ids=10.1038/nature12990

And you will see that (as of this writing), said Nature article has been cited by the Wikipedia article here:

https://en.wikipedia.org/wiki/HE0107-5240#cite_ref-Keller2014_4-0;

PLOS has provided lovely detailed instructions for using the API- So, please, play with the API and see what you make of it. On our side we will be looking at how we can improve performance and expand coverage. We don’t promise much- the logistics here are formidable. As we said above, once you start working with millions of documents, the polling process starts to hit API walls quickly. But that is all part of the experiment. We appreciate your helping us and would like your feedback. We can be contacted at:

Introductory Signals

Geoffrey Bilder — Mon, 23 Mar 2009 00:00:00 +0000

So while doing some background reading today I realized that legal citations already widely support a form of “citation typing” in the form of “Introductory Signals“. The 10 introductory signals break down as follows…

In support of an argument:

1) [no signal]. (NB that, apparently, this is increasingly deprecated.)

2) accord;

3) see;

4) see also;

5) cf.;

For Comparisons:

6) compare … with …;

For contradiction:

7) but see;

8) but cf.;

For background:

9) see generally;

And for examples:

10) e.g.

Clever lawyers.

Citing Data Sets

Tony Hammond — Fri, 30 Mar 2007 00:00:00 +0000

This D-Lib paper by Altman and King looks interesting: “A Proposed Standard for the Scholarly Citation of Quantitative Data”. (And thanks to Herbert Van de Sompel for drawing attention to the paper.) Gist of it (Sect. 3) is

_“We propose that citations to numerical data include, at a minimum, six required components. The first three components are traditional, directly paralleling print documents. … Thus, we add three components using modern technology, each of which is designed to persist even when the technology changes: a unique global identifier, a universal numeric fingerprint, and a bridge service. They are also designed to take advantage of the digital form of quantitative data.

An example of a complete citation, using this minimal version of the proposed standards, is as follows:

**Micah Altman; Karin MacDonald; Michael P. McDonald, 2005, “Computer Use in Redistricting”,

hdl:1902.1/AMXGCNKCLU UNF:3:J0PkMygLPfIyT1E/8xO/EA==

http://id.thedata.org/hdl%3A1902.1%2FAMXGCNKCLU

“_

So the abbreviated citation (author, date, title, unique ID) is supplemented by a UNF which fingerprints the data. UNFs would appear to be a sort of super MD5 in providing a signature of the data content independent of the data serialization to a filestore.

_“Thus, we add as the fifth component a Universal Numeric Fingerprint or UNF. The UNF is a short, fixed-length string of numbers and characters that summarize all the content in the data set, such that a change in any part of the data would produce a completely different UNF. A UNF works by first translating the data into a canonical form with fixed degrees of numerical precision and then applies a cryptographic hash function to produce the short string. The advantage of canonicalization is that UNFs (but not raw hash functions) are format-independent: they keep the same value even if the data set is moved between software programs, file storage systems, compression schemes, operating systems, or hardware platforms.

…

Finally, since most web browsers do not currently recognize global unique identifiers directly (i.e., without typing them into a web form), we add as the sixth and final component of the citation standard a bridge service, which is designed to make this task easier in the medium term.”_

Certainly looks promising. I’m not sure if there’s any other contestants in this arena.

Citation on Crossref

Two billion citation links in Crossref help research travel further

Strengthening support for data citations and saying goodbye to Event Data

Time to say goodbye

Visibility for data citations

On metadata enrichment

Metadata available through Crossref

Metadata enrichment layers

Layer 1: Member updates

Layer 2: Community feedback loop

Layer 3: Metadata matching

Layer 4: Third-party datasets

Layer 5: Unstructured content scraping

Summary

Start citing data now. Not later

TL;DR

Background

So who is already citing data?

Top 10 members depositing data citations from November-May 2022

Top 10 data citations per deposited work

Summaries are great, but I want to see some actual examples!

Find all the DOIs that cite Dataset X (identified by DOI)

Find all data citations from Crossref member X (identified by member prefix)

Find papers with supplementary data

Find all data citations to Crossref member X

Find all data citations to DataCite member X

Service Provider perspectives: A few minutes with our publisher hosting platforms

What is the biggest change you’ve experienced working with publisher metadata over the last few years and how have you adapted to it?

How do you explain to clients (and others!) why correct, quality metadata is important?

What is the one industry development or trend you’re most excited about for the near future and why?

Event Data: A Plan of Action

The Plan

Stability

Consolidation

Future roadmap

Staying in touch

Fast, citable feedback: Peer reviews for preprints and other record types

Reviews for preprints

EASE Council Post: Rachael Lammey on the Research Nexus

Events got the better of us

Why so quiet?

So, where are we today?

What are we doing?

Opening up the discussion

Crossref metadata for bibliometrics

Using the Crossref REST API (with Open Ukrainian Citation Index)

Introducing OUCI

How is Crossref metadata used in OUCI?

What are the future plans for OUCI?

What else would OUCI like to see in Crossref metadata?

Questions?

Metadata Corrections, Updates, and Additions in Metadata Manager

Correcting Errors

For articles originally registered using Metadata Manager

For articles registered using the Web Deposit Form or any other deposit method

Adding additional metadata

What's your (citations') style?

TL;DR

Introduction

Data

Styles

Features

Learning algorithm

Final accuracy estimation

What if I told you that bibliographic references can be structured?

TL;DR

Introduction

References in Crossref

The analysis

The results

Structured vs. unstructured

What’s next?

Underreporting of matched references in Crossref metadata

TL;DR

The gory details

Improved processes, and more via Metadata Manager

Login first, not last

Easily import journals, or add new ones

Adding a Journal DOI

Adding new articles