Event Data on Crossref

Strengthening support for data citations and saying goodbye to Event Data

Martyn Rittman — Tue, 24 Mar 2026 00:00:00 +0000

We’re excited to announce a new data citation API endpoint and are seeking your feedback. The new service makes existing data citation relationships in our metadata available, thereby surfacing this part of the research nexus. At the same time, we’ve decided that it’s time to move on from Event Data.

Time to say goodbye

Metadata about published scholarly research has evolved, and continues to evolve in exciting ways. A published article, book, or conference paper is only a single piece of the puzzle. A host of digital identifiers and items can be put together to form a more complete picture of a research project. This is what forms the basis of the research nexus—a rich and reusable open network of relationships connecting research organisations, people, things, and actions; a scholarly record that the global community can build on forever, for the benefit of society.

Ten years ago, we launched Event Data to surface mentions of research around the Internet. What were people saying about published research? Could this discourse contribute to post-publication review and validation? We set up Event Data to capture use of Crossref DOIs in the online world from a variety of sources, including blog posts, social media, websites, and annotations. The idea was that diverse mentions (or “events”) could supplement traditional citation counts as a way to capture the value of research.

Today, the focus is increasingly on transparency, research integrity, and the completeness of outputs. Trust in research is shaped by knowing who the funder was, being able to reanalyse the original data, or checking for bugs in the analysis code. There are also more identifiers for objects within the research space and they are used more widely. This shift is evident in the relatively low usage of Event Data. We can no longer justify the resources and cost that goes into maintaining it as a service. Instead, we will focus on enabling and surfacing relationships between different types of research outputs, starting with links to datasets.

For these reasons, we have decided to sunset the Event Data API and from 23 April 2026 it will no longer be available (although historical data will still be available). In its place, we’re making available an API endpoint for data citations.

Visibility for data citations

The new API endpoint focuses solely on data citations and uses references and relationships deposited by Crossref members, including Crossref articles referencing datasets with either Crossref or DataCite DOIs. While our metadata contains many data citations, and some of them are labelled as data citations, it is often difficult to find them because they are swamped in number by other citations. If you are trying to get data citations directly from our REST API, it’s like looking for a needle in a haystack. This new endpoint makes connections easier to find, enabling organisations to track when research is reused, cited, and built upon. By putting this metadata into a dedicated service, we are making it easier for interested organisations to track and find data citations. Our goal is to make existing sets of connections easier to access, giving clarity to how scholarly works link to the data that supports them.

This beta version will allow us the opportunity to incorporate feedback from the community and make changes to improve delivery. We received early positive feedback from a number of interested organisations. Later this year we will assess whether it is ready for production, needs more work, or if insufficient interest from the community suggests we should pursue a different solution.

Anyone interested in data citations is invited to try the new endpoint. Please let us know your feedback via the community forum. You can find supporting documentation on our website and Swagger documentation, including the opportunity to try out features.

Edited 15 June 2026: link to historical Event Data added.

Event Data now with added references

Martyn Rittman — Wed, 10 Nov 2021 00:00:00 +0000

Event Data is our service to capture online mentions of Crossref records. We monitor data archives, Wikipedia, social media, blogs, news, and other sources. Our main focus has been on gathering data from external sources, however we know that there is a great deal of Crossref metadata that can be made available as events. Earlier this year we started adding relationship metadata, and over the last few months we have been working on bringing in citations between records.

Our members deposit references alongside other metadata, and we have a lot of them. In fact, we have over 1.2 billion, with hundreds of thousands of new references added each day. While our metadata APIs make it easy to see which works are cited, it is much more difficult to find a list of citations to a specific work. We can make this easier by presenting citations as events in Event Data. Now that the huge majority of our members have responded positively to the Initiative for Open Citations (I4OC) campaign and Crossref’s open-by-default reference policy, the move to make this data available via Event Data is a natural step.

A bumpy ride, but we got there

Adding such a large amount of data means a significant increase in the data coming into Event Data, which has presented some challenges. We’ve known for some time that Event Data is not very stable, but we expected it to cope with the new data coming in. We have mitigated by initially only looking at new data, not trying to immediately back-fill with old references. Unfortunately, even with this limitation it hasn’t been a smooth ride, and our first effort to put references into Event Data uncovered bugs we didn’t know about and we had to walk back the changes. We tried again and found that we were hitting rate limits for our own APIs. This is a sure sign of technical debt: we shouldn’t need to be shifting large amounts of our own data from one place to another, and not at rates that could be putting stress on APIs used by others in the community.

We have managed to work around these problems and I’m pleased to say that we are now adding metadata from reference lists to Event Data. They can be accessed via the Event Data API: https://api-eventdata-crossref-org.pluma.sjfc.edu/v1/events?rows=10&source=crossref&relation-type=references&from-collected-date=2021-10-01

Where to next?

There remains work to be done. We would like to backfill references, and there is also further work to include relationships to objects that have identifiers other than Crossref records (genes, proteins, ArXiv identifiers, and so on). Our work on investigating sources is proceeding and we will be looking to add more next year. While possible, these steps will be costly and time-consuming if we proceed without significant changes to the infrastructure supporting Event Data.

When we started Event Data the volumes of data were much smaller and our infrastructure coped well, but as we’ve said here before, it’s in need of an overhaul. In fact, our recent experience and some other considerations are making us look at some very fundamental changes in how we record events.

We are therefore working on a new data model that will allow events to be stored alongside the rest of our metadata. This work is still in the early stages, but if we are successful it will mean that we won’t need to move data between databases. It will also make it easier to provide access to all of our reference metadata along with other relationships that we’re not currently able to provide, and give us the capacity to add new data sources.

Open references

[EDIT 6th June 2022 - all references are now open by default with the March 2022 board vote to remove any restrictions on reference distribution].

It is worth noting that only open references will be available via Event Data. This covers 88% of works with references at present. Members have the option to deposit references with limited visibility, meaning only Metadata Plus users can access them; or closed visibility, meaning that only the member who owns the cited work can retrieve the citation, via Cited-by.

We encourage our members to make their references open and deposit them as metadata. It makes them usable downstream by thousands of tools that researchers use. Including open references also improves the quality of metadata, and there are reciprocal benefits for the large number of members who openly share their reference data: they contribute to a large, openly available pool of data with many applications that advance research, and drives usage of the content published by our members.

If you are a Crossref member and unsure whether your reference metadata is open or not, check your participation report. This will tell you the percentage of your records with deposited references, and the percentage of those that are open. You can change the reference visibility preference for each DOI prefix that you own by contacting our support team. For guidance on how to deposit references, see our user documentation.

Event Data: Help us fill in the gaps

Martyn Rittman — Fri, 11 Jun 2021 00:00:00 +0000

UPDATE August 2, 2021: This work was awarded to Laura Paglione of the Spherical Cow Group.

To date, we have collected around 740 million events from 12 different source since we launched our Event Data service service in 2017. Each event is an online mention of the research associated with a DOI, either via the DOI directly or using the associated URL. However, we know that there is much more out there. Because of this, we would like to explore where we could expand.

We invite proposals to conduct a gap analysis for Event Data sources, looking at what we currently collect and seeing what more could be added. For the most relevant new sources, we are seeking an estimate of the effort to include them, and establish whether it is possible: we know that there are sources that are paywalled or with restrictive licensing not compatible with Event Data.

The aim of the project is to identify a list of potential new sources. With community input, we will look to add a number of these to Event Data in the future based on needs and priorities.

For full details of the requirements and how to make a proposal, see here. The deadline for proposals is 11 July 2021 and we anticipate that the work will be completed by the end of October 2021.

Service Provider perspectives: A few minutes with our publisher hosting platforms

Jennifer Kemp — Mon, 24 May 2021 00:00:00 +0000

Service Providers work on behalf of our members by creating, registering, querying and/or displaying metadata. We rely on this group to support our schema as it evolves, to roll out new and updated services to members and to work closely with us on a variety of matters of mutual interest. Many of our Service Providers have been with us since the early days of Crossref. Others have joined as scholarly communications has grown and services have evolved. Though fewer than 20 in number, their impact far outweighs the size of the group.

They, like us, work with a great variety of members and have a broad view into publishing trends. In this post, we focus on views from some of the publishing hosting platform Service Providers, who’ve taken the time to share their thoughts on a few questions:

What is the biggest change you’ve experienced working with publisher metadata over the last few years and how have you adapted to it?

It has become more and more important that not only the DOIs are registered with the minimum of necessary metadata to get the DOIs registered, but that a most complete set of metadata is being sent along – including author identifiers, funding information, abstracts, licenses, to support other Crossref services and improve discoverability.

– de Gruyter

Our clients are increasingly aware of the key role metadata plays in the effective dissemination of research. With an increasing number of published articles and a clear domination of “search engines” and aggregation of content, metadata is the primary means of making sure that publications reach the right audience. Publishers’ value-add includes not just copy editing, formatting, and packaging, but also now creating journal articles for the digital age that are discoverable and well linked to the research corpus. Furthermore, we sense a clear move toward standardization, which goes beyond the structure to introduce standardized semantics: adopting common taxonomies for classifying content in different dimensions. Our response is to introduce effective, automated and consistent services that capture, and surface metadata throughout the value chain from authoring to publication and search.

– Atypon

Highwire’s publishers are always looking to use the latest DTD (Document Type Definition) for the content to stay up to current standards. Currently this would be JATS 1.2. They are choosing to remain current so that they can stay on top of all or new metadata that can enrich their deposits. We have handled this well and offer support for the latest version of DTD when they are released, but some publishers are not always familiar with what can/should be deposited with their content and this can be a learning process for them.

– MPS Limited

How do you explain to clients (and others!) why correct, quality metadata is important?

In the digital age, metadata is the key to enabling effective content consumption. Publications that cannot be effectively discovered are of little value. We can only increase the impact of research with “discoverable” and “machine readable” publications. So ensuring correct and quality metadata is the key to optimizing not only the processing (finding the right journal, editor, reviewers) but also to positioning each publication properly. As the volume of published scientific research increases, article metadata is the way forward — it brings “order” and enables our community to manage this volume.

– Atypon

Highwire always positions itself as “good content in” means “good content out”. This is true for our own content stores. Strong and valid metadata will result in valid and strong deposits. We explain this to all new clients on-boarded with Highwire and the use of current standards and for current client projects where content should/can be enriched through re-load.

– MPS Limited

Getting our journals to care about metadata is a two step process: First, make sure they understand how metadata will help their journal succeed (i.e. why it matters to them). Second, make it easy for them to produce metadata while minimizing the cost, time, or complexity of their workflow. The first step – making a case for why metadata matters – is often easier than you’d think. At the very least, most journal editors understand that metadata, e.g., JATS or DOI registration, is an important signifier of professionalism / prestige. In other words, they see that top journals publish metadata and want the same for their journal. From a more technical standpoint, metadata is important because that’s the format computers understand and, like it or not, the publishing ecosystem relies on computers to deliver all sorts of critical services – such as indexing, archiving, and discoverability. So, if you’re not publishing metadata, you’re likely missing the benefit of these services. The second step – making it easy to produce metadata – is more difficult. Journal editors generally understand metadata matters but often lack the technical skills or resources necessary to create metadata. This is where a platform, such as Scholastica, can be very helpful. Because platforms work with many journals, they can invest in tools to automate the creation of metadata, reducing costs for all their clients. For example, most platforms offer integrations to support automatic DOI registration. At Scholastica, we’re pushing this idea even further with automatic integration to more complicated services such as PubMed Central. By reducing cost and complexity, we can help new or small-budget journals have the same quality metadata normally reserved for large, established journals.

– Scholastica

We are sending other publishers’ metadata to academic libraries and distribution channels. Erroneous metadata will have a direct impact on how discoverable a title may be. The more uniform and correct the metadata, the better it will be indexed in other places.

– de Gruyter

What is the one industry development or trend you’re most excited about for the near future and why?

Open Science and the ability to deliver research with the tools for reproducing it is the most exciting and game changing trend. Technology has enabled the output of science to transition from two-dimensional printed text delivery into globally accessible and responsive web-based delivery. We are now taking the next steps to further leverage web technology to enhance research output with rich assets ranging from audio and video, datasets, executable code, high-resolution imagery, interactive applications and more. As more assets accompany research publications, viewing these assets as modular, individually citable, and reusable becomes a requirement. We are reviewing the whole research output flow from authoring to publishing, and most importantly to its dissemination through the myriad of discovery tools now available.

– Atypon

The move of everything to the cloud – this is changing and improving our infrastructure, our possibility to scale and to stay on top of technological development.

– de Gruyter

Thanks very much to the interviewees for their time and thoughts. We look forward to working with our entire Service Provider group on questions like these and many more. If you’d like more details, you can read about our Service Provider program or contact me for more information.

Doing more with relationships - via Event Data

Martyn Rittman — Fri, 14 May 2021 00:00:00 +0000

Crossref aims to link research together, making related items more findable, increasing transparency, and showing how ideas spread and develop. There are a number of moving parts in this effort: some related to capturing and storing linking information, others to making it available.

By including relationship metadata in Event Data, we are taking a big step to improve the visibility of a large number of links between metadata. We know this is long-promised and we’re pleased that making this valuable metadata available supports a number of important initiatives. We will also be backfilling, so all previously deposited relationships will eventually become available as events. The first step will be to add relationships between items that have DOIs, such as between a research article and a related review report or dataset.

What are relationships?

When members register metadata with us, they have the possibility to identify other works, items, and websites that they know are related. This might be supplementary material or previous versions of a work (especially for preprints and working papers). Equally, identifiers for a protein, gene, or organism used in the research can be included. These are recorded as ‘relationships’ and can be accessed in the same way as the rest of the metadata we hold about registered content.

Some examples

Relationships in the metadata show links to the published article from this bioRxiv preprint. In the Crossref Rest API:

"relation": {
 "is-preprint-of": [
 {
 "id-type": "doi",
 "id": "10.1038/s41467-020-17892-0",
 "asserted-by": "subject"
 }
 ],
 "cites": []
},

And now in Event Data:

"subj": {
 "pid": "https://doi-org.pluma.sjfc.edu/10.1101/2020.05.21.109546",
 "url": "https://doi-org.pluma.sjfc.edu/10.1101/2020.05.21.109546",
 "work_type_id": "posted-content"
},
"obj": {
 "pid": "https://doi-org.pluma.sjfc.edu/10.1038/s41467-020-17892-0",
 "url": "https://doi-org.pluma.sjfc.edu/10.1038/s41467-020-17892-0",
 "method": "doi-literal",
 "verification": "literal",
 "work-type-id": "journal-article"
},

Linking to a dataset in the Dryad Digital Repository by a recent eLife article. In the Crossref metadata:

"relation": {
 "is-supplemented-by": [
 {
 "id-type": "doi",
 "id": "10.5061/dryad.s58qh",
 "asserted-by": "subject"
 }
 ],
 "references": [
 {
 "id-type": "doi",
 "id": "10.5061/dryad.s58qh",
 "asserted-by": "subject"
 }
 ],
 "cites": []
},

And now in Event Data:

"subj": {
 "pid": "https://doi-org.pluma.sjfc.edu/10.7554/elife.19920",
 "url": "https://doi-org.pluma.sjfc.edu/10.7554/elife.19920",
 "work_type_id": "journal-article"
},
"obj": {
 "pid": "https://doi-org.pluma.sjfc.edu/10.5061/dryad.s58qh",
 "url": "https://doi-org.pluma.sjfc.edu/10.5061/dryad.s58qh",
 "method": "doi-literal",
 "verification": "literal",
 "work-type-id": "Dataset"
},

If you are interested in relationships for a single DOI, we still recommend checking the metadata of that record, however Event Data is a great option for looking across multiple records. For example, to check for relationships across a prefix, in a given time period, or for a specific type of relationship.

Data citation

Data citations can be included in data deposits in relationship metadata, usually using the ‘is-supplemented-by’ relationship. By creating an event from each relationship, the links between journal articles and books, and the data they rely on are more visible. This makes the data much easier to locate.

Many datasets have DOIs which are usually recorded with DataCite, meaning you are unlikely to find them via searches of Crossref metadata. Making data citation relationship metadata available in Event Data means it will be available in the same format as citations from datasets to articles (which DataCite sends to Event Data) and citations from articles to datasets from Crossref reference metadata (more to come on this later this year). It also means we will convert this information into Scholix format so that it can be harvested and combined with other sets of Scholix-compliant article/data links. Data citations will therefore be available for the community to identify, share, link and recognise research data. We’re working with initiatives like Make Data Count and STM’s research data program to support the growing uptake of good data citation practices. This is a big step forward in making data citation happen for the community; we have more to do, but Crossref is committed to completing this work as a strategic priority.

What’s next?

In this first stage we are adding relationships that link two objects with a DOI, and later this year we will bring in relationships using other identifiers such as accession numbers and URIs. That will make it more straightforward to ask questions of Event Data such as which organisms have relationships to which works with a DOI.

More info and staying in touch

Find out more about Event Data in our support documentation or check out tickets in the GitLab repo.
Keep informed and ask us anything via our community forum for Event Data discussion

Event Data: A Plan of Action

Martyn Rittman — Mon, 01 Feb 2021 00:00:00 +0000

Event Data uncovers links between Crossref-registered DOIs and diverse places where they are mentioned across the internet. Whereas a citation links one research article to another, events are a way to create links to locations such as news articles, data sets, Wikipedia entries, and social media mentions. We’ve collected events for several years and make them openly available via an API for anyone to access, as well as creating open logs of how we found each event. Some organisations are already using Event Data and we are keen for more to come on board.

Last year we gave an update on Event Data with apologies for being so quiet and a promise of more information at a later date. It’s been some time, so here goes…

I joined Crossref in the middle of last year as a Product Manager and was tasked with looking into Event Data. The first thing I found was a large amount of enthusiasm for Event Data, both within Crossref and further afield. The idea of gathering information beyond the metadata deposited by our members is popular, and creates valuable connections between DOIs and a range of other sources. Interest spans the spectrum of academic research, publishing, bibliometrics, and beyond.

At the same time, I found a project with a very solid, well-built code base but unstable performance. After being put into production in 2018, we didn’t provide sufficient support. Coupled with staff changes and other competing priorities, Event Data hasn’t had the opportunity to live up to early expectations.

To address these issues, we have embarked on a plan to make the server infrastructure more robust, improve monitoring, and make sure that the future of Event Data makes the best use of the resources we have without over-stretching. It means working with the community to determine the most essential aspects of Event Data, and providing support where it’s needed.

The steps below are not necessarily sequential and some depend on the completion of work in other parts of Crossref, but they outline the priorities we have for Event Data in 2021.

The Plan

Stability

Since we put in place our original Event Data infrastructure, the amount of incoming data has grown, and at an ever-increasing rate. In 2017 we were creating 2 million new events per month, that number is now over 20 million. We have known for some time that we need to refresh the infrastructure, but didn’t have the resources to move forward: now we do.

In the first part of the plan we will renew the server infrastructure that underpins Event Data. Maybe not a headline-grabbing move, but the aim is to reduce downtime and pull in missing data. Through improving our monitoring and shortening the response time when things go wrong, we will be able to ensure that events are added on a regular basis and the API can reliably handle requests.

We’ve made the first steps in this direction by upgrading our API infrastructure and making some other tweaks to improve performance. There is still work to do, but we’ve already seen a significant improvement in performance with nearly >99.99% uptime in December.

Consolidation

The second component of the plan is to review performance and data quality. We will evaluate the event sources, update artefacts (such as the lists of publisher landing pages and news websites, and review performance reporting. This will help us to have a better understanding of Event Data in its current form: if the stability component is about improving what comes in and goes and out, this part will give us increased confidence in what Event Data already contains.

Future roadmap

While the two steps above are being carried out, we will revisit the applications of Event Data and talk to organisations that currently use it or have expressed an interest. These conversations will feed into future development in which we will evaluate new sources and other ways to optimize the service.

Central to the roadmap will be continued support of the data citation endpoint in Scholix format, which we run in close collaboration with DataCite. Additionally, we will add new data from relationships between Crossref works, for example a preprint is matched to a journal article, or where there are corrections, retractions, or translations of works.

We expect to continue supporting the current sources of events and where there are organisations with either a strong interest in a particular source or a database of events that they can send directly, we are keen to build collaborations. Event Data, like everything that Crossref does, is a community-based effort.

Staying in touch

To join the conversation about Event Data and keep informed, head over to our Community pages. You can also check out our Gitlab pages. At the end of last year we updated the Education pages where you can learn more about Event Data.

Fast, citable feedback: Peer reviews for preprints and other record types

Martyn Rittman — Wed, 09 Dec 2020 00:00:00 +0000

Crossref has supported depositing metadata for preprints since 2016 and peer reviews since 2018. Now we are putting the two together, in fact we will permit peer reviews to be registered for any record type.

Currently, peer reviews can be registered for journal articles, but that means that they can only be related to some of the content our members deposit. Preprints, books, chapters, working papers, dissertations, and a host of other works can also be registered with Crossref. A number of these frequently undergo some form of review and many of our members and voices in the community have called for us to widen the net on peer reviews, including journal publishers, book publishers, review platforms, and preprint servers. We’ve listened and taken action, and from now on Crossref members can add relationship metadata that links peer reviews to any record type. The metadata will also contain the type of review, stating whether it is a referee report, author response, or community comment, etc. This allows accurate reporting on whether the peer review is happening within a traditional editorial process or elsewhere.

Reviews for preprints

In the last decade there has been an increase in the number of disciplines using preprints. Since enabling registration of preprint metadata, it has become our fastest-growing record type. Preprints, working papers, and other forms of early publication help to accelerate dissemination of the latest research and discovery. They can also promote discussion on important topics, and help authors to improve papers before an editorial decision for journal publication. During the COVID-19 pandemic, preprints have become invaluable for speeding the publication of vital research and case studies.

On the other hand, preprints do not undergo formal review and editorial approval, leading to concerns about the dissemination of false information. While the issue of misinformation in preprints has been discussed for some time, the COVID-19 pandemic has brought it more sharply into focus. organisations that post preprints need to balance the benefits of rapid dissemination with promoting their responsible use.

To support the feedback process, preprint servers along with a growing number of other platforms and services offer scholars the opportunity to post public comments on preprints. By doing this, they give extra context for readers, provide suggestions for authors, and raise awareness of work that could be flawed or too preliminary.

Another growing trend is journal publishers adopting editorial processes that involve preprint-first options and open peer review. As Dr. Stephanie Dawson from ScienceOpen says:

“We have long believed in rewarding reviewers by assigning Crossref DOIs to their open reviews to make them citable objects and we were one of the first users of Crossref’s peer review schema. However, a large percentage of the articles reviewed on ScienceOpen are publicly available preprints. The UCL Open: Environment journal hosted on the platform, for example, is based on a workflow of open peer review of preprints. Our customers, editors, reviewers and authors are therefore extremely happy that these reviews can now also be assigned a Crossref peer review DOI for more accountability and transparency in scholarly publishing.”

At Crossref, we’re continually looking to support more record types and relations between them to build trust, support reproducibility and increase discoverability of content. This is another small step in building the research nexus and we look forward to working with members depositing peer reviews of preprints.

Events got the better of us

Bryan Vickery — Fri, 27 Mar 2020 00:00:00 +0000

Publisher metadata is one side of the story surrounding research outputs, but conversations, connections and activities that build further around scholarly research, takes place all over the web. We built Event Data to capture, record and make available these ‘Events’ –– providing open, transparent, and traceable information about the provenance and context of every Event. Events are comments, links, shares, bookmarks, references, etc.

In September 2018 we said Event Data was ‘production ready.’ What we meant was development of the service had reached a point where we expected no further major changes to the code, and we encouraged you to use it. What normally would have followed was a detailed handover to our operations team, for monitoring and performance management, and for Product Management to expand Event Data by adding new Crossref member domains and evaluating additional event sources.

Why so quiet?

But many things changed on the staff front, meaning 2019 was a year of reinvention for the Technical and Product teams and of critical knowledge sharing and learning –– Event Data had to take a back seat as we focused resources on other key projects (more on that later). From a technical perspective, we’ve found the Elasticsearch index is not performing well and the approach taken to specifically support data citations through Scholix has not really scaled.

When things go wrong, whether in ways you can or can’t anticipate, the most important thing is communication –– in dealing with the challenges we forgot to do that. We understand how frustrating that can be and we’re extremely sorry to have gone so quiet.

So, where are we today?

Event Data is important to us and clearly important to you too as you’ve contacted us about your use-cases and the reliability of the service. Event Data remains available and you’re welcome to use it, but you should expect instability to continue and be aware that it does not find events for DOIs/domains of our newer members (who joined Crossref since 2019) –– so we’re conscious it might be hard to say whether it’s a good fit for your project at this point.

What are we doing?

We have brought in additional expert Elasticsearch resources to assist with a separate project to migrate our REST API from SOLR to Elasticsearch. We’re making fantastic progress on this. As soon as we’re confident we can make this switch, we will move those same Elasticsearch resources to shoring up Event Data. The REST API takes priority over Event Data because we need to add support for important new record types (like research grants) that aren’t yet available via the API.

We’re also concluding the process of hiring two new Product Managers which means we’ll be in a position to assign someone to head up the product management of Event Data. When we do return to Event Data in the coming months, our initial priority will be increased support for data citation and Scholix. If that means radical changes to the rest of the service, we’ll let you know.

Opening up the discussion

We will have more news on Event Data in mid-2020. We’d love you to join the Crossref Community Forum; we’ve created a new Category for Event Data where you can post details of how you are using, or plan to use Event Data; post questions to the group; suggestions for future development and provide general feedback on the Event Data service.

What's that DOI?

Joe Wass — Mon, 21 Jan 2019 00:00:00 +0000

This is a long overdue followup to 2016’s “URLs and DOIs: a complicated relationship”. Like that post, this accompanies my talk at PIDapalooza, the festival of open persistent identifiers). I don’t think I need to give a spoiler warning when I tell you that it’s still complicated. But this post presents some vocabulary to describe exactly how complicated it is. Event Data has been up and running and collecting data for a couple of years now, but this post describes changes we made toward the end of 2018.

If Event Data is new to you, you can read about its development in other blog posts and the User Guide. Today I’ll be describing a specific but important part of the machinery: how we match landing pages to DOIs.

Some background

Our Event Data service provides you with a live database of links to DOIs, found from across the web and social media. Data comes from a variety of places, and most of it is produced by Agents operated by Crossref. We have Agents monitoring Twitter, Wikipedia, Reddit, Stack Overflow, blogs and more besides. It is a sad truth that the good news of DOIs has not reached all corners of world, let alone the dustiest vertices of the world wide web. And even within scholarly publishing and academia, not everyone has heard of DOIs and other persistent identifiers.

Of course, this means that when we look for links to content-that-has-DOIs, what we at Crossref call ‘registered content’, we can’t content ourselves with only looking for DOIs. We also have to look for article landing pages. These are the pages you arrive at when you click on a DOI, the page you’re on when you decide to share an article.

Half full or half empty?

So we’re trying to track down links to these landing pages, rather than just DOIs. You could look at this two ways.

The glass-half-empty view would be that it’s a real shame people don’t use DOIs. Don’t they know that their links aren’t future-proof? Don’t they know that DOIs allow you to punch the identifier into other services?

The glass-half-full view is that it’s really exciting that people outside the traditional open identifier crowd are interacting with the literature. We’ve been set a challenge to try and track this usage. By collecting this data and processing it into a form that’s compatible with other services we can add to its value and better help join the dots in and around the community that we serve. Not everyone tweeting about articles counts as ‘scholarly Twitter’, and hopefully we can bridge some divides (the subject of my talk at PIDapalooza last year, 'Bridging Identifiers').

How do we do it?

One of the central tenets of Event Data is transparency. We record as much information as we can about the data we ingest, how we process it, and what we find. Of course, you don’t have to use this data, it’s up to you how much depth you want to go into. But it’s there if you want it.

The resulting data set in Event Data is easy to use, but allows you to peek beneath the surface. We do this by linking every Event that our Agents collect through to an Evidence Record. This in turn links to Artifacts, which describe our working data set.

One such Artifact is the humbly named domain-decision-structure. This is a big tree that records DOI prefixes, domain names, and how they’re connected. It includes information such as “some DOIs with the prefix 10.31139 redirect to the domain polishorthopaedics.pl, and we can confirm that pages on that domain correctly represent their DOI”. We produce this list by visiting a sample of DOIs from every known prefix. We then ask the following questions:

Which webpage does this DOI redirect to, and what domain name does it have?
Does the webpage include its correct DOI in the HTML metadata?

From this we build the Artifact that records prefix → domain relationships, along with a flag to say whether or not the domain correctly represents its DOI in at least one case. You can put this data to a number of uses, but we use it to help inform our URL to DOI matching.

What Agents do

The Agents use the domain list to search for links. For example, the Reddit Agent uses it to query for new discussions about websites on each domain. They then pass this data to the Percolator, which is the machinery that produces Events.

The Percolator takes each input, whether it’s a blog post or a Tweet, and extracts links. If it finds a DOI link, that’s low hanging fruit. It then looks for links to URLs on one of the domains in the list. All of these are considered to be candidate landing page URLs. Once it has found a set of candidate links in the webpage it then has to find which ones correspond to DOIs, and validate that correspondence.

For each candidate URL it follows the link and retrieves the webpage. It looks in the HTML metadata, specifically in the <meta name='dc.identifier' content='10.5555/12345678' >, to see if the article indicates its DOI. It also looks in the webpage to see if it reports its DOI in the body text.

Not so fast

But can you trust the web page to indicate its own DOI? What about sites that say that they have a DOI belonging to another member? What about those pages that have invalid or incorrect DOIs? These situations can, and do, occur.

We have the following methods at our disposal, in order of preference.

doi-literal - This is the most reliable, and it indicates that the URL we found in the webpage was a DOI not a landing page. We didn’t even have to visit the article page.
pii - The input was a PII (Publisher Item Identifier). We used our own metadata to map this into a DOI.
landing-page-url - We thought that the URL was the landing page for an article. Some webpages actually contain the DOI embedded in URL. So we don’t even have to visit the page.
landing-page-meta-tag - We had to visit the article landing page. We found a meta tag, eg. dc.identifier, indicating the DOI.
landing-page-page-text - We visited the webpage but there was no meta tag. We did find a DOI in the body text and we think this is the DOI for this page. This is the least reliable.

On top of this, we have a number of steps of validation. Again, these are listed in order of preference.

literal - We found a DOI literal, so we didn’t have to do any extra work. This is the most reliable.
lookup - We looked up the PII in our own metadata, and we trust that.
checked-url-exact - We visited the landing page and found a DOI. We visited that DOI and confirmed that it does indeed lead back to this landing page. We are therefore confident that this is the correct DOI for the landing page URL.
checked-url-basic - We visited the DOI and it led back to almost the same URL. The protocol (http vs https), query parameters or upper / lower case may be different. This can happen if tracking parameters are automatically added by the website meaning the URLs are no longer identical. We are still quite confident in the match.
confirmed-domain-prefix - We were unable to check the link between the DOI and the landing page URL, so we had to fall back to previously observed data. On previous occasions we have seen that DOIs with the given prefix (e.g. “10.5555”) redirect to webpages with the same domain (e.g. “www.example.com”) and those websites correctly report their DOIs in meta tags. Only the domain and DOI prefix are considered. We therefore believe that the domain reliably reports its own DOIs correctly in at least some cases.
recognised-domain-prefix - On previous occasions we have seen that DOIs with the given prefix (e.g. “10.5555”) redirect to webpages with the same domain (e.g. “www.example.com”). Those websites do not always correctly report their DOIs in meta tags. This is slightly less reliable.
recognised-domain - On previous occasions we have seen that this domain is associated with DOIs in general. This is the least reliable.

We record the method we used to find the DOI, and the way we verified it, right in the Event. Look in the obj.method and obj.verification fields.

Of course, there’s a flowchart.

You can take a closer look in the User Guide.

If you think that’s a bit long-winded, well, you’re right. But it does enable us to capture DOI links without giving a false sense of security.

So, what happens?

If you ask the Event Data Query API for the top ten domains that we matched to DOIs in the first 20 days of January 2019, it would tell you:

Domain	Number of Events captured
`doi.org`	2058433
`dx.doi.org`	242707
`www.nature.com`	170808
`adsabs.harvard.edu`	163387
`www.sciencedirect.com`	96849
`onlinelibrary.wiley.com`	88760
`link.springer.com`	63869
`www.tandfonline.com`	41911
`www.sciencemag.org`	39489
`academic.oup.com`	39267

Here we see a healthy showing for actual DOIs (which you can explain by Wikipedia’s excellent use of DOIs) followed by some of the larger publishers. This demonstrates that we’re capturing a healthy number of Events from Wikipedia pages, tweets, blog posts etc that reference landing pages.

Awkward questions

This is not a perfect process. The whole point of PIDs is to unambiguously identify content. When users don’t use PIDs, there will inevitably be imperfections. But because we collect and make available all the processing along the way, hopefully we can go back to the old data, or allow any researchers to try and squeeze more information out of the data.

Q: Why bother with all of this? Can’t you just use the URLs?

We care about persistent identifiers. They are stable identifiers, which means they don’t change over time. The same DOI will always refer to the same content. In contrast, publishers’ landing pages can and do change their URLs over time. If we didn’t use the DOIs then our data would suffer from link-rot.

DOIs are also compatible across different services. You can use the DOI for an article to look it up in metadata and citation databases, and to make connections with other services.

This is not the only solution to the problem. Other services out there, such as Cobalt Metrics, do record the URLs and store an overlaid data set of identifier mappings. At Crossref we have a specific focus on our members and their content, and we all subscribe to the value of persistent identifiers for their content.

Of course, we don’t throw anything away. The URLs are still included in the Events. Look in the obj.url field.

Q: If DOIs are so amazing why keep URLs?

Event Data is useful to a really wide range of users. Some will need DOIs to work with the data. But others, who may want to research the stuff under the hood, such as the behaviour of social media users, or the processes we employ, may want to know more detail. So we include it all.

Q: Can’t you just decide for me?

In a way, we do. If an Event is included in our data set, we are reasonably confident that it belongs there. All we are doing is providing you with more information.

Q: Why only DOIs?

We specialise in DOIs and believe they are the right solution for unambiguously and persistently identifying content. Furthermore the content registered with Crossref has been done so for the specific benefits that DOIs bring.

Q: What about websites that require cookies and/or JavaScript to execute?

Some sites don’t work unless you allow your browser to accept cookies. Some sites don’t render any content unless you allow their JavaScript to execute. Large crawlers, like Google, emulate web browsers when they scrape content, but it’s resource-intensive and not everyone has the resources of Google!

This is an issue we’ve known about for a while. My talk two years ago was about precisely this topic. We know it’s a hurdle we’ll have to overcome at some point. We do have plans to look into it, but we haven’t found a sufficiently cost-effective and reliable way to do it yet.

Any sites that do do this will be inherently less reliable, so we recommend everyone to put their Dublin Core Identifiers in the HTML, render your HTML server-side (which is the default way of doing things) and don’t require cookies.

Q: What’s the success rate?

This is an interesting question. The results aren’t black and white. At the low end of the confidence spectrum we do have a cut-off point, at which we don’t generate an Event. But when we do create one we qualify it by describing the method we used to match and verify the connection. What level of confidence you want to trust is for you to decide. We just describe the steps we took to verify it.

It’s tricky quantifying false negatives. We have plenty of unmatched links, but not every unmatched link even could be matched to a DOI, for example there are some domains that have some DOI-registered content mixed with non-registered content.

We therefore err on the side of optimism, and let users choose what level of verification they require.

So talking of false positives or false negatives is a complicated question. We’ve not done any analytical work on this yet, but would welcome any input from the community.

Q: Why isn’t the `domain-decision-structure` Artifact more detailed?

We looked into various ways of constructing this, including more detailed statistics. At the end of the day our processes have to be understandable and easy to re-use. The process already takes a flow-chart to understand, and we felt that we got the balance right. Of course, as a user of this data, you are welcome to further refine and verify it.

Event Data is production ready

Christine Buske — Wed, 12 Sep 2018 00:00:00 +0000

We’ve been working on Event Data for some time now, and in the spirit of openness, much of that story has already been shared with the community. In fact, when I recently joined as Crossref’s Product Manager for Event Data, I jumped onto an already fast moving train—headed for a bright horizon.

What’s on the horizon? Well, the reality is you never really reach the horizon. Good product development—in my opinion—is like that train. You keep aiming for the horizon and passing all the stations (milestones) along the way, but the horizon keeps moving as you add features, improve the service, and maybe even review where you are headed. However, for Event Data we are pleased to say we have now arrived at a rather important station.

Technical readiness

Thank you to all the beta testers who have journeyed with us this far—we’ve listened and learned, refined and rebuilt with the help of your feedback. We are now thrilled to say that we are service production ready. We’ve reached the station called ‘technical readiness’, and are eager to see more users board our train!

During this time of building and refining, Event Data has grown to include at least 66,7 million events from sources like (in order of magnitude): Wikipedia, Cambia Lens, Twitter, Datacite, F1000, Newfeeds, Reddit links, Wordpress.com, Crossref, Reddit, Hypothesis, and Stackexchange. Wikipedia alone accounts for 50 million events (and counting).

What does this mean?

Event Data is production ready.

Being production ready means we are not going to make any breaking changes to the code, and we are excited to see more people jump on board to explore where you can go with Event Data, and what product or service you might want to build with it.

Getting started

Having a look at Event Data, and using it, is easy. While the user guide outlines everything you need to know to get fully engrossed, you can get your feet wet with a few sample queries:

Above I mentioned Event Data has about 50 million Wikipedia events, you can check if that has grown by looking at a query that lists all distinct events by source (your browser will need a JSON viewer extension):

https://api-eventdata-crossref-org.pluma.sjfc.edu/v1/events/distinct?facet=source/:*&rows=0

You can also see a live stream of events going through Event Data.

For all events registered for a specific content item, you simply query http://api.eventdata.crossref.org.pluma.sjfc.edu/v1/events?obj-id=https://doi-org.pluma.sjfc.edu/XXX, where XXX is replaced with the DOI.

What next?

We are now focusing on the final stretch towards the official roll-out. Beyond this, we will continue to add sources and features and have a healthy roadmap to keep us on track. We value any feedback you have for us about your own journey with Event Data. Your feedback may help shape the direction we take in the future. Most of all, we are all excited to see what people build with it!

We look forward to continuing on our Event Data journey and we welcome you all aboard the train! Please contact me with your ideas.

Hear this, real insight into the inner workings of Crossref

Joe Wass — Sun, 01 Apr 2018 00:00:00 +0000

You want to hear more from us. We hear you.

We’ve spent the past year building Crossref Event Data, and hope to launch very soon. Building a new piece of infrastructure from scratch has been an exciting project, and we’ve taken the opportunity to incorporate as much feedback from the community as possible. We’d like to take a moment to share some of the suggestions we had, and how we’ve acted on them.

We asked a focus group “What one thing would you change?”. In hindsight, we could have done a better job with the question. We did get some enlightening answers but—for legal and practical reasons—we are unable to end either world hunger or global conflict, or do any of the other things we were invited to do. So we went back to our focus group and asked “What one thing would you change about Crossref?”.

The answers were illuminating. Some of you wanted mundane things like more data dumps. A disappointing number of people wanted us to put the capital ‘R’ back in our name. But two things we heard consistently, loud and clear, were:

“I want to hear more from Crossref”
“I want to know more about what’s going on inside Crossref”

One respondent said:

I like the newsletters, and the Twitter visuals are nice enough, but I want to hear, you know, more from them.

Another:

Crossref is your typical quiet DOI Registration Agency. They make a big thing about being the background infrastructure you don’t notice. But infrastructure doesn’t have to be quiet. I live next to the M25, and I can tell you, that’s the sound of success. I mean, it’s loud.

One final quote which clinched it for us:

The outreach team is doing a great job with their multilingual videos. But you can never cover every world language. In today’s connected world, you should be thinking about the universal language.

She clarified:

No, I don’t mean XML.

We took this advice to heart. When we were building Crossref Event Data, we baked these features right in. Now you can hear what’s going on inside Crossref, any time, day or night.

Introducing the Crossref Thing Action Service!

Turn up your speakers (about half-way, it would be foolhardy to turn them too high) and visit:

live.eventdata.crossref.org/thing-action-service.html

It’s optimized for Google Chrome, but we’ve tested it in Firefox and Safari.

The Thing Action Service shows you, in excruciating sonorous detail, every single action that happens inside the Crossref Event Data system. Every time we receive live data from Twitter or Wikipedia. Every time we check a DOI. Every time we check an RSS feed. Every time we find a link to our Registered Content on the web.

In a pioneering move within the scholarly publishing space, you can hear the data as it’s being processed, live. Furthermore, we think we are the first DOI Registration Agency to offer our services in stereo.

John Chodacki, Professional Working Group Chair, said:

We welcome this innovation. From my experience Chairing, well, everything, I’m certain that hearing-impaired users will like it especially.

So sit back, put the Thing Action Service on the speakers, and relax. You may find it difficult at first, but as you let the sound waves wash over you, think of all that data in flight. That beep could be someone criticizing the article you wrote on Twitter. But don’t worry, the next one might be someone defending it.

Think of it as musique concrète. That’s the Art of Persistence.

Hello, meet Event Data Version 1, and new Product Manager

Christine Buske — Thu, 29 Mar 2018 00:00:00 +0000

I joined Crossref only a few weeks ago, and have happily thrown myself into the world of Event Data as the service’s new product manager. In my first week, a lot of time was spent discussing the ins and outs of Event Data. This learning process made me very much feel like you might when you’ve just bought a house, and you’re studying the blueprints while also planning the house-warming party.

If Event Data is like a house, it’s been built and we’ve recently been putting on a last coat of paint. We’re very happy to announce version 1 of the API today. This is bringing us closer to the launch (house warming party), which will officially present Event Data to the world. Further to that analogy, while I bought into the house, I wasn’t around to see it being built. That’s both incredibly exciting and a little daunting.

Version 1 contains fixes for some challenges we came up against. Like scalability, data modeling for Wikipedia, and polishing. Version 1 is a new release of the data, but it is the same data set you already know and love. It should solve some of the recent stability issues, for which we apologize.

Moving forward, we expect the data model in V1 to persist and are not planning to make further large scale, fundamental changes to the Event Data API. As such, the version 1 release of the API is exceptional and a big step forward. It is important that we address these fixes before we go into production as it affects everyone who uses the service.

Same Event Data, new address

In setting up for the upcoming production service rollout, we have updated the Event Data API domain so that it is in line with Crossref’s suite of APIs. The Query API can now be found at a new URL. Here is an example query: https://api-eventdata-crossref-org.pluma.sjfc.edu/v1/events?rows=1

We have also simplified the standard query parameters in favor of a cleaner filter syntax.

Lastly, we have added a new “Mailto” parameter, just like in our REST API. It is encouraged but optional, so you are not obliged to supply it. We’ll only use it to contact you if there’s a problem.

Changes to the Wikipedia data structure

We’ve done a lot of work to use the canonical URLs for web pages to represent content as consistently as possible. This has entailed updating previously collected Events across data sources. As such, we’ve updated our Wikipedia data model to align with this. Because this update has impacted every Wikipedia Event in the system, we recommend those who have used or saved existing data from the deprecated Query API version to pull a new copy of the data. Read more about the rationale for changing the Wikipedia data model.

Updated data

This then brings me to how we now handle updated data. Sometimes we edit Events to add new features, or we may edit Events if there is an issue processing and/or representing the data when we provision it to the community. And sometimes we must remove Events to comply with a particular data source’s terms and conditions (ex: deleted Tweets). You can read about how updates work in the user guide.

To make life easier moving forward, we’ve split updated Events into two API endpoints. If you are already using Event Data, you will need to make some small updates to your client(s) to align with this. The new endpoints are further described in the documentation.

Event Data beta group

With the version 1 release we are making solid progress towards an official launch (the house-warming party!), we are quite excited to hear how you are using Event Data. Please consider [joining our beta group] (https://groups.google.com/forum/#!forum/crossref-event-data-beta-testers), if you are using the Event Data API or want to hear about updates.

This is also where you can read about these updates in more detail.

For more information and to get started with Crossref Event Data, please refer to the user guide.

I am looking forward to seeing how Event Data is being used, and working with the community to continuously improve what we can offer through this service. Feedback is always welcome, feel free to get in touch with me at eventdata@crossref.org.

Bridging Identifiers at PIDapalooza

Joe Wass — Mon, 22 Jan 2018 00:00:00 +0000

Hello from sunny Girona! I’m heading to PIDapalooza, the Persistent Identifier festival, as it returns for its second year. It’s all about to kick off.

One of the themes this year is “bridging worlds”: how to bring together different communities and the identifiers they use. Something I really enjoyed about PIDapalooza last year was the variety of people who came. We heard about some “traditional” identifier systems (at least, it seems that way to us): DOIs for publications, DOIs for datasets, ORCIDs for researchers. But, gathered in Reykjavik, under dark Icelandic skies, I met oceanographic surveyors assigning DOIs to drilling equipment, heard stories of identifiers in Chinese milk production and consoled librarians trying navigate the identifier landscape.

In addition to the usual scholarly publishing and science communication crowd, it was encouraging to see a real diversity of people from different walks of life encounter the same problems and work on them them collaboratively. The thing that brought everyone together was the understanding that if we’re going to reliably reference things – be they researchers, articles they write, or ships they sail – we need to give them identifiers. And those identifiers should be as good as possible: persistent, resolvable, interoperable.

Who cares about PIDs?

At the turn of the century, a handful of publishers came together to create Crossref (or CrossRef as it was in those days). It was becoming increasingly important to be able to store references in machine-readable format, but publishers were faced with a problem. If an author wants to cite an article, they’ll do so without worrying who published it. This means they needed an identifier system that worked across all publishers. Thus the Crossref DOI was born.

Today we’re heading toward 10,000 members, and the thing that they have in common is that they all produce scholarly content and care about how it’s referenced. As a trade association, we effectively act on behalf of all of our members, allowing them to register their content, share metadata and links, and assign an identifier.

But there’s a whole world out there. Publications have never been the be-all and end-all of scholarship, but they have been a backbone. But more and more scholarship, especially science, is done outside journal publishing. Sometimes it’s done on platforms that care about the scholarly record as much as publishers. And sometimes it isn’t.

The Twitterverse

Lots of people use Twitter to talk about science. Some are scientists, some aren’t. Scientific articles are linked from news reports and discussed on blogs. Gone are the days of scholarly articles being cited only by other scholarly articles. We see links coming in from all over the place. And, although not all of this can be counted as the “scholarly record”, some of it could be.

The barrier-to-entry for journals publishing means that science journals contain only science articles. The barrier-to-entry for Twitter means that anyone can, and does, publish there. My Twitter feed is finely balanced between bibliometrics research, marine biology and pictures of snow leopards with Japanese captions. I don’t understand all of it, but I like looking at the pictures.

Back in the days when the only references to scholarly publications were from other scholarly publications, it was easy to keep track of those references. When an article was published, its references went into a citation database. This happened because the publisher considered this important.

But Twitter, the publisher of tweets, doesn’t care. It is used for a huge variety of communications and although some people choose to use it to engage in scholarship, we’re just a blip on their radar. The same goes for Reddit, a platform that describes itself as “the front page of the Internet”. There are communities engaged in scientific discussions, but Reddit doesn’t feel the need to publish its bibliographic references.

Nor should it.

Bridging those who care with those who don’t

The barrier-to-entry for contributing to scientific discussions has lowered, meaning that the role of more non-specialist platforms has increased.

I imagine that there are other communities out there who have their own concerns about the web. Maybe there are model train enthusiasts who want to keep track of every reference to a particular model. Or political commentators who want to keep track of how certain politicians and policies are discussed. As the scholarly community embraces new platforms for communicating, we should recognise that we are part of a broader universe of people using those platforms for more diverse reasons.

Gone are the days when the only way to reply to an article was by writing a letter to the editor. But also gone are the days when you could guarantee that your letter wouldn’t appear next to cat pictures (assuming you weren’t writing to the Journal of Feline Medicine & Surgery). As a specialist community cohabiting online spaces with non-specialists, it falls to us to do whatever we need to adapt that space and make it our own. In our case, this means recording bibliographic references as and where they occur.

Something like this happened once before. As traditional publishers went online, they created Crossref to build and maintain the necessary infrastructure. We’re acting on behalf of the community again to collect links from non-traditional sources. Because we can’t go to platforms like Twitter and say “please deposit your references”, we’re doing the opposite. We identify a platform, then work out how to scrape its content and extract links.

Working at scale

So we’re broadening out the universe of references that we would like to track from “traditional scholarly publishing” to “the entire web”. There are four broad challenges inherent in this, and we think that Crossref infrastructure is the right way to meet them.

The first challenge is physically finding the links. Because social media platforms aren’t specialised for scholarly publishing, they don’t have the same mechanisms in place for capturing bibliographic references. This means that we have to do it ourselves by scraping webpages for references. As the standard-bearer for scholarly PIDs, we think we can do a good job of this.

The second challenge is doing this at the scale of the web. Because we might, in theory, find a link on any webpage, there is a literally infinite number of publishing platforms. From big websites like BBC News down to tiny blogs run out of a bedroom. It would be impossible to partner with each of these individually. The way to solve this is to run a centralised service which goes out and contacts as many sources as possible. This role is a collaborative one. Our system is open to inspection, suggestions and contributions from the community.

The third challenge is the sheer number of publishers. Because they all register content with us, we are in good position to track their DOIs. In addition to that, every member of Crossref publishes content on their own platform, and has their own set of websites to track. We monitor our members’ websites and create a central list of domains that we look for. If this wasn’t done centrally, each publisher would have to run its own web crawlers and perform the same work, only to filter out their own links.

The fourth challenge is how to get all that data to the public. Even if every publisher were able to run their own infrastructure, it would make it very difficult to consume. Through Crossref metadata services, publishers have built a system where you can look up metadata and link to articles without worrying who published them. We think that the same approach should apply to this new link data.

For these reasons, we’re building Crossref Event Data: a system that monitors as many platforms as we can think of, and brings them into one place, and serves the whole community.

Building bridges

If you’ve been following along you’ll know that my last metaphor was the process of refining crude oil. I like metaphors, and mixing them. After all, you can’t mix a good metaphor without breaking a few eggs into the mixing bowl. Today’s metaphors are bridges. And not just one.

Bridge 1: PIDs and URLs

In the world of Persistent Identifiers, we’re quite good at linking. organisations like Crossref, DataCite and ORCID run separate systems but we work together to record and exchange links. But the web is different. There’s no single organisation in control and there are many organisations working to catalogue it. Event Data is our offering: bridging the web with our identifiers.

Bridge 2: Scholarly link providers

Of course, some platforms and systems do care about persistence and Persistent Identifiers. Event Data is an open platform, and we’re collaborating with a few providers to publish links.

We’ve partnered with The Lens to include Patent to DOI references. We’re working with F1000 to include links between reviews and articles. Hopefully we’ll see more organisations use Event Data to publish their links.

Bridge 3: Crossref / DataCite

Event Data is a collaborative project between DataCite and Crossref. When Crossref Registered Content contains a reference to a DataCite DOI we put it into Event Data. DataCite do the same in reverse. This means that Event Data contains a huge number of article - dataset links.

Bridge 4: Traditional discussions vs new ones

At each moment, scholarly discussions are happening in the literature, on various social media platforms and on the web at large. They are all talking about the same thing, but are spread out. Event Data collects links wherever we find them and brings them into one place. By doing this we hope we can help bring those conversations together.

Bridge 5: Bridging bibliometricians and altmetricians to data sources

Capturing links from social media to published literature underpins the field of altmetrics. By collecting this data and making it available under open licenses, we bring it to altmetrics researchers. We don’t provide metrics, but we do provide the data points that can form the basis for research.

Without infrastructure for collecting data, researchers would have to perform the same work over and over again. Because the data is all open, we allow datasets to be republished, reworked and replicated.

Bridge 6: Bridging the Evidence Gap

Running Event Data involves collecting a lot of data - gigabytes per day - and boiling it down into hundreds of thousands of individual Events per day. People consuming the data may want to do further boiling down. At every point of the process we record the input data that we were working from, the internal thought process of the system, and the Events that were produced. A researcher can use the Evidence Logs to trace through the entire process that led to an Event.

We’re a bridge from websites and social media to data consumers. But we take the role very seriously, and there’s nothing hidden. A glass bridge, you could say.

Interesting challenges

It’s not all plain sailing. There are a few challenges along the way to collecting this data which anyone who wanted to collect this kind of information would face. By collecting it in a central place and running an open platform we can solve each problem once, and improve our process as a community.

One problem is choosing what to include. We include any link that we find from a non-publisher website. That means that invariably some of the links are from spam. This problem isn’t new: we see low-quality articles being published in traditional journals from time to time. We try to include all of the data we can find and pass it onto consumers. They might want to whitelist certain sources, or they may want all of the data because they’re trying to study scholarly spam. We have decided to provide data as Events, which strike the balance between atomicity and usefulness.

Another, which I talked about at last year’s PIDapalooza, is how we track article landing pages. Read the blog post, the user guide or hop in a time machine if you’re interested.

The thing about bridges…

… is that they help people get where they’re going. With a few notable exceptions, they’re not the main attraction. We play a humble part in scholarly publishing, helping collect and distribute metadata. Most of what we do goes unseen, and helps people create tools, platforms and research. Event Data is an API, and whilst we hope people will build all kinds of things with it, including altmetrics tools, we’re not making another metric.

PIDapalooza

All of which brings me to my talk, which I’m giving on Wednesday: Bridging persistent and not-so-persistent identifiers. I would tell you about it, but there isn’t much more left to say.

If you want to find out more, we’re currently in Beta, and open for business. Head over to the User Guide to get started!

A transparent record of life after publication

Madeleine Watson — Wed, 01 Nov 2017 00:00:00 +0000

Crossref Event Data and the importance of understanding what lies beneath the data.

Some things in life are better left a mystery. There is an argument for opaqueness when the act of full disclosure only limits your level of enjoyment: in my case, I need a complete lack of transparency to enjoy both chicken nuggets and David Lynch films. And that works for me. But metrics are not nuggets. Because in order to consume them, you really need to know how they’re made. Knowing the provenance of data, along with the context with which it was derived, provides everyone with the best chance of creating indicators which are fit for purpose. This is just one of the reasons why we built the Event Data infrastructure with transparency in mind.

The transparency problem

For the scholarly community, alternative metrics to citation count (‘altmetrics’) are becoming increasingly popular as they can offer rich and expedited insight into today’s diverse and dynamic research environment. Research artifacts undergo an extended life online as they’re linked, shared, saved and discussed in forums both within and beyond the traditional academic ecosystem. Data on these interactions are initially fragmented and buried within platforms like social media, blogs and news sites. Downstream, there are several value-add services that collate and present that data as a single, aggregated count. We see individual data points like ‘paper X was tweeted 22 times’, and ‘paper X is referenced 16 times on Wikipedia’ being combined, homogenised, weighted and expressed as a single figure, a calculated number serving as a proxy for value. But altmetrics alone don’t tell the whole story, and how they are calculated is not without idiosyncrasy or politics. As we each have our own unique voice and perspective, we need to ensure we understand the lenses through which these metrics are made in order to consume them effectively.

The 2015 Metric Tide report highlighted transparency as one of the five dimensions of responsible metrics. Having access to the context used to create a metric — the provenance of the original data as well as full transparency around its extraction, processing and aggregation — helps consumers to use the data meaningfully and allows for comparison across third-party vendors. But transparency is difficult to achieve when, as the report notes, the systems and infrastructure for collecting and curating altmetrics-style data are fragmented and have limited interoperability.

In the academic community, underlying centralised systems include ORCIDs to identify people and DOIs to identify items. But we’re missing a transparent, centralised infrastructure for describing and recording the relationships between objects and resources¹. These relationships, or links, occur outside publisher platforms and can provide valuable information about the interconnectivity and dissemination of research. Dedicated infrastructure for collecting these relationships would provide a data source for those interested in altmetrics to build upon.

Figure 1.1 Example of some relationships between articles and activity on the web

At Crossref, we call these relationships Events. An Event is the record of a claim made about the existence of a relationship between a registered content item (i.e. a DOI) and a specific activity on the web. Events include:

a DataCite dataset DOI contains a link to a Crossref article DOI
an article was referenced in Wikipedia
an article was mentioned on Twitter
an article has a Hypothes.is annotation
a blog contains a link to an article

In collaboration with DataCite, we are collecting Events for the DOIs registered with our organisations and are making that data available for others in the community to use. This is the Event Data infrastructure, with which we’re plugging the gap in open scholarly relationships infrastructure. https://doi-org.pluma.sjfc.edu/10.64000/kattr-5k219

The Event Data infrastructure

Crossref and DataCite have for many years provided a centralised location for bibliographic metadata and links, and a facility to help our members register Persistent Identifiers (DOIs) for their content. With nearly 100 million DOIs registered with Crossref, we know where research lives. Which got us thinking — could we use these links to find out more about the journey research undertakes after publication? Could we express these interactions as links without any aggregation or counts so it could be maximally reused? And if so, could we then provide this data in an open, centralised, structured format? The answer was yes, subject to some challenges:

Querying for individual DOIs wasn’t scalable for our full corpus of 100 million items, so we had to find something else.
Not everyone uses the DOI link (not a surprise!). Most people will link directly to the publisher’s site. This means we need to look for links using both the DOI and article landing page URLs.
When we find people referring to registered content using its landing page, we find the DOI for that content item so that the link can be referenced in our data set in a stable, link-rot-proof way.
We don’t always know the article landing page URL for every DOI upfront because like many relationships, the one between DOIs and URLs is complicated.

We began by asking the wrong questions and as a result we got the wrong type of data back: instead of returning a record of individual actions, we were returning aggregated counts. Aside from not meeting our use case, aggregation requires the curation of an ever-churning dataset in order to keep totals updated, which is not scalable for the number of DOIs in our corpus.

We soon learnt to ask the right questions. One pivotal change in approach was that instead of counts, we asked instead ‘what activity is happening on Twitter for this article?’. Our data went from ‘DOI X was mentioned 20 times on Twitter as of this date’ to ‘tweet X mentions DOI X on this date’. The data are now represented as a subject-verb-object triple:

Figure 1.2 Triple table.

Ultimately this has allowed us to represent actions like Wikipedia page edits as individual atomic actions (i.e an Event) rather than as a dataset that changes over time. Being open about the provenance of altmetrics with Event Data Crossref Event Data (the Crossref-specific service powered by the shared Event Data infrastructure) has evolved beyond a link store to become a continual stream of Events; each Event tells a new part of the story. Rather than constantly updating an Event whenever a new action takes place, we add a new one instead:

Figure 1.3 A Wikipedia Event.

Events answer a whole range of questions, such as:

What links to what?
How was the link made?
Which Agent collected the Event?
Which data source?
When was the link observed?
When do we think the link actually happened?
What algorithms were used to collect it?
Where’s the evidence?

We’re collecting data from a diverse range of platforms including Twitter, Wikipedia, blogs and news sites, Reddit, StackExchange, Wordpress.com and Hypothes.is. This means that when we observe a link in these platforms to what we think is a DOI, we create an Event and a corresponding Evidence Record to represent our observation. We also have Events to represent the links between research items registered with Crossref and DataCite - for example, when a Crossref DOI cites a DataCite DOI and vice versa.

The provenance of the data is fully transparent and is made available to everyone via an open API. We call this the evidence trail. The record of each link (‘Events’) as well as the corresponding evidence can then be used to feed into tools for impact measurement, discoverability, collaboration and network analysis.

Therefore, one application of Event Data is as an underlying, transparent data source for altmetrics calculations. For example, you might want to know the total number of times your paper has been mentioned on Twitter to date. If I told you that the number was 22, what does that actually mean? Do you know whether I counted both tweets and retweets? Do you consider both of these actions as equal? Is the sentiment of the tweet important to you? Was it a human or a bot that initiated a tweet? Are you interested in tweets containing links to multiple representations of your paper or do you only want to track mentions of your version of record (the final published copy)? With Event Data as your underlying data source, you can answer these questions.

Not only transparent in data, transparent by design

The National Information Standards Organisation (NISO), a US organisation responsible for technical standards for publishing, bibliographic and library applications, has developed a set of recommendations for transparency in their Alternative Assessment Metrics Project report, as well as a Code of Conduct for both altmetric practitioners and aggregators that aims to help improve the quality of altmetrics data. The working groups recognised that without transparency and conforming to a recognised standard, altmetric indicators “are difficult to assess, and thus may be seen as less reliable for purposes of measuring influence or evaluation”¹.

Crossref Event Data is one of the example altmetric data providers listed in the NISO recommendations. My colleague Joe Wass participated in the development and specification of the NISO “Altmetrics Recommended Practices on Data Metrics, Alternative Outputs, and Persistent Identifiers” at the same time as we were working with DataCite on Event Data, so they have mutually informed one another.

Figure 1.4 Martin Fenner (DataCite) and Joe Wass (Crossref) drawing plans for the Event Data infrastructure.

The outcome of our involvement in the NISO recommendations is that Crossref Event Data is a service that is transparent by design. We have opened up our entire extraction and processing workflow so that we can clearly demonstrate the context and environment that was used to generate an Event. This evidence is a core component of our transparency-first principle.

Building services on Event Data

There are some really exciting ways that people are already using Event Data, and we’re still only in beta. Our aim has always been to create an open, portable, transparent data set that can be used by our diverse community including researchers, application developers, publishers, funders and third-party service providers. We have already seen data from our service used in recent research studies, impact reports and even a front-end tool. Launched recently as a prototype, ImpactStory’s Paperbuzz.org uses Event Data as one of its data sources for tracking the online buzz around scholarly articles. Jason Priem, cofounder of ImpactStory, notes:

“Because Crossref Event Data is completely open data, we believe it’s a game-changer for altmetrics. Our latest project, Paperbuzz.org, is just the first of a whole constellation of upcoming tools that will add value on top of Crossref’s open data.”

We are working towards launching Crossref Event Data as a production service. In the meantime though, please do take a look at our comprehensive User Guide. Hopefully you’ll be inspired to go make something cool using the data! Events are being collected constantly; take a look below as they stream in from our data sources or visit our live stream demo site to watch in real time.

Figure 1.5 Screen capture of Crossref Event Data live stream demo.

As the service matures, we’ll continue to add new platforms to track and I also encourage anyone with article link data to get in touch to discuss how we can share it with the community via Event Data.

For researchers in particular, I’m really keen to hear your thoughts on our data model and about the things we could additionally provide you with from an infrastructure perspective that would best support your research needs.

And if you’re a publisher, take a look at our Event Data best practice guidelines — there’s some really important information in there about how you can help give us the best chance possible of collecting Events for your registered content.

And finally, if you’re a consumer of altmetrics data, I encourage you to ask questions. Ask your altmetrics vendors about how they gather their data and what context they apply to the aggregation of the metrics they supply. Ask yourself what behaviours you are interested in tracking and equally those you are not. Think about the endgame; about the type of impact you’re truly trying to measure and the story you want to tell. Because it’s these questions that will help you choose indicators that are the best fit for your own unique narrative.

This content is cross-posted on eLife Labs.

References

¹ Bilder, Geoffrey; Lin, Jennifer; Neylon, Cameron (2015): What exactly is infrastructure? Seeing the leopard’s spots. Retrieved: Oct 16, 2017; https://doi-org.pluma.sjfc.edu/10.6084/m9.figshare.1520432.v1

² NISO, Outputs of the NISO Alternative Assessment Metrics Project. Retrieved: 6th October 2017; https://www.niso.org/publications/rp-25-2016-altmetrics , p.2.

Publishers, help us capture Events for your content

Madeleine Watson — Mon, 02 Oct 2017 00:00:00 +0000

The day I received my learner driver permit, I remember being handed three things: a plastic thermosealed reminder that age sixteen was not a good look on me; a yellow L-plate sign as flimsy as my driving ability; and a weighty ‘how to drive’ guide listing all the things that I absolutely must not, under any circumstances, even-if-it-seems-like-a-really-swell-idea-at-the-time, never, ever do.

The margin space dedicated to finger-wagging left little room for championing any driving-do’s. And as each page delivered a fresh new warning, my enthusiasm for hitting the road sunk to levels usually reserved for activities like trigonometry and visits to my orthodontist.

Many years (and an excellent driving record) later, I’m reminded of this again now when thinking about our own Event Data User Guide. Because it contains a chapter with some really important don’ts for our members. Really good, we’d-love-you-to-consider-not-doing-these-things type of advice. But despite our intent to encourage, I feel the ghost of finger-waggers past. So in the spirit of championing enthusiasm over ennui, I thought I’d attempt to contextualise our Event Data Best Practices Guide for Publishers and show you why there’s a lot of good reasons for publishers to be enthusiastic about these rules.

So if you’re a publisher, I encourage you to read on to learn more about how you can help us have the best chance possible of capturing Events for your content.

What’s in it for you? Well, collecting this data helps to give everyone (Crossref, yourself, and others) a better picture of how your content is being used, including for altmetrics.

1. Please let us in

Please do open the door when we come knocking, we promise not to stay long. You can do this by allowing the User Agent CrossrefEventDataBot to visit your site, and whitelisting it if necessary. The bot is how we visit URLs to confirm if they are for an item of content registered with us. The reason why we’re visiting your site could include:

someone tweeted an article landing page
someone discussed it on Reddit
it was linked to from a blog post

The Bot has only one job: to work out the DOI. No information beyond this is stored. Whenever we become aware of a link that we think points to a DOI or an Article Landing Page, we follow it so we can collect the required metadata. Everything in Crossref Event Data is linked via its DOI, so it’s important that we can collect this information.

The bot will identify itself using the standard method. It sets two headers:

Referer: https://eventdata-crossref-org.pluma.sjfc.edu
User-Agent: CrossrefEventDataBot (eventdata@crossref.org)

Once we confirm that a link points to registered content, we then log an Event for the DOI. You should expect our bot to visit no more than once or twice per second, although if there is a period of activity around your articles, you may see higher rates. The bot also takes a sample of DOIs and visits them to work out which domain names belong to our members, so it can maintain a list. This can happen every few weeks. You may see a small number of requests from the bot, but limited to one per second.

If we can’t enter your site to look for metadata though, then we won’t be able to collect Events for your DOIs. So by allowing our bot, you will be helping us to collect Event Data for your registered content.

If you’re worried about traffic on your site, consider sending us your mapping of article landing pages to DOIs. Because Resource URLs aren’t the same as article landing pages, we need more information than the DOI Resource URLs that you already send us.

If you’re running a blog or website (and you’re not a member of Crossref), you may also see our bot visiting, to look for links that comprise Events. Please allow us to visit, so we can record in our Event Data service the fact that your website links to registered content.

2. We ❤️ robots.txt

Robots.txt files are important and we ensure our Event Data Bot respects yours. If we are instructed not to visit a site, we won’t. So if you want us to visit your site in order to check the metadata of your article landing page, please ensure you provide an exception for our Bot, or make sure that you’re not blocking it. Check the restrictions in your file to see if we’re allowed to visit. This is just another way you can help us work for you.

3. Include the DC Identifier

Including good metadata is general best practice for scholarly publishing. When we visit a publisher’s site, we look for metadata embedded in the HTML document (such as DC.Identifier tags that, amongst other things, enable Crossmark to work).

By ensuring you include a Dublin Core identifier meta tag in each of your articles pages, our system can match your landing pages back to DOIs.

Here’s an example:

4. Let us in, even if we don’t bring cookies

We’re like that friend who turns up for dinner without bringing a bottle of wine. And we hope that you’ll be ok with that. Some Publisher sites don’t allow browsers to visit unless cookies are enabled and they block visitors that don’t accept them. If your site does this, we will be unable to collect Events for your DOIs. Allowing your site to be accessed without cookies will help give us the best chance of successfully reading your metadata.

5. We may not speak your language

Sometimes we come across a publisher’s site that won’t render unless JavaScript is enabled. This means that the site won’t show any content to browsers that don’t execute JavaScript. The Event Data Bot does not execute JavaScript when looking for a DOI. This means that if your site requires JavaScript, then we will be unable to collect DOIs for your Events. Consider allowing your site to be accessed without JavaScript. And if this is not possible, then if you ensure you include the tag in the HTML header, then we’ll do our best to collect Events for your registered content.

If you want to pass this on to your friendly system administrator, the best practice is documented in full here: https://www-eventdata-crossref-org.pluma.sjfc.edu/guide/best-practice/publishers-best-practice/. And sorry about all the don’ts you’ll find on that page…. don’t let them curb your enthusiasm for taking Event Data out for a spin!

BestBlogsRead

Christine Cormack Wood — Tue, 26 Sep 2017 00:00:00 +0000

We know that research communication happens everywhere, and we want your help in finding it!

From October 9th we will be collecting links sent in by you through a social campaign across Twitter and Facebook called #BestBlogsRead.

Simply send us links to the blogs YOU like to read It’s easy to participate, all you have to do is watch out for the daily tweets and facebook posts and then send us links to the blogs (and news sites) you read.

From gardening to gaming, recipes to rock climbing, tennis to taxidermy - whatever blogs you read, we want to hear about them!

Because research happens everywhere! And you’ll be surprised where it is mentioned - for example:

We found a Wiley article mentioned in a blog about the eclipse
An American Chemical Society article in a blog about food allergies
A blog about Neanderthals on the Atlantic links to and article from the American Association for the Advancement of Science

So, watch out for the campaign on Twitter and Facebook, and tell us about your #BestBlogsRead.

Event Data as Underlying Altmetrics Infrastructure at the 4:AM Altmetrics Conference

Joe Wass — Mon, 25 Sep 2017 00:00:00 +0000

I’m here in Toronto and looking forward to a busy week. Maddy Watson and I are in town for the 4:AM Altmetrics Conference, as well as the altmetrics17 workshop and Hack-day. I’ll be speaking at each, and for those of you who aren’t able to make it, I’ve combined both presentations into a handy blog post, which follows on from my last one.

But first, nothing beats a good demo. Take a look at our live stream. This shows the Events passing through Crossref Event Data, live, as they happen. You may need to wait a few seconds before you see anything.

Crossref and scholarly links

You may know about Crossref. If you don’t, we are a non-profit organisation that works with Publishers (getting on for nine thousand) to register scholarly publications, issue Persistent Identifiers (DOIs) and maintain the infrastructure required to keep them working. If you don’t know what a DOI is, it’s a link that looks like this:

https://doi-org.pluma.sjfc.edu/10.5555/12345678

When you click on that, you’ll be taken to the landing page for that article. If the landing page moves, the DOI can be updated so you’re taken to the right place. This is why Crossref was created in the first place: to register Persistent Identifiers to combat link rot and to allow Publishers to work together and cite each other’s content. A DOI is a single, canonical identifier that can be used to refer to scholarly content.

Not only that, we combine that with metadata and links. Links to authors via ORCIDs, references and citations via DOIs, funding bodies and grant numbers, clinical trials… the list goes on. All of this data is provided by our members and most of it is made available via our free API.

Because we are the central place that publishers register their content, and we’ve got approaching 100 million items of Registered Content, we thought that we could also curate and collect altmetrics type data for our corpus of publications. After all, a reference from a Tweet to an article is a link, just like a citation between two articles is a link.

An Experiment

So, a few years back we thought we would try and track altmetrics for DOIs. This was done as a Crossref Labs experiment. We grabbed a copy of PLOS ALM (since renamed Lagotto), loaded a sample of DOIs into it and watched as it struggled to keep up.

It was a good experiment, as it showed that we weren’t asking exactly the right questions. There were a few things that didn’t quite fit. Firstly, it required every DOI to be loaded into it up-front, and, in some cases, for the article landing page for every DOI to be known. This doesn’t scale to tens of millions. Secondly, it had to scan over every DOI on a regular schedule and make an API query for each one. That doesn’t scale either. Thirdly, the kind of data it was requesting was usually in the form of a count. It asked the question:

“How many tweets are there for this article as of today?”

This fulfilled the original use case for PLOS ALM at PLOS. But when running it at Crossref, on behalf of every publisher out there, the results raised more questions than they answered. Which was good, because it was a Labs Experiment.

Asking the right question

The whole journey to Crossref Event Data has been a process of working out how to ask the right question. There are a number of ways in which “How many tweets are there for this article as of today?” isn’t the right question. It doesn’t answer:

Tweeted by who? What about bots?
Tweeted how? Original Tweets? Retweets?
What was tweeted? The DOI? The article landing page? Was there extra text?
When did the tweet occur?

We took one step closer toward the right question. Instead of asking “how many tweets for this article are there as of today” we asked:

“What activity is happening on Twitter concerning this article?”

If we record each activity we can include information that answers all of the above questions. So instead of collecting data like this:

Registered Content	Source	Count	Date
10.5555/12345678	twitter	20	2017-01-01
10.5555/87654321	twitter	5	2017-01-15
10.5555/12345678	twitter	23	2017-02-01

We’re collecting data like this:

Subject	Relation	Object	Source	Date
twitter.com/tweet/1234	references	10.5555/12345678	twitter	2017-01-01
twitter.com/tweet/5678	references	10.5555/987654321	twitter	2017-01-11
twitter.com/tweet/9123	references	10.5555/12345678	twitter	2017-02-06

Now we’re collecting individual links between tweets and DOIs, we’re closer to all the other kinds of links that we store. It’s like the “traditional” links that we already curate except:

It’s not provided by publishers, we have to go and collect it ourselves.
It comes from a very diverse range of places, e.g. Twitter, Wikipedia, Blogs, Reddit, random web pages
The places that the Events do come from don’t play by the normal rules. Web pages work differently to articles.

Non-traditional Publishing is Untraditional

This last point caused us to scratch our heads for a bit. We used to collect links within the ’traditional’ scholarly literature. Generally, journal articles:

get published once
have a publisher looking after them, who can produce structured metadata
are subject to a formal process of retractions or updates

Now we’re collecting links between things that aren’t seen as ’traditional’ scholarship and don’t play by the rules.

The first thing we found is that blog authors don’t reference the literature using DOIs. Instead they use article landing pages. This meant that we had to put in the work to collect links to article landing pages and turn them back into DOIs so that they can be referenced in a stable, link-rot-proof way.

When we looked at Wikipedia we noticed that, as pages are edited, references are added and removed all the time. If our data set reflected this, it would have to evolve over time, with items popping into existence and then vanishing again. This isn’t good.

Our position in the scholarly community is to provide data and infrastructure that others can use to create services, enrich and build things. Curating an ever changing data set, where things can disappear, is not a great idea and is hard to work with.

We realised that a plain old link store (also known as an assertion store, triple store, etc.) wasn’t the right approach as it didn’t capture the nuance in the data with sufficient transparency. At least, it didn’t tell the whole picture.

We settled on a new architecture, and Crossref Event Data as we now know it was born. Instead of a dataset that changes over time, we have a continual stream of Events, where each Event tells a new part of the story. An Event is true at the time it is published, but if we find new information we don’t edit Events, we add new ones.

An Event is the way that we tell you that we observed a link. It includes the link, in “subject - relation type - object” format, but so much more. We realised that one question won’t do, so Events now answer the following questions:

What links to what?
How was the link made? Was it with a article’s DOI or straight to an Article landing page?
Which Agent collected it?
Which data source were they looking at?
When was the link observed?
When do we think the link actually happened?
What algorithms were used to collect it?
How do you know?

I’ll come back to the “how do you know” a bit later.

What is an altmetrics Event?

So, an Event is a package that contains a link plus lots of extra information required to interpret and make sense of it. But how do we choose what comprises an Event?

An Event is created every time we notice an interaction between something we can observe out on the web and a piece of registered content. This simple description gives rise to some interesting quirks.

It means that every time we see a tweet that mentions an article, for example, we create an Event. If a tweet mentions two articles, there are two events. That means that “the number of Twitter events” is not the same as “the number of tweets”.

It means that every time we see a link to a piece of registered content in a webpage, we create an Event. The Event Data system currently tries to visit each webpage once, but we reserve the right to visit a webpage more than once. This means that the number of Events for a particular webpage doesn’t mean there are that many references.

We might go back and check a webpage in future to see if it still has the same links. If it does, we might generate a new set of Events to indicate that.

Because of the evolving nature of Wikipedia, we attempt to visit every page revision and document the links we find. This means that if an article has a very active edit history, and therefore a large number of edits, we will see repeated Events to the literature, once for every version of the page that makes references. So the number of Events in Wikipedia doesn’t mean the number of references.

An Event is created every time we notice an interaction. Each source (Reddit, Wikipedia, Twitter, blogs, the web at large) has different quirks, and you need to understand the underlying source in order to understand the Events.

We put the choice into your hands.

If you want to create a metric based on counting things, you have a lot of decisions to make. Do you care about bots? Do you care about citation rings? Do you care about retweets? Do you care about whether people use DOIs or article landing pages? Do you care what text people included in their tweet? The answer to each of these questions means that you’ll have to look at each data point and decide to put a weighting or score on it.

If you wanted to measure how blogged about a particular article was, you would have to look at the blogs to work out if they all had unique content. For example, Google’s Blogger platform can publish the same blog post under multiple domain names.

A blog full of link spam is still a blog. You may be doing a study into reputable blogs, so you may want to whitelist the set of domain names to exclude less reputable blogs. Or you may be doing a study into blog spam, so lower quality blogs is precisely what you’re interested in,

If you wanted to measure how discussed an article was on Reddit, you might want to go to the conversation and see if people were actually talking about it, or whether it was an empty discussion. You might want to look at the author of the post to see if they were a regular poster, whether they were a bot or an active member of the community.

If you wanted to measure how referenced an article was in Wikipedia, you might want to look at the history of each reference to see if it was deleted immediately. Or if it existed for 50% of the time, and to give a weighting.

We don’t do any scoring, we just record everything we observe. We know that everyone will have different needs, be producing different outcomes and use different methodologies. So it’s important that we tell you everything we know.

So that’s an Event. It’s not just a link, it’s the observation of a link, coupled with extra information to help you understand it.

How do you know?

But what if the Event isn’t enough? To come back to the earlier question, “how do you know?”

Events don’t exist in isolation. Data must be collected and processed. Each Agent in Crossref Event Data monitors a particular data source and feeds data into the system, which goes and retrieves webpages so it can make observations. Things can go wrong.

Any one of these things might prevent an Event from being collected:

We might not know about a particular DOI prefix immediately after it’s registered.
We might not know about a particular landing page domain for a new member immediately.
Article landing pages might not have the right metadata, so we can’t match them to DOIs.
Article landing pages might block the Crossref bot, so we can’t match DOIs.
Article landing pages might require cookies, or convoluted JavaScript, so the bot can’t get the content.
Blogs and webpages might require cookies or JavaScript to execute.
Blogs might block the Event Data bot.
A particular API might have been unavailable for a period of time.
We didn’t know about a particular blog newsfeed at the time.

This is a fact of life, and we can only operate on a best-effort basis. If we don’t have an Event, it doesn’t mean it didn’t happen.

This doesn’t mean that we just give up. Our system generates copious logs. It details every API call it made, the response it got, every scan it made, every URL it looked at. This amounts to about a gigabyte of data per day. If you want to find out why there was no Wikipedia data at a given point in time, you can go back to the log data and see what happened. If you want to see why there was no Event for an article by publisher X, you can look at the logs and see, for example, that Publisher X prevented the bot from visiting.

Every Event that does exist has a link to an Evidence Record, which corresponds with the logs. The Evidence Record tells you:

which version of the Agent was running
which Artifacts and versions it was working from
which API requests were made
which inputs looked like possible links
which matched or failed
which Events were generated

Artifacts are versioned files that contain information that Agents use. For example, there’s a list of domain names, a list of DOI prefixes, a list of blog feed urls, and so on. By indicating which version of these Artifacts were used, we can explain why we visited a certain domain and not another.

All the code is open source. The Evidence Record says which version of each Agent was running so you can see precisely which algorithms were used to generate the data.

Between the Events, Evidence Records, Evidence Logs, Artifacts and Open Source software, we can pinpoint precisely how the system behaved and why. If you have any questions about how a given Event was (or wasn’t) generated, every byte of explanation is freely available.

This forms our “Transparency first” idea. We start the whole process with an open Artifact Registry. Open source software then produces open Evidence Records. The Evidence Record is then consulted and turned into Events. All the while, copious logs are being generated. We’ve designed the system to be transparent, and for each step to be open to inspection.

We’re currently in Beta. We have over thirty million Events in our API, and they’re just waiting for you to use them!

Head over to the User Guide and get stuck in!

If you are in Toronto, come and say hi to Maddy or me.

You do want to see how it's made — seeing what goes into altmetrics

Joe Wass — Mon, 14 Aug 2017 00:00:00 +0000

There’s a saying about oil, something along the lines of “you really don’t want to see how it’s made”. And whilst I’m reluctant to draw too many parallels between the petrochemical industry and scholarly publishing, there are some interesting comparisons to be drawn.

Oil starts its life deep underground as an amorphous sticky substance. Prospectors must identify oil fields, drill, extract the oil and refine it. It finds its way into things as diverse as aspirin, paint and hammocks. And as I lie in my hammock watching paint dry, I’m curious to know how crude oil made its way into the aspirin that I’ve taken for the headache brought on by the paint fumes. Whilst it would be better if I did know how these things were made, not knowing doesn’t impair the efficacy of my aspirin.

Altmetrics start life deep inside a number of systems. Data buried in countless blogs, social media and web platforms must be identified, extracted and refined before it can be used in products like impact assessments, prompts to engagement, and even tenure decisions. But there the similarity ends. Like the benzene in my aspirin, the data that goes into my favourite metric has come a long way from its origins. But that doesn’t mean that I shouldn’t know how it was made. In fact, knowing what went into it can help me reason about it, explain it and even improve it.

Heavy industry or backyard refinery?

When you head out to fill your car, you buy fuel from a company that probably did the whole job itself. It found the crude oil, extracted it, refined it, transported it and pumped it into your car. Of course there are exceptions, but a lot of fuel is made by vertically integrated companies who do the whole job. And whilst there are research scientists who brew up special batches for one-off pieces of research, if you wanted to make a batch of fuel for yourself you’d have to set up your own back-yard fractional distillation column.

Because the collection of a huge amount of data must be boiled down into altmetrics, organisations who want to produce these metrics have a big job to do. They must find data sources, retrieve the data, process it and produce the end product. The foundation of altmetrics is the measurement of impact, and whilst the intermediary data is very interesting, the ultimate goal of a metric is the end product. If you wanted to make a new metric you’d have two choices: set up an oil refinery (i.e. build a whole new system, complete with processing pipeline) or a back-yard still (a one-off research item). Either option involves going out and querying different systems, processing the data and producing an output.

Being able to demonstrate the provenance of a given measurement is important because no measurement is perfect. It’s impossible to query every single extant source out there. And even if you could, it would be impossible to prove that you had. And even then, the process of refinement isn’t always faultless. Every measurement out there has a story behind it, and being able to tell that story is important when using the measurement for something important. Data sources and algorithms change over time, and comparing a year-old measurement to one made today might be difficult without knowing what underlying observations went into it. A solution to this is complete transparency about the source data, how it was processed, and how it relates to the output.

Underlying data

This is where Crossref comes in. It turns out that the underlying data that goes into altmetrics is just our kind of thing. As the DOI Registration Agency for scholarly literature, it’s our job to work with publishers to keep track of everything that’s published, assign DOIs and be the central collection and storage point for metadata and links. Examples of links stored in Crossref are between articles and funders, clinical trial numbers, preprints, datasets etc. With the Event Data project, we are now collecting links between places on the web and our registered content when they’re made via DOIs or article landing pages.

This data has wider use than just than altmetrics. For example, an author might want to know over what time period a link to their article was included in Wikipedia, and which edit to the article was responsible for removing it and why. Or, in these days of “fake news”, someone may want to know everywhere on Twitter that a particular study is referenced so they can engage in conversation.

Whilst the field of altmetrics was the starting point for this project, our goal isn’t to provide any kind of metric. Instead, we provide a stream of Events that occurred concerning a given piece of registered content with a DOI. If you want to build a metric out of it, you’re welcome to. There are a million different things you could build out of the data, and each will have a different methodology. By providing this underlying data set, we hope we’ve found the right level of abstraction to enable people to build a wide range of things.

Every different end-product will use different data and use different algorithms. By providing an open dataset at the right level of granularity, we allow the producers of these end-products to say exactly which input data they were working with. By making the data open, we allow anyone else to duplicate the data if they wish.

Sticky mess

To finish, let me return to the sticky mess of the distillation column. We identify sources (websites, APIs and RSS feeds). We visit each one, and collect data. We process that data into Events. And we provide Events via an API. At each stage of processing, we make the data open:

The Artifact Registry lists all of the sources, RSS feeds and domains we query.
The Evidence Registry lists which sites we visited, what input we got, what version of each Artifact was used, and which Events were produced.
The Evidence Log describes exactly what every part of the system did, including if it ran into problems along the way.
The Events link back to the Evidence so you can trace exactly what activity led up to the Event.
All the code is open source and the version is linked in the Evidence Record, so you can see precisely which algorithms were used to generate a given Event.
Anyone using the Data can link back to Events, which in turn link back to their Evidence.

The end-product, Events, can be used to answer altmetrics-y questions like “who tweeted my article?”. But the layers below that can be put to a range of other uses. For example:

“Why does publisher X have a lower Twitter count?”. The Evidence Logs might show that they tend to block bots from their site, preventing data from being collected.
“Why did their Twitter count rise?”. The Evidence Logs might show that they stopped blocking bots.
“What does Crossref think the DOI is for landing page X?”. A search of the Evidence Logs might show that the Event Data system visited the page on a given date and decided that it corresponded to DOI Y.
“Which domains hold DOI landing pages?”. The “Domains” Artifact will show the domains that Event Data looked at, and the Evidence Logs will show which versions were used over time.

By producing not only Events, but being completely transparent about the refinement process, we hope that people can build things beyond traditional altmetrics, and also make use of the intermediary products as well. And by using open licenses, we allow reuse of the data.

See you in Toronto!

There’s so much more to say but I’ve run out of ink. To find out more, come to 4:AM Altmetrics Conference! I’ll be speaking at the conference in Session 10 on the 28th. I’ll also be at the Altmetrics Workshop on the 26th. Stacy Konkiel and I are hosting the Hackathon on the 29th, where you can get your hands on the data. See you there!

This blog post was originally posted on the 4:AM Altmetrics Conference Blog.

Event Data enters Beta

Jennifer Kemp — Wed, 05 Jul 2017 00:00:00 +0000

We’ve been talking about it at events, blogging about it on our site, living it, breathing it, and even sometimes dreaming about it, and now we are delighted to announce that Crossref Event Data has entered Beta.

A collaborative initiative by Crossref and DataCite, Event Data offers transparency around the way interactions with scholarly research occur online, allowing you to discover where it’s bookmarked, linked, liked, shared, referenced, commented on etc., across the web, and beyond publisher platforms.

The name Event Data reflects the nature of the service, as it collects and stores digital actions that occur on the web, from the quick and simple, such as bookmarking and referencing, through to deeper interconnectivity such as exposing the links between research artifacts. Each individual action is timestamped and recorded in our system as an Event, and made available to the community via an API.

Event Data will be available for absolutely anyone to use; publishers, third party vendors, editors, bibliometricans, researchers, authors, funders etc., and with tens of thousands of events occurring every day, there’s a wealth of insight to be gained for those interested in analyzing and interpreting the data.

It’s important to note that Event Data does not provide metrics. What is does provide is the raw data to help you facilitate your own analysis, giving you the freedom to integrate the data into your own systems.

We are currently working very closely with a few organisations with specific use cases who are helping us to test and refine Beta before we launch our production service later this year. If you decide to take a look at Beta yourself, all the data you collect from Event Data is licensed for public sharing and reuse according to our Terms of Use.

Until Event Data is in production mode, we do not recommend building any commercial or customer-based tools off the data. If you are not in the Beta test group but are interested in participating, please contact me below. For more information about Event Data, please see our user guide.

Please contact me, Jennifer Kemp—Outreach Manager for Event Data—with any questions.

URLs and DOIs: a complicated relationship

Joe Wass — Fri, 04 Nov 2016 00:00:00 +0000

As the linking hub for scholarly content, it’s our job to tame URLs and put in their place something better. Why? Most URLs suffer from link rot and can be created, deleted or changed at any time. And that’s a problem if you’re trying to cite them.

Thus the Crossref DOI was born: an Identifier which is Persistent, which means that it’s designed to live forever (or, as Geoff Bilder rather more prosaically puts it, as long as we do), and also Resolvable, which means that you can click on it. A DOI is a URL, but it’s imbued with special properties. I say special, not magical, because all of the things that make Crossref DOIs what they are, are obtained through agreements and common standards rather than any kind of magic.

As part of the development of Crossref Event Data I’ve been doing some research about the relationship between DOIs and URLs. It’s a problem we have to solve in order to make Event Data work, but it’s a much broader and more interesting story, and the results have wide applicability. I’ll be telling this story at PIDapalooza. If you’re interested in Persistent Identifiers you should go and registration is open, though hurry, as it’s next week and in Rejkjavik, Iceland!

This is also a story in progress. As I write not all of the data is in, and we can be certain that it will evolve in ways we have no idea about. It’s also quite long but I’ll do my best to disqualify it from the bedtime reading list.

Full circle

Crossref was established just over fifteen years ago with the purpose of forming the linking hub between publishers. Our job was — and still is — to register content for publishers and then continue to work with them to ensure their DOIs always point to the right location of the content. To do this we need to do one main thing: send people in the right direction when they click on a DOI, and know which direction to point them in.

Today, linking is still an important part of what Crossref does, but we do a huge amount more. One of the new things we’re working on is Crossref Event Data. It’s a service for tracking how and where people use scholarly content (such as articles) across the web and social media. Early research suggested that if we limited ourselves to just looking for DOIs we wouldn’t find much. Instead we broadened our aims a little: rather than looking for mentions of registered content exclusively via their DOIs, we look for them via the most suitable mechanism. In most cases this means the actual URL of the Item. So we have come full circle: we started linking DOIs to URLs. Now we’re trying to link URLs back to DOIs.

Which URL are we talking about here? The Crossref Guidelines say:

DOI-routed reference links enabled by Crossref must resolve to a response page containing no less than complete bibliographic information about the target content …

http://www.crossref.org.pluma.sjfc.edu/02publishers/59pub_rules.html

This is what’s referred to as the Landing Page. Every Landing Page has a URL. Usually when you want to read information about an Article, it’s the Landing Page that you’re looking at. I should also say at this point that when I say Article I mean any item of Crossref Registered Content with a DOI. So the same applies to books, chapters, conference proceedings etc. But as most items are Articles, I’ll stick with that for now.

I’m going to make some assumptions. Unfortunately, and I don’t want to spoil the surprise here, they all turn out to be false. They’re all reasonable assumptions, though, and you would be forgiven for thinking, or at least wishing, that they were true.

So suspend your disbelief and follow me down the rabbit-hole…

Assumption 1: A DOI points directly to a Landing Page URL

When you click on a DOI you are taken to the Article Landing Page. It seems like a perfectly valid assumption to think that you are taken directly there.

The DOI system is essentially a big lookup table. In the first column is the DOI and in the second column is the URL. Publishers request that we register each item’s DOI and supply us with the URL it should point to. We work with CNRI and the International DOI Foundation to keep the system running and it means that when you, the reader at home, click on a DOI, you end up on the article’s Landing Page.

It would be very convenient if our assumption were true. If we wanted to turn a URL back into an article page, we could just swap the two columns and find the DOI by looking up the URL.

It turns out that it’s not quite so simple.

The Landing Page is under control of the publisher, as is the URL that they supply us with. They don’t need to supply us with the final landing page URL, only with one that leads to the landing page.

HTTP redirects

When you request a URL, either by typing it into your browser or by clicking on a link, your browser contacts the server and gets a reply. That reply can be “200 OK, here’s your page”, “303, look over there” or the dreaded “404, I can’t find it”. Other HTTP response codes are available, including well-known classics such as 201, 500 and 418.

If it’s a 303, your browser will follow the redirect URL. The response that comes back from that redirect could be another 303. You could end up following a whole chain of redirects. You wouldn’t notice anything, except having to wait an extra few milliseconds.

Extraordinary diversity

Crossref was created by a group of publishers who needed a way to link between articles. It was an ambitious goal: create a central system with which any publisher can integrate their own systems; one that allows linking to any article no matter who published it. Today we have over 5,000 members and counting, all contributing to our metadata engine. And up to 2 million DOIs are resolved every day, by all kinds of people and systems. Our wide range of members means a wide range of systems with a wide range of designs.

This brings an extraordinary diversity of behavior. If we want to make observations about DOIs we can’t just take a random sample of the over 80 million. Instead, we need to take a sample of DOIs per Publisher System. Even taking a sample per publisher might not do the job because some publishers run a variety of systems.

Experiment 1: Does Crossref know all Landing Pages?

By NASA / Paul Riedel (Great Images in NASA: Home - info - pic) [Public domain], via Wikimedia Commons

Hypothesis: Crossref knows the Landing Page URL for all DOIs.

For a sample of Items, we can follow the DOI link all the way through to the Landing Page, following any redirects, then compare the final Landing Page URL to the one that Crossref knows about. If there are extra redirects, that means that the one we have on file isn’t the final one.

We need to tighten up the terminology at this stage:

DOI URL - The full DOI, e.g. https://doi-org.pluma.sjfc.edu/10.5555/12345678 .
Resource URL - The URL that Crossref has on file (stored in our system). This is where the browser is initially redirected.
Destination URL - The URL that we end up at if we follow all the redirects.
Article Landing Page - The page that represents the item. If everything works, this should be the same as the Destination URL.

The reason we’re talking about the Destination URL as distinct from the Article Landing Page when they should be the same thing will become clear later. Consider yourself foreshadowed.

So let’s re-word our hypothesis:

Hypothesis: The Destination URL is the same as the Resource URL.

Method: A sample of DOIs was taken (most items updated in 2016, all from 2009 or earlier). The Resource URL was obtained for all of them. The DOIs were split by the domain name of the Resource URL (to give a good coverage of all Publisher systems). A sample of Resource URLs was followed per domain, at least 200 (or fewer if that exceeds the number of DOIs available). Where there were HTTP redirects they were followed.

Observations:

Number of Items sampled Destination URL: 253,381
Number where Resource URL = Destination URL: 46,995 or 19.96%

Conclusion: Not all Resource URLs are the same as the Destination URL by a long shot. Crossref does not automatically know every landing page URL.

Now we know the truth about our first assumption: DOIs don’t point directly to Landing Pages. If we want to reverse Landing Pages back into DOIs, we’re going to need to go a bit deeper…

Interlude

But first, an interlude with some information about publishers, owners, and systems, because now seems like the right time to do it.

Assumption 2: You can tell the publisher of a DOI by looking at its prefix

This is a real one one that people believe. Again, it’s entirely understandable. People look at a DOI like https://doi-org.pluma.sjfc.edu/10.1371/journal.pone.0136117.g001 , which takes them to PLoS and naturally assume that another DOI like https://doi-org.pluma.sjfc.edu/10.1371/journal.pone.0136053.t003 — because it has the same prefix of 10.1371 — is also for a PLoS item.

Whilst this turns out to be true most of the time, it’s not true for all Items, which makes it a dangerous assumption to make.

It is true that every publisher is given a prefix. They can then register DOIs with this prefix. It is also true that Items can be transferred between publishers. Because DOIs are persistent, the prefix in the DOI doesn’t change. So you might find a DOI that belongs to a publisher that has an unexpected prefix. Publishers can also be bought and sold, merged and split, which means that whilst most publishers have a single prefix, some, like Elsevier, have several. Take the case of Elsevier, who has 26 at the time of writing (you can see this in Elsevier’s entry in the Crossref Metadata API).

Every Item has an ‘owner prefix’ in addition to the prefix in the DOI. The owner prefix is the same as the DOI prefix when the Item is created, but over time, as articles are transferred, that can change to indicate that it is owned by another publisher.

Every Item has a DOI, and every DOI has a prefix. But every Item also has an Owner Prefix (you can check this in the Metadata API in the ‘prefix’ field).

So Assumption 2 has been laid to rest. The only thing you can tell from looking at a DOI is that it is, in fact, a DOI (you can tell by the “10.” index code).

Why do we care about identifying publishers anyway?

A Fair Test

We fundamentally want to conduct a fair test. The reason we can’t just take a random sample from the set of all DOIs is that there are lots of members who all do things slightly differently. Therefore we need to take a sample per publisher ‘system’. The word ‘system’ is a bit fuzzy, but my assumption is that two articles in the same system will behave the same way so we can treat them the same.

We also know that each Crossref member may be running more than one system, or a mixture. Therefore just looking at the owner of a DOI may not give accurate results if we want to conduct a survey of all the systems out there.

There’s no perfect answer, but the approach I’m taking is to look at the domain name of the Resource URL. We often find lots of subdomains for the same publisher, for example, “psw.sagepub.com”, “pol.sagepub.com”, “psx.sagepub.com” and “bpi.sagepub.com”. It’s clear that these are all operated by Sage, but they might or might not all be running on different ‘systems’.

Therefore I’m splitting DOIs up into groups based on the domain of their Resource URL. It may turn out that some publishers use a single system running on many domains, or it may turn out that some publishers use a different system for each domain they use. The key point is to find a sampling technique that broadly works, and that allows us to explore and differentiate, as keenly as possible, the variety of systems and behaviours.

Why all the redirects?

Curious minds might at this stage be wondering about all these extra redirects. Surely it’s extra stuff for the publisher to maintain. Why don’t they just point the DOI directly to the landing page?

The answer must be prefaced by repeating that there is a huge number of publishers, running a variety of systems, so we’ll never be able to completely answer that. But some humble suggestions:

They might want to be able to change the URLs of the Landing Pages. It may be easier to update their internal systems than send the update to Crossref, especially in bulk.
Different parts of their technology stack may be owned by different parts of the company, or outsourced. It’s easier to define internal boundaries than to co-ordinate business units and cross an external one.
A publisher may run a mix of different technology. As part of their systems integration process, they set up a redirect server to make everything work together.
A publisher assigns DOIs to articles but also has their own internal IDs. They maintain their own DOI-to-internal-ID lookup service.

Internal DOI resolvers

That last point is an interesting one. The DOI system is the canonical “DOI-to-URL resolver”. That doesn’t prevent publishers from running their own. Indeed, many do.

To take a real example of PLoS, an Open Access publisher who registers lots of content with Crossref. To follow one of their DOIs we go on the following journey of redirects:

http://dx.doi.org.pluma.sjfc.edu/10.1371/journal.pone.0164910
http://dx.plos.org/10.1371/journal.pone.0164910
http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0164910
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0164910

Given that the last step uses a DOI, this suggests that they use the DOI as an internal identifier. All those redirects were for some purpose, but they weren’t mapping a DOI to an internal ID. This is therefore not an internal DOI resolver.

Another example from JAMA Surgery:

In this case we see a mapping from the DOI 10.1001/archsurg.142.7.595 to the ID 487551.

Can we define a heuristic for this pattern? Yes, but not a perfect one. My test is this:

Does the resource URL contain the DOI?
If so, does it redirect to a different destination URL?
If so, does the destination URL not contain the DOI?

The last step is important, because we can’t really say the publisher is running a DOI resolver if they use the DOI all the way through.

It’s not perfect and no doubt has false negatives. But we’re just trying to find out whether some publishers run their own DOI resolver systems.

Experiment 2: Determine how widespread use of internal DOI resolvers is:

Hypothesis: Some publishers run their own DOI resolvers.

Method: A number of Destination URLs were sampled per Resource URL Domain. If the Resource URL contains the DOI but the Destination URL doesn’t, that’s marked as a Publisher DOI resolver redirect.

Observations:

Number of Items sampled with Resource URL and Destination URL: 253,381
Number of Items that appear to be DOI resolvers: 166,352 = 65.6%

Conclusions: Some publishers run their own DOI resolvers.

This isn’t of much practical use, but it’s interesting to know, and hints at the way the Crossref system and DOIs are integrated with Publishers’ systems. Now that we’ve got a little insight into the reasons that publishers might run their own DOI resolvers, we can resume our journey of assumptions.

Assumption 3: We can find the Landing Page for Every DOI

Now we know that we can’t just use the lookup table in reverse, but have to follow the links all the way to their destination. Does this approach actually work?

This is a pretty big question and we need to be clear about what we mean by ‘every’ DOI. The set of DOIs I’m using (although I’m using a subset) is “all DOIs in our Metadata API that are found in doi.org”.

What is a DOI? Geoff Bilder went over it in the DOI-like-strings blog post earlier this year. The definition I’m working to here is:

A DOI is an identifier for an item of content registered in the DOI system.

That is, if you resolve the DOI on https://doi-org.pluma.sjfc.edu/ and it’s recognised, that counts as a DOI. I’m working from the set of DOIs found in the Crossref system as I’m primarily concerned with Crossref DOIs. However, we collaborate closely with DataCite.

Back to our assumption: “we can find the Landing Page for every DOI”. The answer is that we can, most of the time. But because Crossref Event Data has to work as well as possible, and therefore work with as many DOIs as possible, we have to scour all the nooks and crannies.

Assumption 4: Every DOI points somewhere unique

Stop me when you find the deliberate mistake:

Every Item corresponds to a different thing
Every Item has a single DOI
Every DOI is different
Every DOI points to a landing page
Therefore every DOI points to a different landing page

Two things immediately suggest themselves:

“Every item has a single DOI” should be true, but it isn’t. We find that sometimes two DOIs are assigned to the same item. This can happen when publications change hands between publishers, or when mistakes are made, or for a variety of other reasons. We also find that in some cases Publishers registered a DOI for the metadata and one for the article abstract. The two DOIs point to the same place. In some cases where there were two DOIs registered for the same thing we create an Alias.

When we alias a DOI we simply say “this DOI should actually point to this one”. Both DOIs still exist, and both still point to the ‘correct’ thing, it’s just that they both point to the same place. If we have two DOIs pointing to the same place, then there isn’t a one-to-one mapping, and Assumption 4 is incorrect.

Experiment 4: Aliased DOIs

Hypothesis: There isn’t a one-to-one mapping between DOIs and URLs because some DOIs are aliased to others.

Method: We collected a sample of Resource URLs from the DOI API. We count how many DOIs are classified as Aliases in the DOI system.

Observations:

From a sample of 11,227,458 DOIs
14,566 are aliased to others, or 0.129%

Conclusion: There aren’t many aliases. But there are some, and we should be aware of them.

Experiment 5: Duplicate Resource URLs

Hypothesis: There isn’t a one-to-one mapping between DOIs and URLs because some DOIs have duplicate Resource URLs.

Method: A sample of Resource URLs was collected from the DOI API. We counted how many DOIs have Resource URLs that aren’t unique. We subtract the number of deleted DOIs because all deleted DOIs have the same resource URL.

Observations:

From a sample size of 11,227,458
a total of 112,195 have duplicate resource URLs, or 0.99%
of these duplicates, 77,896 have the ‘deleted’ URL
leaving 34,229, or 0.30% having non-unique Resource URLs

Conclusion: A small number of DOIs have duplicate Resource URLs, even if we exclude those that have been deleted, which means that not every DOI can have a unique URL.

Assumption 5: The Landing Page is the same as the Destination Page.

HTTP has a very neat system for doing redirects. If it were that simple, then we could easily look up every Destination page and confidently say that it was the Landing Page. Not so.

Cookies

Web browsers aren’t the only tools that use HTTP. Most programming languages have HTTP capabilities built in.

Using cookies is a requirement of some websites, but it’s not a requirement of HTTP. Most websites use cookies in some way or another. When you log into a site, you expect cookies. But when you’re just browsing there isn’t any technical need. A small number of websites absolutely require cookies to be enabled to use the site, even if you’re just browsing and not logged in. Unfortunately, this includes some publishers.

Requiring cookies to use a publisher site means that you can’t fully resolve a DOI without enabling cookies. Most tools out there don’t. Some privacy-conscious people quite reasonably don’t enable cookies from all sites.

Using cookies when resolving a DOI adds considerable overhead and isn’t fool-proof.

Let’s try a quick experiment to see when we land up on a cookie page. Here’s an example page that tells us that we should have enabled cookies: http://www-tandfonline-com.pluma.sjfc.edu/action/cookieAbsent . It’s reachable from the DOI: https://doi-org.pluma.sjfc.edu/10.1016/j.envhaz.2007.09.007 .

Experiment 6: Some DOIs can’t be resolved without cookies

Hypothesis: We can’t resolve some DOIs to the Landing Page using standard tools because cookies are required.

Method: A sample of DOIs was taken per Resource URL Domain. They were resolved by following HTTP links. Where the Destination URL contains the word ‘cookie’, we mark that as a DOI requiring a cookie.

Observations:

A sample of 253,381 DOIs were resolved following HTTP redirects where necessary
a total of 6305 resolved to a page with ‘cookie’ in the URL or 2.48%

Conclusion: There are cookies at play for at least 2.48% of DOIs. This is probably a very conservative estimate, as we’re using a blunt tool looking for ‘cookie’ in the URL.

Cookies Required

For one DOI I found, the publisher system set cookies, then sent us on a series of redirects which set cookies that expired in the past and then, as far as I can tell, checked whether or not they were sent back. My working hypothesis is that it was profiling the behaviour to see what browser I was using.

I have also seen javascript-based redirects. This is where a web page loads a javascript file, which executes and sends the browser onto another URL. This seems to be to be a browser detection method. There is no way you can follow these DOIs without actually using a real browser.

This is a problem for Crossref Event Data. We can’t fire up a browser and follow every DOI: it isn’t practical. When I tried this for a sample as an experiment I got an email from another publisher who was worried that we were scraping data (good bot operators always put contact details in their request headers!).

The Crossref member rules leave some wiggle-room about whether this is allowed, but for the Event Data service, we can say that it’s a physical impossibility to collect all Event Data for DOIs like this.

Bring in the Browser

To quantify the size of the problem, we need to bring in a web browser. If we assume that some Publishers design their sites to work only with real browsers, that’s what we’ll use. Luckily there are web browsers packaged up into an automatable package, and we can use these to visit the DOI.

Using one of these is considerably slower than just following link headers.

I have split the ‘destination’ concept into two:

Naïve destination URL: The URL that you get from following HTTP redirects acccording to the HTTP specification
Browser destination URL: The URL that you get from letting a browser follow the DOI doing whatever a browser does.

Rather than defining a complicated spectrum of types of DOI resolution behaviour, I am classifying DOIs into two groups: those where standard HTTP redirects are sufficient and everything else.

The method I am using is to resolve a sample of URLs using the browser. I can then compare the Naïve Destination URL with the Browser Destination URL. If they are the same, then I didn’t need to use the browser after all. If they give a different result however, I trust the Browser one better and declare that DOI to require a browser to resolve.

Again, I took a sample of DOIs per Resource URL domain.

Experiment 7: Quantify proportion of DOIs that require a browser to redirect

Hypothesis: A number of DOIs can’t be resolved with standard tools but instead require a browser.

Method: A sample of DOIs was selected per Resource URL domain. The links were followed using standard HTTP and using a browser. Where the URLs between the two were different, the DOI was counted as requiring a browser to resolve.

Observations:

A total of 59,453 items were followed both using the Naïve and Browser methods.
Of these 5,883 items have a different URL between the two methods, or 9.88%

Conclusion: We can’t rely on the Naïve redirect, and would have to fire up the browser in about 10% of cases in the sample.

Other gnarly things

There are one or two supplementary gnarly things that crop up.

First, session IDs are sometimes embedded in the URL. This is a tracking technique similar to cookies, but instead of sending cookies, which are invisible to the user, a unique code is placed on the end of the URL. This means that everyone gets a different URL. The most popular of these is the JSESSIONID, which is used by servers in the Java ecosystem. An example URL is:

http://onlinelibrary.wiley.com.pluma.sjfc.edu/doi/10.1002/047084289X.rn00615.pub3/abstract;jsessionid=0D1B7AC4689A494E0EA78BD2F0A710C4.f04t04

We can easily remove these if they appear at the end of a URL. Sometimes they occur in the middle of a URL, as above. Sometimes they appear as query parameters:

http://jpharmsci.org/action/consumeSharedSessionAction?SERVER=WZ6myaEXBLGvmNGtLlDx7g%3D%3D&MAID=npYBLvZTaUI3JTHw%2BH63WQ%3D%3D&JSESSIONID=aaajjhdDL5ssK6d1HHrFv&ORIGIN=207988872&RD=RD

In this case we make no attempt to remove them. These URLs won’t be any use for matching, and we have to acknowledge that and move on.

Interpreting the results

All the above experiments involved taking as many DOIs as we had time for, gathering the Resource URLs, and then grouping the DOIs per Resource URL Domain. A sample of DOIs was investigated per each Resource URL domain to give the best chance at even coverage. The above figures have been presented as a proportion of the sampled data-set.

Now it’s time to draw some practical conclusions. I grouped the results per Resource URL Domain, so I can say that “for this domain, X% of DOIs was deleted, or aliased, or whatever”. This means that we can look at the statistics for a given domain and work out the best method for working with DOIs that belong to it.

I have created histograms of domains by their various proportions.

Our first chart is histogram of Resource URL Domains where the Naïve Destination = the Resource URL. Each domain is given a proportion which represents how many DOIs sampled on that domain have a Landing Page equal to the Resource URL.

There’s a clear bimodal distribution here. The conclusion here is “most domains require you to follow the link to find the destination URL“. Furthermore, the domains are consistent: there are virtually no domains that have a mix of DOIs that behave differently.

Our second chart is a histogram of Resource URLs where the Browser-based redirect = the Naive URL. Each domain is given a proportion which represents how many DOIs sampled on that domain require us to fire up a browser.

Overwhelmingly, the Browser Redirect URL is the same as the Naïve Redirect URL, meaning that we don’t need to fire up the browser, we can just use the Naïve URL, which is much easier to compute. There are some resource URL domains which require every DOI to be followed in a browser rather than just following links.

We know from this that we don’t have to use the browser most of the time. There is a small number of domains where we’re unsure (under 500) and a small number of domains where we know that we have to use a browser. This means we can focus our efforts.

There are lots of DOIs and they all behave differently.

There are thousands of publishers out there registering DOIs. There are thousands of domains. Some publishers have lots of domains. This makes it impossible to make many general observations about DOIs.

You can’t tell anything by looking at the DOI

Just by looking at the DOI you can’t tell who published it, or which publisher’s system is hosting it. Therefore you can’t tell how it’s going to behave.

We’ve looked at five kinds of URLs:

The DOI itself
The Resource URL
The “naïve” redirect URL
The “browser” redirect URL
The Article Landing Page

In some cases, the Resource URL, naïve redirect URL, browser redirect and Article Landing Page are the same. In some cases they aren’t. Of these, the fifth is somewhat mythical.

DOIs fall into classifications

Each DOI falls into a category, most preferable first:

The Resource URL is the same as the Landing Page.
The Landing Page can be discovered by following HTTP redirects.
The Landing Page can be discovered by firing up a web browser to follow redirects.
The Landing Page can’t be determined.

We can predictively group DOIs

We can group DOIs by their Resource URLs and take a sample per Resource URL Domain. If all samples for a domain behave a certain way, we can place the DOIs into one of the above four groups with a probability.

We’ll never know the full story.

Because of the diversity of Publisher Systems and the long history of Crossref DOIs, we’ll never be able to describe exactly what’s going on for all DOIs.

What next?

We’re continuing to develop Crossref Event Data. The part of the system that handles turning URLs back into DOIs will never be perfect, but we know from this research that we can at least work with a subset.

I’m also working on another project which will attempt to reverse a Landing Page URL back into a DOI by looking at the metadata on the Landing Page. You can read about it here. Ultimately we’re going to have to take a blended approach. Building a useful set of Landing Page URL to DOI mappings will be part of the mix.

As Event Data matures we’ll be sharing all the datasets automatically as part of our infrastructure, including our DOI-to-URL mapping.

And any members reading, please make your DOIs as easy to follow as possible! Please don’t require JavaScript or cookies when resolving DOIs.

If you’re read this far, perhaps you’re as interested in DOIs as we are. There’s a lot more to say on the subject, but that’s enough for now. See you at PIDapalooza!

Image Credits

All images from Wikipedia Commons. Click or hover on the image to see the attribution.

Using the Crossref Metadata API. Part 2 (with PaperHive)

Rachael Lammey — Thu, 08 Sep 2016 00:00:00 +0000

We first met the team from PaperHive at SSP in June, pointed them in the direction of the Crossref Metadata API and let things progress from there. That’s the nice thing about having an API - because it’s a common and easy way for developers to access and use metadata, it makes it possible to use with lots of diverse systems and services.

So how are things going? Alexander Naydenov, PaperHive’s Co-founder gives us an update on how they’re working with the Crossref metadata:

PaperHive

PaperHive is a web-platform for collaborative reading and a cross-publisher layer of interaction on top of research documents. It lets researchers communicate in published documents in a productive and time-saving way. PaperHive thus puts academic literature, which is integrated with the platform, in the limelight and increases content usage and reader engagement.

Transforming reading into a process of collaboration gives researchers a reason to return to the content and discover new enrichments they can benefit from. Functionality like hiving, deep linking, and the PaperHive browser extension embeds communication in the researcher’s workflow. PaperHive is free to use!

How is the Crossref API used within PaperHive?

PaperHive extends the concept of a living document and offers an innovative way of displaying content without hosting it. Instead, academic documents are dynamically pulled from the publisher’s servers thus ensuring compliance with content licensing. It enables readers to stay in touch with the articles of interest beyond just saving them in an offline folder.

Crossref is the common ground on which third party companies and initiatives can build valuable services for publishers and researchers. It facilitates the integration of content into PaperHive by providing the metadata of articles and books from numerous publishers independent of the technology behind their content platforms. Moreover, if the publishers provide ORCID identifiers of authors in the Crossref metadata, researchers can immediately interact with the readers of their works.

What are the future plans for PaperHive?

In addition to integrating further publishers’ content and extending PaperHive’s feature set for readers, we also plan to extend our partnerships with other technology providers.

As far as our cooperation with Crossref is concerned, we are looking forward to the implementation of the Crossref Event Data API.

What else would you like to see in Crossref metadata?

- The quality of the existing metadata should be improved significantly. We noticed that important fields such as author or title are missing in the metadata of many documents. PaperHive ignores articles and books with incomplete metadata because it impairs the user experience. Publishers, authors and readers can only benefit from the wider and more active usage of content, so we hope that more publishers will improve the data their provide Crossref with.

- Since researchers are working with full texts on PaperHive, it would be great if links to the full text are provided in the metadata of all articles and books. The metadata should also contain information about the format of the full text (e.g., PDF, EPUB, HTML).

Thanks Alex!

Just getting started with the API or what to know more? Get in touch via feedback@crossref.org and pass on your questions and comments.

Using AWS S3 as a large key-value store for Chronograph

Joe Wass — Tue, 02 Aug 2016 00:00:00 +0000

One of the cool things about working in Crossref Labs is that interesting experiments come up from time to time. One experiment, entitled “what happens if you plot DOI referral domains on a chart?” turned into the Chronograph project. In case you missed it, Chronograph analyses our DOI resolution logs and shows how many times each DOI link was resolved per month, and also how many times a given domain referred traffic to DOI links per day.

We’ve released a new version of Chronograph. This post explains how it was put together. One for the programmers out there.

Big enough to be annoying

Chronograph sits on the boundary between normal-sized data and large-enough-to-be-annoying-size data. It doesn’t store data for all DOIs (it includes only those that are used on average once a day), but it has information on up to 1 million DOIs per month over about 5 years, and about 500 million data points in total.

Storing 500 million data points is within the capabilities of a well-configured database. In the first iteration of Chronograph a MySQL database was used. But that kind of data starts to get tricky to back up, move around and index.

Every month or two new data comes in for processing, and it needs to be uploaded and merged into the database. Indexes need to be updated. Disk space needs to be monitored. This can be tedious.

Key values

Because the data for a DOI is all retrieved at once, it can be stored together. So instead of a table that looks like

<td>
<span >2010-01-01</span>
</td>
<td>
<span >5</span>
</td>

<td>
<span >2010-02-01</span>
</td>
<td>
<span >7</span>
</td>

<td>
<span >2010-03-01</span>
</td>
<td>
<span >3</span>
</td>

10.5555/12345678

Instead we can store

<td>
{&#8220;2010-01-01&#8221;: 5, &#8220;2010-02-01&#8221;: 7, &#8220;2010-03-01&#8221;: 3}
</td>

10.5555/12345678

This is much lighter on the indexes and takes much less space to store. However, it means that adding new data is expensive. Every time there’s new data for a month, the structure must be parsed, merged with the new data, serialised and stored again millions of times over.

After trials with MySql, MongoDB and MapDB, this approach was taken with MySQL in the original Chronograph.

Keep it Simple Storage Service Stupid

In the original version of Chronograph the data was processed using Apache Spark. There are various solutions for storing this kind of data, including Cassandra, time-series databases and so on.

The flip side of being able to do interesting experiments is wanting them to stick around without having to bother a sysadmin. The data is important to us, but we’d rather not have to worry about running another server and database if possible.

Chronograph fits into the category of ‘interesting’ rather than ‘mission-critical’ projects, so we’d rather not have to maintain expensive infrastructure if possible.

I decided to look into using Amazon Web Services Simple Storage Service (AWS S3) to store the data. AWS itself is a key-value store, so it seems like a good fit. S3 is a great service because, as the name suggests, it’s a simple service for storing a large number of files. It’s cheap and its capabilities and cost scale well.

However, storing and updating up to 80 million very small keys (one per DOI) isn’t very clever, and certainly isn’t practical. I looked at DynamoDB, but we still face the overhead of making a large number of small updates.

Is it weird?

In these days of plentiful databases with cheap indexes (and by ‘these days’ I mean the 1970s onward) it seems somehow wrong to use plain old text files. However, the whole Hadoop “Big Data” movement was predicated on a return to batch processing files. Commoditisation of services like S3 and the shift to do more in the browser have precipitated a bit of a rethink. The movement to abandon LAMP stacks and use static site generators is picking up pace. The term ‘serverless architecture’ is hard to avoid if you read certain news sites.

Using Apache Spark (with its brilliant RDD concept) was useful for bootstrapping the data processing for Chronograph, but the new code has an entirely flat-file workflow. The simplicity of not having to unnecessarily maintain a Hadoop HDFS instance seems to be the right choice in this case.

Repurposing the Wheel

The solution was to use S3 as a big hash table to store the final data that’s served to users.

The processing pipeline uses flat files all the way through from input log files to projections to aggregations. At the penultimate stage of the pipeline blocks of CSV per DOI are produced that represent date-value pairs.

<td>
2010-01
</td>
<td>
2010-01-01,05<br /> 2010-02-01,02<br /> 2010-01-03,08<br /> &#8230;
</td>

<td>
2010-02
</td>
<td>
2010-02-1,10<br /> 2010-02-01,7<br /> 2010-02-03,22<br /> &#8230;
</td>

10.5555/12345678

At the last stage, these are combined into blocks of all dates for a DOI

<td>
2010-01
</td>
<td>
2010-01-01,05<br /> 2010-02-01,02<br /> 2010-01-03,08<br /> &#8230;<br /> 2010-02-1,10<br /> 2010-02-01,7<br /> 2010-02-03,22<br /> &#8230;
</td>

10.5555/12345678

The DOIs are then hashed into 12 bits and stored as chunks of CSV

day-doi.csv-chunks_8841:

10.1038/ng.3020
2014-06-24,4
2014-06-25,4
2014-06-26,3
...
10.1007/978-94-007-2869-1_7
2012-06-01,12
2012-06-02,8
...
10.1371/journal.pone.0145509
2016-02-01,13
2016-02-02,75
2016-02-03,30
...

There are 65,536 (0x000 to 0xFFFF) possible files, each with about a thousand DOIs worth of data in each.

When the browser requests data for a DOI, it is hashed and then the request for the appropriate file in S3 is made. The browser then has to perform a linear scan of the file to find the DOI it is looking for.

This is the simplest possible form of hash table: simple addressing with separate linear chaining. The hash function is a 16-bit mask of MD5, chosen because of availability in the browser. It does a great job of evenly distributing the DOIs over all 65,536 possible files.

Striking the balance

In any data structure implementation, there are balances to be struck. Traditionally these concern memory layout, the shape of the data, practicalities of disk access and CPU cost.

In this instance, the factors in play included the number of buckets that need to be uploaded and the cost of the browser downloading an over-large bucket. The size of the bucket doesn’t matter much for CPU (as far as the user is concerned it takes about the same time to scan 10 entries as it does 10,000), but it does make a difference asking user to download a 10kb bucket or a 10MB one.

I struck the balance at 4096 buckets, resulting in files of around 100k, which is the size of a medium sized image.

It works

The result is a simple system that allows people to look up data for millions of DOIs, without having to look after another server. It’s also portable to any other file storage service.

The approach isn’t groundbreaking, but it works.

HTTPS and Wikipedia

Joe Wass — Tue, 31 May 2016 00:00:00 +0000

This is a joint blog post with Dario Taraborelli, coming from WikiCite 2016.

In 2014 we were taking our first steps along the path that would lead us to Crossref Event Data. At this time I started looking into the DOI resolution logs to see if we could get any interesting information out of them. This project, which became Chronograph, showed which domains were driving traffic to Crossref DOIs.

You can read about the latest results from this analysis in the “Where do DOI Clicks Come From” blog post.

Having this data tells us, amongst other things:

where people are using DOIs in unexpected places
where people are using DOIs in unexpected ways
where we knew people were using DOIs but the links are more popular than we realised

By the time the ALM Workshop 2014 rolled around there was some preliminary data and we realised that Wikipedia came into the third category. There are lots of DOIs in Wikipedia and people click them!

I met with Dario Taraborelli, head of research at the Wikimedia Foundation, and shared the data. Dario — who co-authored in 2010 the Altmetrics Manifesto — has been interested in understanding how scholarly citations are used in Wikipedia. Over the years, Wikipedia contributors have made extensive use of references to the scientific literature using DOIs, and by doing so they have created a resource that represents today in many ways the “front matter to all research”. There is growing interest in the community in understanding how DOIs are being used in Wikipedia and in non traditional scholarship.

During our discussions the subject of Wikipedia’s gradual transition to HTTPS was raised: we anticipated that this change would affect our data gathering.

Changes

When you’re reading webpage and click on a link to another page, your web browser will usually tell the server of that second page the last page you were on. This forms the basis of trackers like Google Analytics.

In the days before HTTPS, the next site would know the full URL that you were previously on. With the change to HTTPS, this was reduced to just sending the domain name and not the full URL, or no data at all if you click from an HTTPS page to HTTP.

DOI hyperlinks are just like any other hyperlink, and are mostly HTTP not HTTPS.

Up until 2015, Wikipedia was served over HTTP, only switching to HTTPS when users were logged in or if they requested it. The Wikimedia Foundation started planning to move to HTTPS and we knew that if they did that, and continued to use HTTP DOIs then we would lose valuable research data.

A Plan

We decided that the best course of action was to try and change the DOIs in Wikipedia to use HTTPS. Simple, right?

After some further research, Dario posted a proposal on how to mitigate the impact of the HTTPS rollout, to make sure that Wikipedia can still signal its importance as a traffic source, while preserving the privacy of its users. Discussion followed and the conclusion was to change the format of every single DOI on Wikipedia, which fortunately could be done without having to edit millions of pages. You can read the full story in this post from a year ago.

The result of this effort was that well in advance of the HTTPS switchover, the DOI links were ready to continue reporting referral data.

The Switch

In June 2015 the Wikimedia foundation made the announcement that they were finalising the switch, and that within a few weeks all traffic would be HTTPS.

We held our breath. Would it work? Would we lose all referral data from Wikipedia sites? In February 2016 the last piece of the puzzle fell into place as Wikipedia gained a ‘meta referrer’ tag to explicitly specify how they would like referrers to be sent: a detailed report on the effect of this change is coming up on the Wikimedia Foundation’s blog.

The results

As detailed in the last blog post the traffic that we measured coming from Wikipedia doesn’t seem to have slowed down during 2015:

I’d call that a success! Over the period covered in the graph, Wikipedia remained prominent as a non-publisher referral of traffic to DOIs.

Looking at the balance of HTTP vs HTTPS traffic coming from wikipedia.org, the switchover was dramatic:

Thank you to Dario Taraborelli, Nemo (Federico Leva), Aaron Halfaker, Alex Stinson and everyone who put in this effort.

I’ll leave the last word to Dario:

It’s great to see this data. It shows that the switchover happened successfully, which better protects the privacy of our users whilst still reporting the fact that Wikipedia is a prominent source of traffic. This is important validation of the increasing role that Wikipedia plays in the education and scientific community.

Where do DOI clicks come from?

Joe Wass — Thu, 19 May 2016 00:00:00 +0000

As part of our Event Data work we’ve been investigating where DOI resolutions come from. A resolution could be someone clicking a DOI hyperlink, or a search engine spider gathering data or a publisher’s system performing its duties. Our server logs tell us every time a DOI was resolved and, if it was by someone using a web browser, which website they were on when they clicked the DOI. This is called a referral.

This information is interesting because it shows not only where DOI hyperlinks are found across the web, but also when they are actually followed. This data allows us a glimpse into scholarly citation beyond references in traditional literature.

Last year Crossref Labs announced Chronograph, an experimental system for browsing some of this data. We’re working toward a new version, but in the meantime I’d like to share the results for 2015 and some of 2016. We have filtered out domains that belong to Crossref member publishers to highlight citations beyond traditional publications.

Top 10 DOI referrals from websites in 2015

This chart shows the top 10 referring non-primary-publisher domains of DOIs per month. Note that if browsers don’t send the referrer (e.g. from an HTTPS page), we don’t get to find out. Because the top 10 can be different month to month, the total number of domains mentioned can be more than 10. Subdomains are combined, which means that, for example, the wikipedia.org entry covers all Wikipedia languages. This chart covers all of 2015 and the first two months of 2016.

The top 10 referring domains for the period:

webofknowledge.com
baidu.com
serialssolutions.com
scopus.com
exlibrisgroup.com
wikipedia.org
google.com
uni-trier.de
ebsco.com
google.co.uk

It’s not surprising to see some of these domains here: for example serialssolutions.com and exlibrisgroup.com are effectively proxies for link resolvers, Baidu and Google are incredibly popular search engines which would show up anywhere. But it is exciting to see Wikipedia ranked amongst these. For more detail look out for the new Chronograph.

HTTP vs HTTPS in 2015

We’ve also seen a steady increase in HTTPS referral traffic, i.e. people clicking on DOIs from sites that are using HTTPS. While it is still dwarfed by HTTP, there was a steady uptick throughout 2015.

This chart shows HTTP vs HTTPS referrals per day, which shows up the weekly spikes. It doesn’t include resolutions where we don’t know the referrer.

Increasing numbers of people are moving to HTTPS for reasons of security, privacy and protection from tampering. Google has announced plans to take HTTPS into account when ranking search results. Wikipedia has moved exclusively to HTTPS, and I’ll be telling the story of how Crossref and Wikipedia collaborated in an upcoming blog post.

Chronograph

Another version of Chronograph will be available soon. It will contain full data for all non-primary-publisher referring domains. Stay tuned!

Crossref Event Data: early preview now available

Madeleine Watson — Mon, 18 Apr 2016 00:00:00 +0000

Test out the early preview of Event Data while we continue to develop it. Share your thoughts. And be warned: we may break a few eggs from time to time!

Chicken by anbileru adaleru from the The Noun Project

Want to discover which research works are being shared, liked and commented on? What about the number of times a scholarly item is referenced? Starting today, you can whet your appetite with an early preview of the forthcoming Crossref Event Data service. We invite you to start exploring the activity of DOIs as they permeate and interact with the world after publication.

But first, a bit of background

Discussion around scholarly research increasingly occurs online after publication, for example on blogs, sharing services, social media, and wikis. These ‘events’ occur across the web on numerous platforms and are a critical part of the scholarly enterprise. We are developing an infrastructure service (Crossref Event Data) that collects, stores, and delivers raw data of the events occurring with Crossref DOIs. We will store the data in an open, auditable and portable form for the community to access. Publishers, platforms, funders, bibliometricians and service providers may benefit from access to this raw data, and it can be used to feed into research records or proprietary tools and services that offer aggregation and analysis.

For more information, see our pilot blog post and description of potential use cases.

Collaborative, transparent development

Developers Martin Fenner (DataCite) and Joe Wass (Crossref) enjoy a tofu break

Lagotto, the software originally developed at PLOS, has been extended and improved in a joint effort between DataCite and Crossref. The two DOI Registration Agencies have partnered to envision, build and release the service. On the 13th of April, after a year of collaboration, we jointly released Lagotto 5.0. You can read about the collaboration on the DataCite blog post.

Crossref and DataCite will continue to work closely together to develop Lagotto and the Event Data service. Although Crossref Event Data has mostly Crossref DOIs at launch, you will be able to find DataCite DOIs if they are cited in Crossref or Wikipedia.

All of the software that runs Event Data, including Lagotto, is developed in the open and is open source. Please refer to the Crossref Event Data Technical User Guide for full details.

Preview the data

This service is currently under development with a full launch expected the second half of 2016. Before it is launched however, we invite you to take a look around and preview a subset of the data sources we plan to include. You may experience occasional hiccups while we continue building the service.

At this stage, we are working with data from three sources although we will greatly expand the variety of platforms from which we collect data as development progresses. At this stage, you can view Mendeley bookmarks, Wikipedia references, and Crossref to DataCite links.

Mendeley

Mendeley is a reference manager and academic social network for scholars. View the number of social bookmarks from scholars or groups on Mendeley.

For example, doi.org/10.1016/J.JIP.2016.03.007 currently has 8 readers on Mendeley to date.

Wikipedia

Wikipedia is an online encyclopaedia, the Internet’s largest and most popular general reference work. View references in Wikipedia of Crossref publications in Wikipedia articles in all languages.

For example, doi.org/10.3897/ZOOKEYS.565.7185 was referenced in the Russian Wikipedia page on Oxyscelio

Crossref to DataCite links

DataCite is a global consortium that assigns DOIs to research data. This enables people to find, share, use, and cite data. You can view all the data citations to DataCite research outputs found in Crossref publications (work is underway to make the links found in DataCite metadata available in Event Data).

For example, Global, Regional, and National Fossil-Fuel CO2 Emissions (doi.org/10.3334/CDIAC/00001) dataset has been referenced by six Crossref publications to date. Software links are also included. Another example is PGOPHER (doi.org/10.5523/bris.huflggvpcuc1zvliqed497r2), a general purpose software for simulating and fitting rotational, vibrational and electronic spectra, which has been referenced by seven Crossref publications to date.

Ready to take a spin?

You can explore the Crossref Event Data early preview by visiting http://eventdata.crossref.org.pluma.sjfc.edu and following the links to featured examples within our interim application for inspecting the data, technical documentation, and our Quick Start guide.

This service is currently under development and as such we welcome your thoughts and feedback on the data we are collecting currently from our three active sources. As a reminder, we expect to include the following sources as part of our full service launch later this year (pending confirmation):

[table id=1 /]

We’re also on the lookout for new data sources to investigate for future inclusion in the Event Data service so please do get in touch with requests and recommendations. As we continue to build the service throughout 2016, we will be committing to a model of continuous development so that we can make new sources available as they are completed.

Watch this blog for regular updates on our progress, or subscribe to receive new blog posts by email (just add your details to the upper right side of this page).

Event Data: open for your interpretation

Madeleine Watson — Thu, 25 Feb 2016 00:00:00 +0000

What happens to a research work outside of the formal literature? That’s what Event Data will aim to answer when the service launches later this year.

Following the successful DOI Event Tracker pilot in Spring 2014, development has been underway to build our new service, newly re-named Crossref Event Data. It’s an open data service that registers online activity (specifically, events) associated with Crossref metadata. Event Data will collect and store a record of any activity surrounding a research work from a defined set of web sources. The data will be made available as part of our metadata search service or via our Metadata API and normalised across a diverse set of sources. Data will be open, audit-able and replicable.

We expect to include the following sources at the launch of the clearinghouse in Q3 (pending final confirmation):

[table id=1 /]

What could you achieve?

Anyone interested in metrics and analytics will have direct and open access to a single collection of DOI activity data of events occurring outside of the formal literature. As Event Data records are time-stamped, you can be assured that the data you receive is both auditable and replicable. Collected and stored by Crossref in the one location, we invite researchers, publishers, funders and altmetrics providers to consider the possibilities Event Data offers to enrich and expand your work.

With such a corpus of open, transferable and auditable raw data at your fingertips, what could you achieve?

General and altmetrics service providers

Crossref Event Data is a centrally-managed resource, therefore as a third party vendor you will have the ability to collect real-time data from a central location to enrich, analyze, interpret and report via your own tools. Using our API, you will gain regular access to our collection of raw, auditable data to feed into your own tools and services ready for aggregation and analysis. Additionally, the optional benefit of an SLA with Crossref will ensure that your clients have access to a reliable and flexible source of event data.

Journal editors

Using the data collected in our service, as an editor you can attract authors by offering data on the audience’s research interest, track the full-scope of article dissemination and gain a better understanding of how the publications you manage compare to each other. By analysing the Event Data records, you can quickly find reviewers based on publication network analysis, identify new areas to grow author submissions and track the reach of submissions selected for publication.

Funders

As a Funder, you can use Event Data to isolate and track the dissemination and usage of the research you funded outside of the scholarly literature. As the data is portable, you can be assured that should a journal move, your ability to track its dissemination moves with it. Using the Event Data records collection, you can:

Efficiently track progress of the research impact of grant awardees in an automated fashion, with the signals most relevant to your organisation

Develop measurements of research engagement at the article level which reflect your mission and current funding priorities

Gain visibility into the potent success stories highlighting the impact of your work for your development campaigns

Analyze trends of past and future funding programs

More effectively pursue your funding strategy and manage your portfolio based on data-driven decision making.

Publishers and publishing platforms

By analyzing and interpreting the Event Data collection, as a publisher or content distributor you can use the records to undertake the following metric-lead analysis to help drive your business needs:

Conduct more robust publication growth analysis across titles, subject areas, or all published literature

Gain a balanced understanding of the engagement on your publications across subject areas, titles, or managing editors

Enhance author services (personalization, content discovery, profile management, etc.)

Focused and data-driven product development of tools and services to drive audience engagement

Provide content distributors data on downstream reach of publications.

Bibliometricians

Event Data heavily supports Bibliometric research by facilitating the tracking of DOI-related research activity across different platforms and channels. As a Bibliometrician, use trusted raw data as the underlying data for your research, which you can easily obtain from Crossref in a single, normalized format across a variety of sources. Additionally, as Event Data data is replicable, portable and auditable, you will be assured of high quality results in your research projects.

Research institutions

All of the stakeholders in your institution, from the research, development and marketing offices to the researchers themselves, will benefit from access to data about where and how your research is being discussed in mainstream and social media. As a research institution, Event Data can help you:

Track dissemination of publications (types of channels, rate of growth, etc.) by members of the institution

Access up-to-date information on the research progress of faculty members, useful for tenure and promotion decisions

View data on downstream impact of publications

Roll up data for custom reporting of department’s research activities

Stay tuned, testing begins soon!

With development work on the MVP (Minimum Viable Product) scheduled to complete shortly, we will soon be releasing a small subset of data sources that are collecting event data as well as a testing environment for interested parties to explore a very preliminary version of the software as we continue to work towards implementation of the full Event Data clearinghouse release in Q3. Look out for our MVP announcement, with full technical specifications and confirmation of the selected initial pull and push sources, over the coming weeks.

Crossref & the Art of Cartography: an Open Map for Scholarly Communications

Jennifer Lin — Fri, 08 Jan 2016 00:00:00 +0000

In the 2015 Crossref Annual Meeting, I introduced a metaphor for the work that we do at Crossref. I re-present it here for broader discussion as this narrative continues to play a guiding role in the development of products and services this year.

Metadata enable connections

At Crossref, we make research outputs easy to find, cite, link, and assess through DOIs. Publishers register their publications and deposit metadata through a variety of channels (XML, CSV, PDF, manual entry), which we process and transform into Crossref XML for inclusion into our corpus. This data infrastructure which makes possible scholarly communications without restrictions on publisher, subject area, geography, etc. is far more than a reference list, index or directory.

If research builds on what came before, one could claim that the process of knowledge production is partly the story of the very relationships between results disseminated (i.e., publications). So let’s consider each publication as a node in a graph where each has a coordinate and is connected by its citations to other publications (as well those that cite it). Additionally, each is associated with a set of people and places, along with a whole host of elements involved in the research and dissemination process.

But take a wider berth, and we begin to capture relationships between all such contributing agents and objects involved in the research process. Here we find an array of entities belonging to the scholarly graph, including different types of research artifacts, publisher and journal, funders, ORCIDs, peer reviews, publication status updates (corrections, retractions, etc.), citations, license information, additional URLs (machine destinations, hosting platforms, etc.), underlying data, software and protocols, materials, discussions and blog posts, recommendations, reference work mentions, etc. The entities on the graph multiply at an even higher rate as researchers share more outputs across more channels. And over time, the graph expands exponentially, producing a webbing that is far more dense and far more vast than we can currently imagine. Perhaps even to the point we realize Borges’ story where a cartographer builds a map so large it replicates the territory itself (On Exactitude in Science)!

From graph to cartography

At the heart of Borges’s poignant story is the map. Crossref’s graph of scholarly communications could be seen in the same light. It has a representational aspect, which is not purely abstract and can be visualized. Here, a map becomes an incredibly potent metaphor. Each link enabled by publisher-deposited metadata is a new street, bridge, or highway that takes us to a particular place (i.e., entity) of interest. These roads lead to articles, researchers, funders, institutions, etc., and in doing so, make them discoverable. They tell a story about the roles of each in the broader research in the landscape dotted with a plethora of places.

The scholarly web has a growing corpus of more than 78 million publications at this very moment registered with Crossref. On average ten to fifteen thousand new objects appear every day. Maps are all the more essential for getting around in a bewildering environment of new and unfamiliar places, even for known ones in areas of exploding growth. They are critical for orienteering, discovering relationships, identifying sets of associated objects, naming new neighborhoods that emerge (i.e., new research specialties), etc. And if each connection on the map is seen as an event, maps can also represent micro-narratives about the research process and the agents involved. A multi-dimensional map containing all these entities, which serves as an evolving representation of spacetime that is constantly updated and always available, would finally begin to depict the process of scholarly activity as a dynamic, evolving, almost living system.

An open map for scholarly communication

Crossref builds such a scholarly map of the research enterprise and makes it openly available for the entire research ecosystem. Call this a meta map or, more recently, call it metastructure. No matter what name it goes by we call it infrastructure at Crossref.

Crossref’s open map for scholarly communications is a core part of the open information infrastructure for scholarly research. Crossref map data are open, portable, as well as licensed and provisioned for maximum reuse to serve the whole community. This open resource has two entrances: one for humans, another for machines. The Crossref REST API enables machines to traverse this environment and mine it in equal measure to the humans behind them. It is configured so that a robot can learn, a phone can access, and platforms can be built.

OpenStreetMap and Google Maps, both widely used and mature infrastructure maps, are instructive examples when we consider a map of this kind for scholarly communications. Map data can be represented in unlimited ways, depending on any variety of needs and users. Third parties can add content via interactive layers that tell different stories such as health expenditure by country based on GDP and coral reefs at risk. They have a broad base of users across business models from philanthropic services aimed at disaster relief (Refugeemaps.eu) to commercial entities providing drivers with locations on open parking spaces (AppyParking on Google Map, PocketParker on OpenStreetMap). They power platforms and services that build maps for others (MapQuest, MapBox). They have applications far beyond the business of maps. For example, Place picker is a Google Maps widget that supports easy auto-complete the entry of any place or location on a mobile app where typing is a chore. And as far use cases close to home, the two have served as raw data for academic research (ex: workflow for generating multi-agent traffic simulation scenarios, automatic classification of GPS trajectories for transportation modes, etc.).

In kind, the Crossref infrastructure map also supports: the development of any variety of new maps which re-present the data, the makers of map platforms that power the research enterprise, tools that use map data, as well as academic research (bibliometrics). We extract slices of data of common interest from the map and add them as additional layers by which anyone can access and create applications on or across these bands of data:

Contributors (authors, editors, reviewers)
Funding information (funding body, grant number)
Trial & study information (clinical trials registry number, registered report, replication study)
Publication history (versions, updates, revisions, corrections, retractions, dates received/accepted/published)
Peer review (status, type, reviews)
Access indicators (publication license for text & data mining, machine mining URLs)
Resources & associated research artifacts (preprints, figures & tables, datasets, software, protocols, research resource IDs)
Activity surrounding the publication (peer reviews, comments & discussions, bookmarks, social shares, recommendations).

Today, the map powers a host of public and commercial organisations alike for a wide range of scholarly and non-scholarly purposes:

<td style="border: 1px solid #ffffff;">
<ul>
<li>
<span >Indexing services</span>
</li>
<li>
<span >Publishing vendors</span>
</li>
<li>
<span >Peer review systems</span>
</li>
<li>
<span >Reference manager systems</span>
</li>
<li>
<span >Lab & diagnostics suppliers</span>
</li>
<li>
<span >Info management systems</span>
</li>
<li>
<span >Educational tools</span>
</li>
<li>
<span >Data analytics systems</span>
</li>
<li>
<span >Literature discovery services</span>
</li>
</ul>
</td>

Publishers
Funders
Research institutions
Archives & repositories
Research councils
Data centres
Professional networks
Patent offices
Registration Agencies

We will follow up this post to highlight a cross-section of these consumers in the Crossref map ecosystem and elaborate on what & how they have built from our data. An infrastructure map offers endless potential to third parties across publishers, funders, research institutions, and vendors working to serve the scholarly research enterprise.

The art of cartography

In the Crossref Product Management team, we have ambitious plans for map enhancements this year. They focus on expanding information density and ease of access to the data. In the former case, we will introduce a new class of locations where activity surrounding the publications are occurring when we launch the DOI Event Tracker. We will also initiate an extensive publisher campaign to achieve full metadata deposit completeness across our membership. No one can keep pace with the sheer volume of research activity happening online nor wander the Lonely Web of research alone. The more metadata publishers provide for a publication, the more roads lead to its map location. After all, discoverability is closely associated with connectedness on a map. And finally, in the latter case, we will refresh and enhance the user interface to make it more powerful for humans to traverse the ever-changing landscape (as easily as the REST API enables machines!).

I gratefully acknowledge the feedback received from the following who served as generous and insightful sounding boards: Virginia Barbour, Theo Bloom, Martin Eve, Daniel S. Katz, Amye Kenall, Catriona MacCullum, Cameron Neylon, Mark Patterson, Kristen Ratan, Carly Strasser, and Kaitlin Thaney.

Crossref Labs plays with the Raspberry Pi Zero

Joe Wass — Wed, 02 Dec 2015 00:00:00 +0000

If you’re anything like us at Crossref Labs (and we know some of you are) you would have been very excited about the launch of the Raspberry Pi Zero a couple of days ago. In case you missed it, this is a new edition of the tiny low-priced Raspberry Pi computer. Very tiny and very low-priced. At $5 we just had to have one, and ordered one before we knew exactly what we want to do with it. You would have done the same. Bad luck if it was out of stock.

We love the way DOIs are being used in Wikipedia, but you probably already know that by now. Not only is it a brilliant source of information, mostly well cited, it’s also an organic living thing, with countless people and bots working together on countless articles. Our live stream of edits that cite (or uncite) DOIs shows new scholarly literature unfold, as it happens. From new articles to new references to improved citations to edit wars to bots cleaning up all the mess, it captivates everyone we show it to. The latest version has a live chart to show exactly how much activity is going on.

Crossref works in five ways: Rally, Tag, Run, Play, and Make and this definitely comes under ‘Play’. By the time our Raspberry Pi Zero arrived it was clear what we had to do. We ordered a servo, a driver board and a wireless adapter and got to work.

We have some new neighbours in the basement. Oxford Hackspace is a community of people who want to work on projects from electronics to metalwork, hack things to improve them or find out how they work. A diverse bunch who at the last visit were working on squeezing unprecedented color capabilities from the 30 year old ZX Spectrum, a nixie tube display, a smartphone controlled doorbell and a robotic glockenspiel. They let us use their soldering iron to solder a few header pins.

A bit of hacky Python, a pictureframe and lots of duck tape later, we have a live display of how many DOIs are cited and uncited per hour. It updates live every minute, fetches the latest numbers from the Wikipedia DOI citation stream and moves the hand.

(For the worried engineers amongst you, rest assured that sufficient duck tape was added after this picture)

It’s extraordinary to think that a fully fledged computer with very capable specifications can be manufactured and sold for $5. Within the space of a lunchtime we had it up and running, all connected and fetching data over the internet via wireless. A generation ago you would have had to use punched cards, send them by post and load them in by hand. The live stream would have been at least a month behind.

It now sits in our Oxford office reminding us that DOIs Aren’t Just for Traditional Bibliographies. Below Geoff Bilder’s reminder about what happens when you have too many standards (they’re telephone plugs from round the world).

You can find source code and instructions on the github repository so you can make your own if you want.

DOIs in Reddit

Joe Wass — Wed, 30 Sep 2015 00:00:00 +0000

Skimming the headlines on Hacker News yesterday morning, I noticed something exciting. A dump of all the submissions to Reddit since 2006. “How many of those are DOIs?”, I thought. Reddit is a very broad community, but has some very interesting parts, including some great science communication. How much are DOIs used in Reddit?

(There has since been a discussion about this blog post on Hacker News)

We have a whole strategy for DOI Event Tracking, but nothing beats a quick hack or is more irresistible than a data dump.

What is a DOI?

If you know what a DOI is, skip this! The DOI system (Digital Object Identifier) is a link redirection service. When a publisher puts some content online they could just hand out the URL. But the URL can change, and within a very short space of time, link-rot happens. DOIs are designed to fight link rot. When a publisher mints a DOI to an article they just published, they can change the article’s URL and then update the DOI to point to the new place. DOIs are persistent. They are URLs. They’re also identifiers (kind of like ISBNs), and they’re used in scholarly publishing as to do citations.

Crossref is the DOI registration agency for scholarly publishing. That means mostly things like journal articles. There are other registration agencies, for example, DataCite, who do DOIs for research datasets. But at this point in time, most DOIs are Crossref’s.

What does finding DOIs in Reddit mean?

It means someone used a DOI to cite something! DOIs can be used for any kind of content, but because of the sheer volume of scientific publishing, lots of DOIs are for science. Having a DOI doesn’t say anything about quality or content. But it does indicate that the person who created the DOI probably intended it to be cited. We care because it means that every time a DOI is used a tiny bit of link-rot doesn’t have the opportunity to take hold. Every time something is discussed on Reddit and the DOI is used, it means that archaeologists using the data dump in 100 years will have identifiers to find the things being discussed, even if the web and URLs have long since crumbled to dust.

Or, more likely, in five year’s time when a few URLs will have shuffled around.

The results

DOIs have been used on Reddit since 2008 (the logs start in 2006). After a rocky start, we see hundreds being used per year.

That’s dozens per month.

The best subreddit to find DOIs is /r/Scholar, followed by /r/science. And then a lot of others with one or two per year.

Opportunities

It’s great to see DOIs being used in Reddit. But let’s be honest, it’s not a massive amount.

We have a list of domains that our DOIs point to. They mostly belong to publishers, so every time we see a link to a domain on the list, there’s a chance (not a certainty) that the link could have been made using a DOI. We found a large number of these, orders of magnitude more than DOIs. We’re still crunching the data.

The data

The data is quite large. It’s a 40 Gigabyte download compressed, which comes to about 170 GB that uncompressed. It contains the submissions to reddit between 2006 and 2015, not the comments, so each data point represents a thread of conversation about a DOI.

Reproducibility (updated)

You can find the source code and reproduce the figures at http://github.com/crossref/reddit-dump-experiment. We use Apache Spark for this kind of thing.

The data and methodology are very experimental. You can download all results here:

https://s3-eu-west-1.amazonaws.com/crossref-labs-data/2015-10-06/reddit-dump-experiment.zip

It includes all data for charts in this post, as well as the full list of DOIs, the full list of URLs that could possibly have DOIs, and the full JSON input line for each of these.

More info

Read about our DOI Event Tracking strategy, including our live stream of Wikipedia citations.

DOI Event Tracker (DET): Pilot progresses and is poised for launch

Jennifer Lin — Tue, 15 Sep 2015 00:00:00 +0000

Publishers, researchers, funders, institutions and technology providers are all interested in better understanding how scholarly research is used. Scholarly content has always been discussed by scholars outside the formal literature and by others beyond the academic community. We need a way to monitor and distribute this valuable information.

The Crossref DOI Event Tracker (DET)

To meet this need, Crossref will be introducing a new service that tracks activity surrounding a research work from potentially any web source where an event is associated with a DOI. Following a successful pilot run started Spring 2014, the service has been approved to move toward production and is expected to launch in 2016. Any party wishing to join this phase is welcome to contact Jennifer Lin. The DOI Event Tracker (DET) registers a wide variety of events such as bookmarks, comments, social shares, citations, and links to other research entities, from a growing list of online sources. DET aggregates them, and stores and delivers the data in many ways.

Open, portable, and licensed for maximum reuse

Crossref has long served as the citation linking and metadata infrastructure provider for scholarly communication; the new DOI Event Tracker is a natural next step, providing a practical solution as a resource for the whole community. The tracker offers the following features:

Data on event activity across a common pool of online channels.
Near real-time alerting for select sources with push notifications to the system.
Cross-publisher monitoring to enable benchmarking and provide context to the data.
Common format for normalizing data results across the diverse set of sources via modern REST API.
Secure and regularly refreshed backups of critical data for long term data preservation.
Transparency of data collection so as to ensure auditable, replicable, and trustworthy results.
Query-initiated retrieval or real-time alerts when an event of interest occurs.
CC-0 license for open and flexible propagation of data.

A number of platforms are already confirmed and more parties are welcomed at any stage. So far we have confirmation to track DOI events on the following platforms:

[table id=1 /]

This set of sources reflects our initial focus on parties willing to allow their data to be redistributed in the common pool. Efforts are underway to expand the source list to include Twitter and MyScienceWork, among others. Publishers can also act as sources by publishing and distributing DOI event data via the DET when an event occurs on its platform (for example, when a PDF is downloaded, or when a comment mentions a DOI in a locally hosted discussion forum, etc.). This would make local DOI activity globally available to funders, researchers, institutions, etc.

DET provides benefits of scale and ease of access as a central point for collecting and propagating data to the community. As a single point of access, it overcomes the business and technical hurdles that are a part of managing multiple online sources where scholarly activity occurs, in a rapidly changing landscape of online channels. This resource covers content across publishers and serves as a strong foundation to support the development of tools and services by any party. DET users will always be able to combine the DET data with those individually collected via negotiated or paid access. DET remains a utility separate from any value-added amenities, such as analytics, presentation, and reporting.

DET Service-Level Agreement

For those who seek the highest level of service and a more flexible range of access options, Crossref will provide a Service-Level Agreement (SLA) service for the DOI Event Tracker. The DET SLA includes the following additional features on top of the common data offering:

Access to the complete suite of sources, which includes restricted and/or paid sources in addition to common data, providing the fullest picture of DOI usage activity possible.
Guaranteed uptime and response time to the latest raw data on the aggregate activity surrounding a DOI.
Guaranteed support response time to questions and issues surrounding data and data delivery.
Flexible data access options: on-demand real time data access and scheduled bulk downloads for processing batch analytics.
Optimum retrieval rates and accelerated delivery speeds with the dedicated SLA API.
Access to a webhook API for events of interest as an alternative to polling DET.
Standardized and enhanced linkback service for the difficult-to-track, grey literature.

The DET SLA service has a simple, value-based pricing model based on subscriber size. Register your interest in Crossref’s DOI Event Tracker and the DET SLA service if you would like stay informed of the upcoming launch. Please contact Jennifer Lin for more information.

Image modified from “Radar” icon by Karsten Barnett from the Noun Project.

Real-time Stream of DOIs being cited in Wikipedia

Joe Wass — Tue, 03 Mar 2015 00:00:00 +0000

TL;DR

Watch a real-time stream of DOIs being cited (and “un-cited!” ) in Wikipedia articles across the world: https://live-eventdata-crossref-org.pluma.sjfc.edu/live.html

Background

For years we’ve known that the Wikipedia was a major referrer of Crossref DOIs and about a year ago we confirmed that, in fact, the Wikipedia is the 8th largest refer of Crossref DOIs. We know that people follow the DOIs, too. This despite a fraction of Wikipedia citations to the scholarly literature even using DOIs. So back in August we decided to create a Wikimedia Ambassador programme. The goal of the programme was to promote the use of persistent identifiers in citation and attribution in Wikipedia articles. We would do this through outreach and through the development of better citation-related tools.

Remember when we originally wrote about our experiments with the PLOS ALM code and how that has transitioned into the DOI Event Tracking Pilot? In those posts we mentioned that one of the hurdles in gathering information about DOI events is the actual process of polling third party APIs for activity related to millions of DOIs. Most parties simply wouldn’t be willing handle the load of a 100K API calls an hour. Besides, polling is a tremendously inefficient process, only a fraction of DOIs are ever going to generate events, but we’d have to poll for each of them, repeatedly, forever, to get an accurate picture of DOI activity. We needed a better way. We needed to see if we could reverse this process and convince some parties to instead “push” us information whenever they saw DOI related events (e.g. citations, downloads, shares, etc). If only we could convince somebody to try this…

Wikipedia DOI Events

In December 2014 we took the opportunity of the 2014 PLOS/Crossref ALM Workshop in San Francisco too meet with Max Klein and Anthony Di Franco where we kicked off a very exciting project.

There’s always someone editing a Wikipedia somewhere in the world. In fact, you can see a dizzying live stream of edits. We thought that given that there are so many DOIs in Wikipedia, that live stream may contain some diamonds (DOIs are made of diamond, that’s how they can be persistent). Max and Anthony went away and came back with a demo that contains a surprising amount of DOI activity.

That demo is evolving into a concrete service, called Cocytus. It is running at Wikimedia Labs monitoring live edits as you read this.

For now we’re feeding that data into the DOI Events Collection app (which is an off-shoot of the Chronograph project). We are in the process of modifying the Lagotto code so that we can instead push those events into the DOI Event Tracking Instance.

The first DOI event we noticed was delightfully prosaic: The DOI for “The polymath project” is cited by the Wikipedia page for “Polymath Project”. Prosaic perhaps, but the authors of that paper probably want to know. Maybe they can help edit the page.

Or how about this. Someone wrote a a paper about why people edit Wikipedia and then it was cited by Wikipedia. And then the citation was removed. The plot thickens…

We’re interested in seeing how DOIs are used outside of the formal scholarly literature. What does that mean? We don’t fully know, that’s the point. We have retractions in scholarly literature (and our Crossmark metadata and service allow publishers to record that), but it’s a bit different on Wikipedia. Edit wars are fought over … well you can see for yourself.

Citations can slip in and out of articles. We saw the DOI 10.1001/archpediatrics.2011.832 deleted from “Bipolar disorder in children”. If we’d not been monitoring the live feed (we had considered analysing snapshots of the Wikipedia in bulk) we might never have seen that. This is part of what non-traditional citations means, and it wasn’t obvious until we’d seen it.

You can see this activity on the Chronograph’s stream. Or check your favourite DOI. Please be aware that we’re only collecting newly added citations as of today. We do intend to go back and back-fill, but that may take some time- as it * cough * requires polling again.

Some Technical Things

A few interesting things that happened as a result of all this:

Secure URLs

SSL and HTTPS were invented so you could do things like banking on the web without fear of interception or tampering. As the web becomes a more important part of life, many sites are upgrading from HTTP to HTTPS, the secure version. This is not only because your confidential details may be tampered with, but because certain governments might not like you reading certain materials.

Because of this, some time ago, Wikipedia decided to embark on an upgrade to HTTPS last year, and they are a certain way along the path. The IDF, who are responsible for running the DOI system, upgraded to HTTPS this Summer, although most DOIs are referred to by HTTP still.

We met with Dario Taraborelli at the ALM workshop and discussed the DOI referral data that is fed into the Chronograph. We put two and two together and realised that Wikipedia was linking to DOIs (which are mostly HTTP) from pages which might be served over HTTPS. New policies in HTML5 specify that referrer URL headers shouldn’t be sent from HTTPS to HTTP (in case there was something secret in them). The upshot of this is, if someone’s browsing Wikipedia via HTTPS and click on a normal DOI, we won’t know that the user came from Wikipedia. Not a huge problem today, but as Wikipedia switches over to entirely secure, we’re going to miss out on very useful information.

Fortunately, the HTML5 specification includes a way to fix this (without leaking sensitive information). We discussed this with Dario, and he did some research, and came up with a suggestion, which got discussed. It’s fascinating to watch a democratic process like this take place and take part in it.

We’re waiting to see how the discussion turns out, and hope that it all works out so we can continue to report on how amazing Wikipedia is at sending people to scholarly literature.

How shall I cite thee?

Another discussion grew out of that process, and we started talking to a Wikipedian called Nemo (note to Latin scholars: we weren’t just talking to ourselves). Nemo (real name Federico Leva) had a few suggestions of his own. Another way to solve the referrer problem is by using HTTPS URLs (HTML5 allows browsers to send the referrer domain when going from HTTPS to HTTPS).

This means going back to all the articles that use DOIs and change them from HTTP to HTTPS. Not as simple as it sounds, and it doesn’t sound simple. We started looking into how DOIs were cited on Wikipedia.

After some research we found that there are more ways that we expected to cite DOIs.

First, there’s the URL. You can see it in action in this article. URLs can take various forms.

Second there’s the official template tag, seen in action here:

<ref name="SCI-20140731">{{cite journal |title=Sustained miniaturization and anatomical innovation in the dinosaurian ancestors of birds |url=http://www.sciencemag.org.pluma.sjfc.edu/content/345/6196/562 |date=1 August 2014 |journal=[[Science (journal)|Science]] |volume=345 |issue=6196 |pages=562–566 |doi=10.1126/science.1252243 |accessdate=2 August 2014 |last1=Lee |first1=Michael S. Y. |first2=Andrea|last2=Cau |first3=Darren|last3=Naish|first4=Gareth J.|last4=Dyke}}</ref>

There’s a DOI in there somewhere. This is the best way to cite DOIs, firstly as it’s actually a proper traditional citation and there’s nothing magic about DOIs, secondly because it’s a template tag and can be re-rendered to look slightly different if needed.

Third there’s the old official DOI template tag that’s now discouraged:

<ref name="Example2006">{{Cite doi|10.1146/annurev.earth.33.092203.122621}}</ref>

And then there’s another one.

{{doi|10.5555/123456789}}

Knowing all this helps us find DOIs. But if we want to convert DOIs links in Wikipedia to use HTTPS, it means that there are more template tags to modify and more pages to re-render.

Nemo also put DOIs on the Interwiki Map which should make automatically changing some of the URLs a lot easier.

We’re very grateful to Nemo for his suggestions and work on this. We’ll report back!

The elephant in the room

Those of you who know how DOIs work will have spotted an unsecured elephant in the room. When you visit a DOI, you visit the URL, which hits the DOI resolver proxy server, which returns a message to your browser to redirect to the landing page on the publisher’s site.

Securely talking to the DOI resolver by using HTTPS instead of HTTP means that no-one can eavesdrop and see which DOI you are visiting, or tamper with the result and send you off to a different page. But the page you are sent to will be, in nearly all cases, still HTTP. Upgrading infrastructure isn’t trivial, and, with over 4000 members (mostly publishers), most Crossref DOIs will still redirect to standard HTTP pages for the foreseeable future.

You can keep as secure as possible by using HTTPS Everywhere.

Fin

There’s lots going on, watch this space to see developments. Thanks for reading this, and all the links. We’d love to know what you think.

Bootnote

Not long after this blog post was published we saw something very interesting.

That’s no DOI. We like interesting things, but they can panic us. This turned out to be a great example of why this kind of thing can be useful. A minute’s digging and we found the article edit:

It turns out that this was a typo: someone put a title when they should have put in a DOI. And, as the event shows, this was removed from the Wikipedia article.

Crossref’s DOI Event Tracker Pilot

Geoffrey Bilder — Mon, 02 Mar 2015 00:00:00 +0000

TL;DR

Crossref’s “DOI Event Tracker Pilot”- 11 million+ DOIs & 64 million+ events. You can play with it at: http://goo.gl/OxImJa

Tracking DOI Events

So have you been wondering what we’ve been doing since we posted about the experiments we were conducting using PLOS’s open source ALM code? A lot, it turns out. About a week after our post, we were contacted by a group of our members from OASPA who expressed an interest in working with the system. Apparently they were all about to conduct similar experiments using the ALM code, and they thought that it might be more efficient and interesting if they did so together using our installation. Yippee. Publishers working together. That’s what we’re all about.

So we convened the interested parties and had a meeting to discuss what problems they were trying to solve and how Crossref might be able to help them. That early meeting came to a consensus on a number of issues:

The group was interested in exploring the role Crossref could play in providing an open, common infrastructure to track activities around DOIs, they were not interested in having Crossref play a role in the value-add services of reporting on an interpreting the meaning of said activities.
The working group needed representatives from multiple stakeholders in the industry. Not just open access publishers from OASPA, but from subscription based publishers, funders, researchers and third party service providers as well.
That it was desirable to conduct a pilot to see if the proposed approach was both technically feasible and financially sustainable.

And so after that meeting, the “experiment” graduated to becoming a “pilot.” This Crossref pilot is based on the premise that the infrastructure involved in tracking common information about “DOI events” can be usefully separated from the value-added services of analysing and presenting these events in the form of qualitative indicators. There are many forms of events and interactions which may be of interest. Service providers will wish to analyse, aggregate and present those in a range of different ways depending on the customer and their problem. The capture of the underlying events can be kept separate from those services.

In order to ensure that the Crossref pilot is not mistaken for some sub rosa attempt to establish new metrics for evaluating scholarly output, we also decided eschew any moniker that includes the word “metrics” or synonyms. So the “ALM Experiment” is dead. Long live the “”DOI Event Tracker” (DET) pilot. Similarly PLOS’s open source “ALM software” has been resurrected under the name “Lagotto.”

The Technical Issues

Crossref members are interested in knowing about “events” relating to the DOIs that identify their content. But our members face a now-classic problem. There are a large number of sources for scholarly publications (3k+ Crossref members) and that list is still growing. Similarly, there are an unbounded number of potential sources for usage information. For example:

Supplemental and grey literature (e.g. data, software, working papers)
Orthogonal professional literature (e.g. patents, legal documents, governmental/NGO/IGO reports, consultation reports, professional trade literature).
Scholarly tools (e.g. citation management systems, text and data mining applications).
Secondary outlets for scholarly literature (institutional and disciplinary repositories, A&I services).
Mainstream media (e.g. BBC, New York Times).
Social media (e.g. Wikipedia, Twitter, Facebook, Blogs, Yo).

Finally, there is a broad and growing audience of stakeholders who are interested in seeing how the literature is being used. The audience includes publishers themselves as well as funders, researchers, institutions, policy makers and citizens.

Publishers (or other stakeholders) could conceivably each choose to run their own system to collect this information and redistribute it to interested parties. Or they can work with a vendor to do the same. But either case, they would face the following problems:

The N sources will change. New ones will emerge. Old ones will vanish.
The N audiences will change. New ones will emerge. Old ones will vanish.
Each publisher/vendor will need to deal with N source’s different APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the publishers/vendors and for the sources.
Each audience will need to deal with N publisher/vendor APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the audiences and for the publishers.
If publishers/vendors use different systems which in turn look at different sources, it will be difficult to compare or audit results across publishers/vendors.
If a journal moves from one publisher to another, then how are the metrics for that journal’s articles going to follow the journal?

And then there is the simple issue of scale. Most parties will be interested in comparing the data that they collect for their own content, with data about their competitors. Hence, if they all run their own system, they will each be querying much more than their own data. If, for example, just the commercial third-party providers were interested in collecting data covering the formal scholarly literature, they would each find themselves querying the same sources for the same 80 million DOIs. To put this into perspective, to refresh the data for 10 million DOIs once a month, would require sources to support ~ 14K API calls an hour. 60 million DOIs would require 100K API calls an hour. Current standard API caps for many of the sources that people are interested in querying hover around 2K per hour. We may see these sources lift that cap for exceptional cases, but they are unlikely to do so for many different clients all of whom are querying essentially the same thing.

These issues typify the “multiple bilateral relationships” problem that Crossref was founded to try and ameliorate. When we have many organisations trying to access the exact same APIs to process the exact same data (albeit to different ends), then it seems likely that Crossref could help make the process more efficient.

Piloting A Proposed Solution

The Crossref DET pilot aims to show the feasibility of providing a hub for the collection, storage and propagation of DOI events from multiple sources to multiple audiences.

Data Collection

Pull: DET will collect DOI event data from sources that are of common interest to the membership, but which are unlikely to make special efforts to accommodate the scholarly communications industry. Examples of this class of source include large, broadly popular services like FaceBook, Twitter, VK, Sina Weibo, etc.
Push: DET will allow sources to send DOI event data directly to Crossref in one of three ways:
- Standard Linkback: Using standards that are widely used on the web. This will automatically enable linkback-aware systems like WordPress, Moveable Type, etc. to alert DET to DOI events.
- Scholarly Linkback: A to-be-defined augmented linkback-style API which will be optimized to work with scholarly resources and which will allow for more sophisticated payloads including other identifiers (e.g. ORCIDs, FundRefs), metadata, provenance information and authorization information. This system could be used by tools designed for scholarly communications. So, for example, it could be used by publisher platforms to distribute events related to downloads or comments within their discussion forums. It could also be used by third party scholarly apps like Zotero, Mendeley, Papers, Authorea, IRUS-UK, etc. in order to alert interested parties in events related to specific DOIs.
- Redirect: DET will also be able to serve as a service discovery layer that will allow sources to push DOI event data directly to an appropriate publisher-controlled endpoint using the above scholarly linkback mechanism. This can be used by sources like repositories in order to send sensitive usage data directly to the relevant publishers.

Data Propagation

Parties may want to use the DET in order to propagate information about DOI events. The system will support two broad data propagation patterns:

one-to-many: DOI events that are commonly harvested (pulled) by the DET system from a single source will be distributed freely to anybody who queries the DET API. Similarly, sources that push DOI events via the standard or scholarly linkback mechanisms, will also propagate their DOI events openly to anybody who queries the DET API. DOI events that are propagated in either of these cases will be kept and logged by the DET system along with appropriate provenance information. This will be the most common, default propagation model for the DET system.
one-to-one: Sources of DOI events can also report (push) DOI event data directly to owner of the relevant DOI if the DOI owner provides & registers a suitable end-point with the DET system. In these cases, data sources seeking to report information relating to a DOI, will be redirected (with a suitable 30X HTTP status and relevant headers) to the end-point specified by the DOI owner. The DET system will not keep the request or provenance information. One-to-one propagation model is designed to handle use cases where the source of the DOI event has put restrictions on the data and will only share the DOI events with the owner (registrant) of the DOI. This use case may be used, for example, by aggregators or A&I services that want to report confidential data directly back to a publisher. The advantage of the redirect mechanism is that Crossref is not put into the position of having to secure sensitive data as said data will never reside on Crossref systems.

Note that the two patterns can be combined. So, for example, a publisher might want to have public social media events reported to the DET and propagated accordingly, but to also to private third parties report confidential information directly to the publisher.

So Where Are We?

So to start with, the DET Working Group has grown substantially since the early days and we have representatives from a wide variety of stakeholders. The group includes:

Cameron Neylon, PLOS
Chris Shillum, Elsevier
Dom Mitchell, Co-action Publishing
Euan Adie, Altmetric
Jennifer Lin, PLOS
Juan Pablo Alperin, PKP
Kevin Dolby, Wellcome Trust
Liz Ferguson, Wiley
Maciej Rymarz, Mendeley
Mark Patterson, eLife
Martin Fenner, PLOS
Mike Thelwell, U Wolverhampton
Rachel Craven, BMC
Richard O’Beirne, OUP
Ruth Ivimey-Cook, eLife
Victoria Rao, Elsevier

As well as the usual contingent of Crossref cat-herders including: Geoffrey Bilder, Rachael Lammey & Joe Wass.

When we announced the then-DET experiment, we said that one of the biggest challenges would be to create something that scaled to industry levels. At launch, we only loaded in about 317,500+ Crossref DOIs representing publications from 2014 and we could see the system was going to struggle. Since then Martin Fenner and Jennifer Lin at PLOS have been focusing on making sure that the Lagotto code scales appropriately and now it is currently humming along with just over 11.5 million DOIs for which we’ve gathered over 64 million “events.” We aren’t worried about scalability on that front any more.

We’ve also shown that third parties should be able to access the API to provide value added reporting and metrics. As a demonstration of this, PLOS configured a copy of its reporting software “Parascope” to point at the Crossref DET instance. The next step we’re taking is to start testing the “push” API mechanism and the “point-to-point redirect” API mechanism. For the push API, we should have a really exciting demo available to show within the next few days. And on the point-to-point redirect, we have a sub-group exploring how the point-to-point redirect mechanism could potentially be used for reporting COUNTER stats as a compliment to the Sushi initiative.

The other major outstanding task we have before us is to calculate what the costs will be of running the DET system as a production service. In this case we expect to have some pretty accurate data to go on as we will have had close to half a year of running the pilot with a non-trivial number of DOIs and sources. Note that the work group is concerned to ensure that the underlying data from the system remains open to all. Keeping this raw data open as seen as critical to establishing trust in the metrics and reporting systems that third parties build on the data. The group has also committed to leaving the creation of value-add services to third parties. As such we have been focusing on exploring business models based around service-level-agreement backed versions of the API to complement the free version of the same API. The free API will come with no guarantees of uptime, performance characteristics or support. For those users that depend on the API in order to deliver their services, we will offer paid-for SLA-backed versions of the free APIs. We can then configure our systems so that we can independently scale these SLA-backed APIs in order to meet SLA agreements.

Our goal is to have these calculations complete in time for the working group to make a recommendation to the Crossref board meeting in July 2015.

Until then, we’ll use CrossTech as a venue for notifying people when we’ve hit new milestones or added new capabilities to the DET Pilot system.

Introducing the Crossref Labs DOI Chronograph

Joe Wass — Mon, 12 Jan 2015 00:00:00 +0000

tl;dr http://chronograph.labs.crossref.org.pluma.sjfc.edu

At Crossref we mint DOIs for publications and send them out into the world, but we like to hear how they’re getting on out there. Obviously, DOIs are used heavily within the formal scholarly literature and for citations, but they’re increasingly being used outside of formal publications in places we didn’t expect. With our DOI Event Tracking / ALM pilot project we’re collecting information about how DOIs are mentioned on the open web to try and build a picture about new methods of citation.

As part of the preparation for collaborating with Wikipedia, we looked at our statistics about when DOIs are clicked and discovered that Wikipedia was, over a two year period from 2012, the eighth largest referrer of DOIs. This means that not only does Wikipedia have a lot of DOIs, but people click them too. This bit of one-off data analysis (which surprised us) gave us enough of a prod to kickstart our collaboration with Wikipedia.

At the ALM Workshop 2014 in San Francisco we talked to some Wikipedians and bibliometricians and realised that we were sitting on a really interesting data-set and that it would be churlish not to share it. At the hackathon (read the report here) we started work on a service to gather information about DOIs and, a month later, we’re ready to unveil the DOI Chronograph.

Show me the goods

You can see:

Daily referrals (clicks) from top level domains, e.g. Wikipedia.org: http://chronograph.labs.crossref.org.pluma.sjfc.edu/domain.html?domain=wikipedia.org

Daily referrals from specific subdomains, e.g. fr.wikipedia.org: http://chronograph.labs.crossref.org.pluma.sjfc.edu/domain.html?domain=fr.wikipedia.org

Daily resolutions per DOI: http://chronograph.labs.crossref.org.pluma.sjfc.edu/doi.html?doi=10.1787%2F20752288

And, the chart that kicked this all off: DOI referring domains league tables. This shows that Wikipedia is the 3rd or 4th non-traditional referrer of DOIs (i.e. excluding referrals from Publishers’ domains): http://chronograph.labs.crossref.org.pluma.sjfc.edu/top.html

Try it out

Visit the Chronograph and give it a try chronograph.labs.crossref.org on your favourite DOI (everyone has one).

More data

Talking to a bibliometrician we also realised we can correlate other data for DOIs. We’re getting the issue date (approximately the publication date) from our own metadata, as well as the date that the Crossref metadata was updated. This gives interesting results, like the resolutions for 10.1038/ncomms2953, which peak after publication and then tails off. We are attempting to collect the following information:

daily resolution counts
day on which resolution was first successful
day on which it’s possible to resolve the DOI (we’ve got a bot running for new publications)
day on which the publisher says the article was published
day on which the metadata was most recently deposited with us
day on which the metadata was first deposited with us

We’re not there yet, but we’ve made a start and we’ve already got some pretty interesting data!

Weasel words

It’s a labs project so the usual weasel words apply. Specifically, we currently have the logs for 2012 to 2014 (we’re working at digging out the rest), and the referral information for 50 million DOIs (out of 71 million). That number will be higher by the time you read this. If your page is slow to load, be patient, as it’s currently working hard crunching numbers.

This project is focused on exploring the use of DOIs outside of the formal literature. As such, we are only looking at referrals from domains that do not appear to belong to primary publishers (i.e. our members). If you try a domain and it doesn’t work, it could be that the domain belongs to one of our members. If you’ve notice any mistakes, please email us at labs@crossref.org .

Finally, these numbers contain all DOI resolutions. That’s human clicks but also content negotiation to retrieve metadata, robots etc. We might try to filter them in future, but for now be aware that not every visitor is a human.

I’ll detail some of the the technical stuff (it’s very interesting) and what happened next with Wikipedia in a future post. Watch this space.

Many Metrics. Such Data. Wow.

Geoffrey Bilder — Mon, 24 Feb 2014 00:00:00 +0000

[

Crossref Labs loves to be the last to jump on an internet trend, so what better than than to combine the Doge meme with altmetrics?

Note: The API calls below have been superceeded with the development of the Event Data project. See the latest API documentation for equivalent functionality

Want to know how many times a Crossref DOI is cited by the Wikipedia?

http://det.labs.crossref.org.pluma.sjfc.edu/works/doi/10.1371/journal.pone.0086859

Or how many times one has been mentioned in Europe PubMed Central?

http://det.labs.crossref.org.pluma.sjfc.edu/works/doi/10.1016/j.neuropsychologia.2013.10.021

Or DataCite?

http://det.labs.crossref.org.pluma.sjfc.edu/works/doi/10.1111/jeb.12289

Background

Back in 2011 PLOS released its awesome ALM system as open source software (OSS). At Crossref Labs, we thought it might be interesting to see what would happen if we ran our own instance of the system and loaded it up with a few Crossref DOIs. So we did. And the code fell over. Oops. Somehow it didn’t like dealing with 10 million DOIs. Funny that.

But the beauty of OSS is that we were able to work with PLOS to scale the code to handle our volume of data. Crossref contracted with Cottage Labs and we both worked with PLOS to make changes to the system. These eventually got fed back into the main ALM source on Github. Now everybody benefits from our work. Yay for OSS.

So if you want to know technical details, skip to Details for Propellerheads. But if you want to know why we did this, and what we plan to do with it, read on.

Why?

There are (cough) some problems in our industry that we can best solve with shared infrastructure. When publishers first put scholarly content online, they used to make bilateral reference linking agreements. These agreements allowed them to link citations using each other’s proprietary reference linking APIs. But this system didn’t scale. It was too time-consuming to negotiate all the agreements needed to link to other publishers. And linking through many proprietary citation APIs was too complex and too fragile. So the industry founded Crossref to create a common, cross-publisher citation linking API. Crossref has since obviated the need for bilateral linking arrangements.

So-called altmetrics look like they might have similar characteristics. You have ~4000 Crossref member publishers and N sources (e.g. Twitter, Mendeley, Facebook, CiteULike, etc.) where people use (e.g. discuss, bookmark, annotate, etc.) scholarly publications. Publishers could conceivably each choose to run their own system to collect this information. But if they did, they would face the following problems:

The N sources will be volatile. New ones will emerge. Old ones will vanish.
Each publisher will need to deal with each source’s different APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the publishers and for the sources.
If publishers use different systems which in turn look at different sources, it will be difficult to compare results across publishers.
If a journal moves from one publisher to another, then how are the metrics for that journal’s articles going to follow the journal? This isn’t a complete list, but it shows that there might be some virtue in publishers sharing an infrastructure for collecting this data. But what about commercial providers? Couldn’t they provide these ALM services? Of course - and some of them currently do. But normally they look on the actual collection of this data as a means to an end. The real value they provide is in the analysis, reporting and tools that they build on top of the data. Crossref has no interest in building front-ends to this data. If there is a role for us to play here, it is simply in the collection and distribution of the data.

No, really, WHY?

Aren’t these altmetrics an ill-conceived and meretricious idea? By providing this kind of information, isn’t Crossref just encouraging feckless, neoliberal university administrators to hasten academia’s slide into a Stakhanovite dystopia? Can’t these systems be gamed?

FOR THE LOVE OF FSM, WHY IS CROSSREF DABBLING IN SOMETHING OF SUCH QUESTIONABLE VALUE?

takes deep breath. wipes spittle from beard

These are all serious concerns. Goodhart’s Law and all that… If a university’s appointments and promotion committee is largely swayed by Impact Factor, it won’t improve a thing if they substitute or supplement Impact Factor with altmetrics. Amy Brand has repeatedly pointed out, the best institutions simply don’t use metrics this way at all (PowerPoint presentation). They know better.

But yes, it is still likely that some powerful people will come to lazy conclusions based on altmetrics. And following that, other lazy, unscrupulous and opportunistic people will attempt to game said metrics. We may even see an industry emerge to exploit this mess and provide the scholarly equivalent of SEO. Feh. Now I’m depressed and I need a drink.

So again, why is Crossref doing this? Though we have our doubts about how effective altmetrics will be in evaluating the quality of content, we do believe that they are a useful tool for understanding how scholarly content is used and interpreted. The most eloquent arguments against altmetrics for measuring quality, inadvertently make the case for altmetrics as a tool for monitoring attention.

Critics of altmetrics point out that much of the attention that research receives outside of formal scholarly communications channels can be ascribed to:

Puffery. Researchers and/or university/publisher “PR wonks” over-promoting research results.
Innocent misinterpretation. A lay audience simply doesn’t understand the research results.
Deliberate misinterpretation. Ideologues misrepresent research results to support their agendas.
Salaciousness. The research appears to be about sex, drugs, crime, video games or other popular bogeymen.
Neurobollocks. A category unto itself these days.

In short, scholarly research might be misinterpreted. Shock horror. Ban all metrics. Whew. That won’t happen again.

Scholarly research has always been discussed outside of formal scholarly venues. Both by scholars themselves and by interested laity. Sometimes these discussions advance the scientific cause. Sometimes they undermine it. The University of Utah didn’t depend on widespread Internet access or social networks to promote yet-to-be peer-reviewed claims about cold fusion. That was just old-fashioned analogue puffery. And the Internet played no role in the Laetrile or DMSO crazes of the 1980s. You see, there were once these things called “newspapers.” And another thing called “television.” And a sophisticated meatspace-based social network called a “town square.”

But there are critical differences between then and now. As citizens get more access to the scholarly literature, it is far more likely that research is going to be discussed outside of formal scholarly venues. Now we can build tools to help researchers track these discussions. Now researchers can, if they need to, engage in the conversations as well. One would think that conscientious researchers would see it as their responsibility to remain engaged, to know how their research is being used. And especially to know when it is being misused.

That isn’t to say that we expect researchers will welcome this task. We are no Pollyannas. Researchers are already famously overstretched. They barely have time to keep up with the formally published literature. It seems cruel to expect them to keep up with the firehose of the Internet as well.

Which gets us back to the value of altmetrics tools. Our hope is that, as altmetrics tools evolve, they will provide publishers and researchers with an efficient mechanism for monitoring the use of their content in non-traditional venues. Just in the way that citations were used before they were distorted into proxies for credit and kudos.

We don’t think altmetrics are there yet. Partly because some parties are still tantalized by the prospect of usurping one metric for another. But mostly because the entire field is still nascent. People don’t yet know how the information can be combined and used effectively. So we still make naive assumptions such as “link=like” and “more=better.” Surely it will eventually occur to somebody that, instead, there may be a connection between repeated headline-grabbing research and academic fraud. A neuroscientist might be interested in a tool that alerts them if the MRI scans in their research paper are being misinterpreted on the web to promote neurobollocks. An immunologist may want to know if their research is being misused by the anti-vaccination movement. Perhaps the real value in gathering this data will be seen when somebody builds tools to help researchers DETECT puffery, social-citation cabals, and misinterpretation of research results?

But Crossref won’t be building those tools. What we might be able to do is help others overcome another hurdle that blocks the development of more sophisticated tools; getting hold of the needed data in the first place. This is why we are dabbling in altmetrics.

Wikipedia is already the 8th largest referrer of Crossref DOIs. Note that this doesn’t just mean that the Wikipedia cites lots of Crossref DOIs, it means that people actually click on and follow those DOIs to the scholarly literature. As scholarly communication transcends traditional outlets and as the audience for scholarly research broadens, we think that it will be more important for publishers and researcher to be aware of how their research is being discussed and used. They may even need to engage more with non-scholarly audiences. In order to do this, they need to be aware of the conversations. Crossref is providing this experimental data source in the hope that we can spur the development of more sophisticated tools for detecting and analyzing these conversations. Thankfully, this is an inexpensive experiment to conduct - largely thanks to the decision on the part of PLOS to open source its ALM code.

What Now?

Crossref’s instance of PLOS’s ALM code is an experiment. We mentioned that we had encountered scalability problems and that we had resolved some of them. But there are still big scalability issues to address. For example, assuming a response time of 1 second, if we wanted to poll the English-language version of the Wikipedia to see what had cited each of the 65 million DOIs held in Crossref, the process would take years to complete. But this is how the system is designed to work at the moment. It polls various source APIs to see if a particular DOI is “mentioned”. Parallelizing the queries might reduce the amount of time it takes to poll the Wikipedia, but it doesn’t reduce the work. Another obvious way in which we could improve the scalability of the system is to add a push mechanism to supplement the pull mechanism. Instead of going out and polling the Wikipedia 65 million times, we could establish a “scholarly linkback” mechanism that would allow third parties to alert us when DOIs and other scholarly identifiers are referenced (e.g. cited, bookmarked, shared). If the Wikipedia used this, then even in an extreme case scenario (i.e. everything in Wikipedia cites at least one Crossref DOI), this would mean that we would only need to process ~ 4 million trackbacks.

The other significant advantage of adding a push API is that it would take the burden off of Crossref to know what sources we want to poll. At the moment, if a new source comes online, we’d need to know about it and build a custom plugin to poll their data. This needlessly disadvantages new tools and services as it means that their data will not be gathered until they are big enough for us to pay attention to. If the service in question addresses a niche of the scholarly ecosystem, they may never become big enough. But if we allow sources to push data to us using a common infrastructure, then new sources do not need to wait for us to take notice before they can participate in the system.

Supporting (potentially) many new sources will raise another technical issue- tracking and maintaining the provenance of the data that we gather. The current ALM system does a pretty good job of keeping data, but if we ever want third parties to be able to rely on the system, we probably need to extend the provenance information so that the data is cheaply and easily auditable.

Perhaps the most important thing we want to learn from running this experimental ALM instance is: what it would take to run the system as a production service? What technical resources would it require? How could they be supported? And from this we hope to gain enough information to decide whether the service is worth running and, if so, by whom. Crossref is just one of several organisations that could run such a service, but it is not clear if it would be the best one. We hope that as we work with PLOS, our members and the rest of the scholarly community, we’ll get a better idea of how such a service should be governed and sustained.

Details for Propellerheads

Warning, Caveats and Weasel Words

The Crossref ALM instance is a Crossref Labs project. It is running on R&D equipment in a non-production environment administered by an orangutang on a diet of Redbulls and vodka.

So what is working?

The system has been initially loaded with 317,500+ Crossref DOIs representing publications from 2014. We will load more DOIs in reverse chronological order until we get bored or until the system falls over again.

We have activated the following sources:

PubMed

DataCite

PubMedCentral Europe Citations and Usage

We have data from the following sources but will need some work to achieve stability:

Facebook

Wikipedia

CiteULike

Twitter

Some of them are faster than others. Some are more temperamental than others. WordPress, for example, seems to go into a sulk and shut itself off after approximately 1,300 API calls.

In any case, we will be monitoring and tweaking the sources as we gather data. We will also add new sources as we get requested API keys. We will probably even create one or two new sources ourselves. Watch this blog and we’ll update you as we add/tweak sources.

Dammit, shut up already and tell me how to query stuff.

You can login to the Crossref ALM instance simply using a Mozilla Persona (yes, we’d eventually like to support ORCID too). Once logged-in, your account page will list an API key. Using the API key, you can do things like:

http://det.labs.crossref.org.pluma.sjfc.edu/api/v5/articles?ids=10.1038/nature12990

And you will see that (as of this writing), said Nature article has been cited by the Wikipedia article here:

https://en.wikipedia.org/wiki/HE0107-5240#cite_ref-Keller2014_4-0;

PLOS has provided lovely detailed instructions for using the API- So, please, play with the API and see what you make of it. On our side we will be looking at how we can improve performance and expand coverage. We don’t promise much- the logistics here are formidable. As we said above, once you start working with millions of documents, the polling process starts to hit API walls quickly. But that is all part of the experiment. We appreciate your helping us and would like your feedback. We can be contacted at:

Event Data on Crossref

Strengthening support for data citations and saying goodbye to Event Data

Time to say goodbye

Visibility for data citations

Event Data now with added references

A bumpy ride, but we got there

Where to next?

Open references

Event Data: Help us fill in the gaps

Service Provider perspectives: A few minutes with our publisher hosting platforms

What is the biggest change you’ve experienced working with publisher metadata over the last few years and how have you adapted to it?

How do you explain to clients (and others!) why correct, quality metadata is important?

What is the one industry development or trend you’re most excited about for the near future and why?

Doing more with relationships - via Event Data

What are relationships?

Some examples

Relationships in the metadata show links to the published article from this bioRxiv preprint. In the Crossref Rest API:

And now in Event Data:

Linking to a dataset in the Dryad Digital Repository by a recent eLife article. In the Crossref metadata:

And now in Event Data:

Data citation

What’s next?

More info and staying in touch

Event Data: A Plan of Action

The Plan

Stability

Consolidation

Future roadmap

Staying in touch

Fast, citable feedback: Peer reviews for preprints and other record types

Reviews for preprints

Events got the better of us

Why so quiet?

So, where are we today?

What are we doing?

Opening up the discussion

What's that DOI?

Some background

Half full or half empty?

How do we do it?

What Agents do

Not so fast

So, what happens?

Awkward questions

Q: Why bother with all of this? Can’t you just use the URLs?

Q: If DOIs are so amazing why keep URLs?

Q: Can’t you just decide for me?

Q: Why only DOIs?

Q: What about websites that require cookies and/or JavaScript to execute?

Q: What’s the success rate?

Q: Why isn’t the domain-decision-structure Artifact more detailed?

Event Data is production ready

Technical readiness

What does this mean?

Getting started

What next?

Hear this, real insight into the inner workings of Crossref

You want to hear more from us. We hear you.

Introducing the Crossref Thing Action Service!

live.eventdata.crossref.org/thing-action-service.html

Hello, meet Event Data Version 1, and new Product Manager

Same Event Data, new address

Changes to the Wikipedia data structure

Updated data

Event Data beta group

Bridging Identifiers at PIDapalooza

Who cares about PIDs?

The Twitterverse

Bridging those who care with those who don’t

Working at scale

Building bridges

Bridge 1: PIDs and URLs

Bridge 2: Scholarly link providers

Bridge 3: Crossref / DataCite

Bridge 4: Traditional discussions vs new ones

Bridge 5: Bridging bibliometricians and altmetricians to data sources

Bridge 6: Bridging the Evidence Gap

Interesting challenges

The thing about bridges…

PIDapalooza

Q: Why isn’t the `domain-decision-structure` Artifact more detailed?