<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Technology on Crossref</title><link>https://www-crossref-org.pluma.sjfc.edu/categories/technology/</link><description>Recent content in Technology on Crossref</description><generator>Hugo 0.139.4</generator><language>en-us</language><managingEditor>support@crossref.org (Crossref/Cazinc/Benoît Benedetti)</managingEditor><webMaster>support@crossref.org (Crossref/Cazinc/Benoît Benedetti)</webMaster><lastBuildDate>Thu, 21 May 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://www-crossref-org.pluma.sjfc.edu/categories/technology/" rel="self" type="application/rss+xml"/><item><title>Mission Accessible: building better user interfaces for everyone</title><link>https://www-crossref-org.pluma.sjfc.edu/blog/mission-accessible-building-better-user-interfaces-for-everyone/</link><pubDate>Thu, 21 May 2026 00:00:00 +0000</pubDate><author>Lena Stoll</author><guid>https://www-crossref-org.pluma.sjfc.edu/blog/mission-accessible-building-better-user-interfaces-for-everyone/</guid><description>&lt;p>Today is &lt;a href="https://accessibility.day/" target="_blank">Global Accessibility Awareness Day&lt;/a>, and accessibility has been on our minds lately. We&amp;rsquo;ve recently completed an internal audit of all our user interfaces, and have added a new &lt;a href="https://www-crossref-org.pluma.sjfc.edu/operations-and-sustainability/accessibility/">accessibility page&lt;/a> to our website, where you can find the accessibility documentation that we put together as part of the audit.&lt;/p>
&lt;h2 id="why-accessibility-matters">Why accessibility matters&lt;/h2>
&lt;p>Of course we want to keep the barriers to participation in Crossref as low as possible for users with various disabilities. But also, more accessible tools work better for everyone. A person&amp;rsquo;s access needs can change really quickly: even if you consider yourself to be relatively able-bodied, you are only one minor inconvenience away from at least a temporary disability. All it takes is some dazzling sunlight hitting your eye or your phone screen, or perhaps your dog going after a rabbit in an awkward direction while you are holding the lead (ask me how I know!) - and before you know it, you will be relying on accessibility features to navigate the digital and/or physical world for a while.&lt;/p>
&lt;p>An accessible user interface is one that you can navigate and interact with by various methods, including a mouse or touchpad, keyboard, screen reader, voice control, and other assistive technologies. It can be used on various screen sizes and supports zooming in or out without losing any content or functionality. It has sufficient colour contrast, doesn&amp;rsquo;t flash fast-moving images at you, and has a clear structure that can be understood by both humans and machines.&lt;/p>
&lt;h2 id="where-we-are-today">Where we are today&lt;/h2>
&lt;p>It is worth mentioning that we didn&amp;rsquo;t only start thinking about accessibility when we started tackling the full audit of our user interfaces in March 2026. For example, Patrick Vale has previously &lt;a href="https://doi-org.pluma.sjfc.edu/10.64000/pp4rw-mtv44" target="_blank">written in this blog&lt;/a> about a browser extension he has created to improve the accessibility of DOI links anywhere on the Internet. And we have known for a long time that there were accessibility gaps in many of our tools, but we didn&amp;rsquo;t have this centrally documented anywhere.&lt;/p>
&lt;p>When we did begin testing all our interfaces for compliance with level AA of the &lt;a href="https://www.w3.org/TR/WCAG22/" target="_blank">Web Content Accessibility Guidelines (WCAG 2.2)&lt;/a> as part of the audit, we knew that some of what we would find was not going to be pretty. In the 26+ years of working with and for the scholarly community, Crossref has built countless tools and reports to offer to members and users, many of which we still maintain today. These are often decades old and have been built in a way that makes it virtually impossible to make them more accessible without rebuilding them entirely. So we know that we will continue to have accessibility gaps for the foreseeable future, but at least now we have a better idea of the scale of the challenge.&lt;/p>
&lt;p>It&amp;rsquo;s also not all doom and gloom: more recently created user interfaces, such our &lt;a href="https://www-crossref-org.pluma.sjfc.edu/documentation/register-maintain-records/metadata-manager/">new Metadata Manager&lt;/a>, performed much better in the audit than legacy alternatives such as the &lt;a href="https://www-crossref-org.pluma.sjfc.edu/documentation/register-maintain-records/web-deposit-form/">web deposit form&lt;/a>. We found a similar trend when looking at our &lt;a href="https://www-crossref-org.pluma.sjfc.edu/documentation/reports/">report interfaces&lt;/a>. To illustrate this, compare what happens when running the &lt;a href="https://chromewebstore.google.com/detail/axe-devtools-web-accessib/lhdoppojpmngadmnindnejefpokejbdd" target="_blank">axe DevTools extension for Google Chrome&lt;/a> on a member&amp;rsquo;s &lt;a href="https://www-crossref-org.pluma.sjfc.edu/documentation/reports/participation-reports/">participation report&lt;/a> - this is a user interface that was completely re-implemented in 2025. Doing this brings up 26 issues:&lt;/p>
&lt;div style="text-align:center;margin:10px">
&lt;figure class="img-responsive">&lt;img src="https://www-crossref-org.pluma.sjfc.edu/images/blog/2026/accessibility-issues-participation-reports.png"
alt="Screenshot of the Participation Reports interface with axe DevTools showing 26 total issues" width="800px">
&lt;/figure>
&lt;/div>
&lt;p>Meanwhile, the &lt;a href="https://www-crossref-org.pluma.sjfc.edu/documentation/reports/browsable-title-list/">browsable title list&lt;/a>, which has completed a few more trips around the sun, has 254 issues listed:&lt;/p>
&lt;div style="text-align:center;margin:10px">
&lt;figure class="img-responsive">&lt;img src="https://www-crossref-org.pluma.sjfc.edu/images/blog/2026/accessibility-issues-title-list.png"
alt="Screenshot of the browsable title list interface with axe DevTools showing 254 total issues" width="800px">
&lt;/figure>
&lt;/div>
&lt;h2 id="beyond-wcag">Beyond WCAG&lt;/h2>
&lt;p>If you&amp;rsquo;ve read this far, I hope you are convinced that accessibility is more than just ticking boxes on a conformance report. But especially for a global community like ours, there are other, less technical barriers to participation that we have to consider. For example, language is a major accessibility factor: much of what we as Crossref staff write and say is in English. When we host &lt;a href="https://www-crossref-org.pluma.sjfc.edu/events/">community events&lt;/a>, we enable captions, and we try to leave space for these captions at the bottom of our slides.&lt;/p>
&lt;p>We have also started experimenting with simultaneous interpretation during our online events, such as our recent project showcase event for the 2026 &lt;a href="https://doi-org.pluma.sjfc.edu/10.64000/a5qzf-k1738" target="_blank">metadata sprint in São Paulo&lt;/a>. You can find recordings of this event in &lt;a href="https://www.youtube.com/watch?v=ws9qrLJ1aCc" target="_blank">Spanish&lt;/a>, &lt;a href="https://www.youtube.com/watch?v=ocRP_UIq0Qs" target="_blank">Portuguese&lt;/a>, and &lt;a href="https://www.youtube.com/watch?v=RU0Uq71Q944" target="_blank">English&lt;/a> on our YouTube channel to see the promising results of these efforts.&lt;/p>
&lt;h2 id="what-we-are-working-on-next">What we are working on next&lt;/h2>
&lt;p>We are currently addressing the accessibility issues identified in our audit of the &lt;a href="https://www-crossref-org.pluma.sjfc.edu/services/crossmark/">Crossmark&lt;/a> service. Many Crossref members have implemented the Crossmark button and pop-up on their own platforms and websites, so we thought this was a great place to start the remediation efforts following our audit.&lt;/p>
&lt;p>We are also in the process of redesigning our main website, &lt;a href="https://www-crossref-org.pluma.sjfc.edu" target="_blank">www.crossref.org&lt;/a>, following an &lt;a href="https://doi-org.pluma.sjfc.edu/10.64000/058mr-k3s56" target="_blank">information architecture review&lt;/a> completed in 2025. Making changes to the design and navigation of our website will be the perfect opportunity to make our content not just more discoverable and more understandable, but also more accessible.&lt;/p>
&lt;p>Clearly there is even more to be done, so watch this space for more updates on our accessibility roadmap and improvements. And if you have first-hand experience of using Crossref services and interfaces with assistive technologies, or you have other input or feedback you&amp;rsquo;d like to share, leave a comment below or start a discussion in our &lt;a href="https://community-crossref-org.pluma.sjfc.edu/" target="_blank">community forum&lt;/a>.&lt;/p>
&lt;h3 id="references">References&lt;/h3>
&lt;ol>
&lt;li>Vale, P. (2025). Enhancing DOI Accessibility for All Users. Crossref. &lt;a href="https://doi-org.pluma.sjfc.edu/10.64000/pp4rw-mtv44" target="_blank">https://doi-org.pluma.sjfc.edu/10.64000/pp4rw-mtv44&lt;/a>&lt;/li>
&lt;li>World Wide Web Consortium (2024). Web Content Accessibility Guidelines (WCAG) 2.2. Retrieved May 8, 2026, from &lt;a href="https://www.w3.org/TR/WCAG22/" target="_blank">https://www.w3.org/TR/WCAG22/&lt;/a>&lt;/li>
&lt;li>Montilla, L. &amp;amp; Mahomed, R. (2026). Voices from Crossref Metadata Sprint in São Paulo. Crossref. &lt;a href="https://doi-org.pluma.sjfc.edu/10.64000/a5qzf-k1738" target="_blank">https://doi-org.pluma.sjfc.edu/10.64000/a5qzf-k1738&lt;/a>&lt;/li>
&lt;li>Stoll, L. &amp;amp; Korzec, K. (2025). Request for proposals: Crossref website information architecture review. Crossref. &lt;a href="https://doi-org.pluma.sjfc.edu/10.64000/058mr-k3s56" target="_blank">https://doi-org.pluma.sjfc.edu/10.64000/058mr-k3s56&lt;/a>&lt;/li>
&lt;/ol></description></item><item><title>Enhancing DOI Accessibility for All Users</title><link>https://www-crossref-org.pluma.sjfc.edu/blog/enhancing-doi-accessibility-for-all-users/</link><pubDate>Mon, 28 Apr 2025 00:00:00 +0000</pubDate><author>Patrick Vale</author><guid>https://www-crossref-org.pluma.sjfc.edu/blog/enhancing-doi-accessibility-for-all-users/</guid><description>&lt;h2 id="2025-update">2025 Update&lt;/h2>
&lt;p>In 2022, we set out to update our DOI display guidelines with the intention to adopt &lt;a href="https://doi-org.pluma.sjfc.edu/10.64000/gyw3h-trd87" target="_blank">the proposals&lt;/a> in 2025. It’s important to note from the outset that we are not mandating any immediate changes to the &lt;a href="https://www-crossref-org.pluma.sjfc.edu/display-guidelines/" target="_blank">DOI display guidelines&lt;/a>. Instead, we are working with our community to co-create a solution that addresses the diverse needs of all users, rather than imposing technical changes that may not suit everyone.&lt;/p>
&lt;h2 id="background">Background&lt;/h2>
&lt;p>DOI links are the lifeblood of scholarly communication. They’re the canonical identifiers that enable researchers to find, cite, and assess academic work. In essence, they’re stable, reliable, and easy to use—provided you can see them. But what happens when a user can’t rely on visual cues?&lt;/p>
&lt;h2 id="the-accessibility-challenge">The Accessibility Challenge&lt;/h2>
&lt;p>For users of screen readers and other assistive technologies, the full value of a DOI link can be lost. While sighted users benefit from the context surrounding a DOI link—such as the title, abstract, and other metadata—screen reader users often hear just the bare URL. This means they might not know what content the DOI link represents, leading to confusion and a diminished browsing experience.&lt;/p>
&lt;p>The problem is compounded by the technical nature of DOI links. Being URIs (Uniform Resource Identifiers), they don&amp;rsquo;t naturally lend themselves to the same accessibility techniques as standard URLs. When we attempted to tweak DOI links directly, every change that improved accessibility for one group inadvertently hindered another. Whether it was a WCAG (Web Content Accessibility Guidelines) rule or an ARIA (Accessible Rich Internet Applications) attribute, a solution that worked in one area would break in another.&lt;/p>
&lt;h2 id="a-community-driven-approach">A Community-Driven Approach&lt;/h2>
&lt;p>Realizing that a one-size-fits-all fix wouldn’t work, we took a different approach - one that involved the community from the outset. After consulting with early adopters and attending an insightful session with the &lt;a href="https://jats4r.niso.org/" target="_blank">JATS4R&lt;/a> accessibility group, it became clear that the answer lay in experimentation and iteration. Rather than modifying the DOI display guidelines immediately, we are developing a tool that enhances the user experience without disrupting the current standards.&lt;/p>
&lt;p>It’s worth noting that this solution places the responsibility on the end user rather than on publishers and platform providers. However, by doing so, users can have a consistent browsing experience regardless of the platform they use to access scholarly content. This approach also serves as an important stepping stone toward a future publisher-provided solution—be it via accessibility-focused JavaScript or a mandated dual-link implementation—and any efforts to recommend or mandate such changes will benefit greatly from concrete evidence of the effectiveness and scalability of this approach.&lt;/p>
&lt;h2 id="introducing-the-doi-accessibility-enhancer">Introducing the DOI Accessibility Enhancer&lt;/h2>
&lt;p>First &lt;a href="https://www.youtube.com/watch?v=VBnfkOxVr6s&amp;amp;t=1916s" target="_blank">demonstrated at the recent Crossref Annual Meeting&lt;/a>, here we share our DOI Accessibility Enhancer browser extension. Available now on the &lt;a href="https://addons.mozilla.org/en-GB/firefox/addon/crossref-doi-a11y-tool/" target="_blank">Firefox Add-on Store&lt;/a> and the &lt;a href="https://chromewebstore.google.com/detail/crossref-doi-accessibilit/nmpnpkdfcdnbnpiekngokijfoilpfpbc" target="_blank">Chrome Web Store&lt;/a>, this extension is designed to improve the experience of DOI links for screen reader users without altering the default behavior for sighted users.&lt;/p>
&lt;div style="text-align:center;margin:10px">
&lt;img src="https://www-crossref-org.pluma.sjfc.edu/images/blog/2025/doi-a11y-enhancer-2025.png" width="60%" alt="The Crossref DOI Accessibility Enhancer browser extension running in Firefox" >
&lt;/div>
&lt;h2 id="how-it-works">How It Works&lt;/h2>
&lt;ul>
&lt;li>
&lt;p>&lt;strong>Scanning for DOI Links:&lt;/strong> The extension scans any webpage for DOI links.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Querying Metadata:&lt;/strong> Once a DOI is detected, it queries the Crossref REST API to retrieve the title of the corresponding scholarly work.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Enhancing the Link:&lt;/strong> The title is then injected as a screen-reader–only link. This means that when a screen reader user navigates to the DOI, they hear the title of the paper rather than the opaque URL.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Maintaining Visual Integrity:&lt;/strong> For sighted users, the original DOI link remains unchanged—visible, clickable, and easy to copy-and-paste.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Highlighting for Testing:&lt;/strong> An optional feature highlights updated links, making it easier for developers and testers to see the changes in action.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h2 id="get-involved">Get Involved&lt;/h2>
&lt;p>This project is very much a community effort. The extension is open-source, and we welcome feedback and contributions via our &lt;a href="https://gitlab.com/crossref/doi-accessibility-enhancer" target="_blank">GitLab repository&lt;/a>, email, or &lt;a href="https://community-crossref-org.pluma.sjfc.edu/" target="_blank">Community Forum&lt;/a>. Your real-life experiences and insights will drive future improvements, ensuring that our solution meets the diverse needs of all users.&lt;/p>
&lt;h2 id="try-it-out">Try It Out&lt;/h2>
&lt;div style="text-align:center;margin:10px">
&lt;img src="https://www-crossref-org.pluma.sjfc.edu/images/blog/2025/doi-a11y-enchancer-firefox-addons-2025.png" width="60%" alt="The Crossref DOI Accessibility Enhancer browser extension in Firefox Add-ons" >
&lt;/div>
&lt;p>If you’re using Firefox, head over to the Firefox Add-on Store and install the &lt;a href="https://addons.mozilla.org/en-GB/firefox/addon/crossref-doi-a11y-tool/" target="_blank">DOI Accessibility Enhancer&lt;/a> today. If you’re a Chrome user, you can find the extension directly in the &lt;a href="https://chromewebstore.google.com/detail/crossref-doi-accessibilit/nmpnpkdfcdnbnpiekngokijfoilpfpbc" target="_blank">Chrome Web Store&lt;/a>. If you use a screen reader you’ll experience the difference firsthand - and if you don’t, give it a try with VoiceOver enabled (Command-F5 on a Mac).&lt;/p>
&lt;p>Together, we can advance scholarly accessibility and ensure that critical research remains discoverable for everyone.&lt;/p></description></item><item><title>A Request for Comment - Automatic Digital Preservation and Self-Healing DOIs</title><link>https://www-crossref-org.pluma.sjfc.edu/blog/a-request-for-comment-automatic-digital-preservation-and-self-healing-dois/</link><pubDate>Thu, 29 Jun 2023 00:00:00 +0000</pubDate><author>Martin Eve</author><guid>https://www-crossref-org.pluma.sjfc.edu/blog/a-request-for-comment-automatic-digital-preservation-and-self-healing-dois/</guid><description>&lt;p>Digital preservation is crucial to the &amp;ldquo;persistence&amp;rdquo; of persistent identifiers. Without a reliable archival solution, if a Crossref member ceases operations or there is a technical disaster, the identifier will no longer resolve. This is why the Crossref member terms insist that publishers make best efforts to ensure deposit in a reputable archive service. This means that, if there is a system failure, the DOI will continue to resolve and the content will remain accessible. This is how we protect the integrity of the scholarly record.&lt;/p>
&lt;p>I will write another post, soon, on the reality of preservation of items with a Crossref DOI, but recent work in the Labs team has determined that we have a situation of drastic under-preservation of much scholarly material that has been assigned a persistent identifier. In particular, content from our smaller Crossref members, with limited financial resources, is often precariously preserved. Further, DOI URLs are not always updated, even when, for instance, the underlying domain has been registered by a different third party. This results in DOIs pointing to new, hijacked, and elapsed content that does not reflect the metadata that we hold.&lt;/p>
&lt;p>We (&lt;a href="https://www-crossref-org.pluma.sjfc.edu/people/geoffrey-bilder/" target="_blank">Geoffrey&lt;/a>) have (has) long-harboured ambitions to build a system that would allow for automatic deposit into an archive and then to present access options to the resolving user. This would ensure that all Crossref content had at least one archival solution backing it and greatly contribute to the improved persistent resolvability of our DOIs. We refer to this, internally, as &amp;ldquo;Project Op Cit&amp;rdquo;. And we&amp;rsquo;re now in a position to begin building it.&lt;/p>
&lt;p>However, we need to get this right from the design phase out. We need input from librarians working in the digital preservation space. We need input from members on whether they would use such a service. We are not digital preservation experts and we are acutely aware that we need the expertise of those who are, particularly where we&amp;rsquo;ve had to take some shortcuts. For instance: we are aware that the Internet Archive is perhaps not the first choice of many digital preservation librarians and specialists, who opt for specific scholarly-communications solutions. However, it is easy, open, and free. Hence, we propose for the prototype to use IA, on the assumption that this will be a proof-of-concept only, which we will expand to other archives if there is demand and once it works.&lt;/p>
&lt;p>So: please do read the below and add your comments and questions to this thread in the community forum (link below), or &lt;a href="mailto:meve@crossref.org">send me queries/concerns by email&lt;/a>. It would be excellent if we could receive comments by mid-August 2023. If you would rather comment on a Google doc, &lt;a href="https://docs.google.com/document/d/1UHW8n_ohJhETc4aLK6ZHB3OK0A0270mgM1l4IsNudZ0/edit?usp=sharing" target="_blank">that&amp;rsquo;s also possible&lt;/a>.&lt;/p>
&lt;p>If enough people are interested, we could also host a community call to discuss this design and its prototyping. Do please, when emailing, let me know if this is of interest.&lt;/p>
&lt;hr>
&lt;h2 id="project-op-cit-self-healing-dois">Project Op Cit (Self-Healing DOIs)&lt;/h2>
&lt;h2 id="request-for-comment">Request for Comment&lt;/h2>
&lt;p>This document sets out the problem statement, a proposed prototype solution, and a transition path to production if successful.&lt;/p>
&lt;h2 id="proposed-prototype-solution">Proposed Prototype Solution&lt;/h2>
&lt;p>For members who opt-in to the service, We have a special class of DOI (only for open-access content) where, when the DOI is registered:&lt;/p>
&lt;ul>
&lt;li>
&lt;p>We immediately make an archive of the item with any archiving services that care to participate in the project (minimally, the Internet Archive, which is the easiest for us to begin with, but a modular/pluggable archival system). The &lt;a href="https://archive.org/developers/internetarchive/" target="_blank">Internet Archive Python Librar&lt;/a>y should let us submit to them. We could pursue other arrangements with CLOCKSS, LOCKSS, and Portico.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>We update the XML to reflect the archives to which it has been submitted.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>The DOI landing page is redirected to an interstitial page that we control. This page gives the user access options.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>We develop processes to determine whether the original URL &amp;ldquo;works&amp;rdquo;. The heuristics that define whether a resource has changed substantially or works need long-term consideration and real-world testing. Using the interstitial page approach will allow us to refine this, with a long-term goal of eradicating it.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;center>&lt;figure class="img-responsive">&lt;img src="https://www-crossref-org.pluma.sjfc.edu/images/blog/2023/deposit-process-blog.png"
alt="Image showing the process flow from user to OpCit Deposit Endpoint to Preservation System (archive) to Crossref Deposit System (Live API)" width="75%">&lt;figcaption>
&lt;p>Figure 1: The Deposit Process&lt;/p>
&lt;/figcaption>
&lt;/figure>
&lt;p>&lt;figure class="img-responsive">&lt;img src="https://www-crossref-org.pluma.sjfc.edu/images/blog/2023/resolution-process-blog.png"
alt="Image showing the process flow from user to DOI to OpCit Deposit Endpoint to Preservation System (preserved copy) to Crossref Deposit System (original publications)" width="75%">&lt;figcaption>
&lt;p>Figure 2: The Resolution Process&lt;/p>
&lt;/figcaption>
&lt;/figure>
&lt;/center>&lt;/p>
&lt;h3 id="potential-challenges">Potential Challenges&lt;/h3>
&lt;ul>
&lt;li>
&lt;p>Content drift. It would be extremely difficult to detect content change vs. (eg) page structure change, except in the case of binary fulltext. However, we can poll for the DOI at an HTML endpoint and detect when binary fulltext items, such as a PDF, change.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Latency on resolver if lookup is real-time. For this reason, we need a periodic crawler so that resolvers do not wait for real-time detection on access.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>If using Internet Archive, the domain owner (at the present moment) can request the removal of content. We would need the capacity to &amp;ldquo;lock&amp;rdquo; records that are being used as Op Cit redirection archival copies. This requires a further conversation with the Internet Archive.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h2 id="prototype-componentsarchitecture">Prototype Components/Architecture&lt;/h2>
&lt;h3 id="registration-proxy-and-database-fleming">Registration Proxy and Database (&amp;ldquo;Fleming&amp;rdquo;)&lt;/h3>
&lt;p>The registration proxy implements a pass-through to the deposit API and hosts a relational database of self-healing DOIs (Postgres). It will be hosted at api.labs.crossref.org/deposit/opcit and clients will have to use this endpoint to deposit. Simultaneously, the proxy will:&lt;/p>
&lt;ul>
&lt;li>
&lt;p>Determine the license status of the incoming item.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>If the license is open and fulltext is provided, deposit a copy in selected digital preservation archives. Store proof of licensing attestation.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>In the case of binary files (fulltext PDF), store a hash of the content.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Store the DOI, binary hash, and all URLs in a relational database under &amp;ldquo;pending&amp;rdquo; state.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Pass through the request to Crossref&amp;rsquo;s content registration system.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Monitor the result of this request and remove stored data if registration fails.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Re-registration through Fleming will update existing entries and re-fix their data against content drift at this time.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h3 id="spider-shelob">Spider (&amp;ldquo;Shelob&amp;rdquo;)&lt;/h3>
&lt;p>A series of components that:&lt;/p>
&lt;ul>
&lt;li>
&lt;p>Check that &amp;ldquo;pending&amp;rdquo; DOIs have been successfully registered. Remove those that have not and move those that have to &amp;ldquo;active&amp;rdquo; state.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Dereference &amp;ldquo;active&amp;rdquo; DOIs and ensure that we have the most current URL in case updates have gone directly to the live resolver.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Periodically crawl URLs in the self-healing database.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>On HTTP 301 code, update database entry to point to new permanent URL.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>On HTTP 302 code, follow the temporary redirect expecting the original content.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>On HTTP 4xx codes, mark the entry as dead.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>On HTTP 200 code of HTML landing page, parse the page for the presence of the DOI. If the DOI is not present, mark the entry as dead.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h3 id="resolver-proxy-hippocrates">Resolver Proxy (&amp;ldquo;Hippocrates&amp;rdquo;)&lt;/h3>
&lt;ul>
&lt;li>
&lt;p>Display an interstitial landing page with archival versions and an explanation.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>At some future point, for active entries, resolve to the stored URL (faster but could be de-synced) or pass the request to the live resolver (requires an extra hop but will always be in-sync with deposit).&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h3 id="observability-and-statistics">Observability and statistics&lt;/h3>
&lt;p>Metrics we will collect:&lt;/p>
&lt;ul>
&lt;li>
&lt;p>Count of DOIs using Op Cit&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Count of visitors arriving on Op Cit landing pages&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Usage count of each outgoing link/access option&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>A daily report will present:&lt;/p>
&lt;ul>
&lt;li>
&lt;p>Newly &amp;ldquo;failed&amp;rdquo; entries that we believe have died&lt;/p>
&lt;/li>
&lt;li>
&lt;p>These will be checked extensively, particularly at first, to ascertain whether our failure heuristics are valid&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Entries that have recovered&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>Errors will be logged and monitored via Grafana.&lt;/p>
&lt;h3 id="documentation-and-automated-tests">Documentation and Automated Tests&lt;/h3>
&lt;ul>
&lt;li>
&lt;p>Core assumptions and new behaviours of the platform will be documented as part of the prototype.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Automated tests will be written, especially for the spider (&amp;ldquo;Shelob&amp;rdquo;), which must handle a diverse variety of real-world situations.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h3 id="prototype-architecture-requirements">Prototype Architecture Requirements&lt;/h3>
&lt;ul>
&lt;li>
&lt;p>Postgres RDS for resolution/self-healing DOI data (AWS).&lt;/p>
&lt;/li>
&lt;li>
&lt;p>FastAPI hosting for passthrough proxy (fly.io).&lt;/p>
&lt;/li>
&lt;li>
&lt;p>EC2 hosting for the spider (AWS).&lt;/p>
&lt;/li>
&lt;li>
&lt;p>FastAPI hosting for resolver proxy (fly.io).&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h2 id="transition-to-production">Transition to Production&lt;/h2>
&lt;p>If this prototype garners popular appeal, a transition to production would need to keep some prototype components and rewrite others.&lt;/p>
&lt;ul>
&lt;li>
&lt;p>&amp;ldquo;Fleming&amp;rdquo; would need to be rewritten as a deposit module / integrated with Manifold&amp;rsquo;s (the next-generation system at Crossref) deposit. If this would create too much overhead, it need not be a blocking process in the deposit.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&amp;ldquo;Shelob&amp;rdquo; would continue to need to run continuously and to scale with the adoption of self-healing DOIs unless one of the other options were used.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Prototype architecture will be written so that spidering can be distributed between several servers, if required.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&amp;ldquo;Hippocrates&amp;rdquo; would need to be integrated into the live link resolver. Depending on how a field for a self-healing DOI is embedded in Manifold, this may not need any additional database hits.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;h3 id="back-content">Back Content&lt;/h3>
&lt;p>We also have a database of back content stored by the Internet Archive, mapped to DOIs where they have been able to do so. This data source could be used to enable self-healing DOIs on all content in this archive.&lt;/p></description></item><item><title>Crossref Research and Development: Releasing our Tools from the Ground Up</title><link>https://www-crossref-org.pluma.sjfc.edu/blog/crossref-research-and-development-releasing-our-tools-from-the-ground-up/</link><pubDate>Wed, 21 Jun 2023 00:00:00 +0000</pubDate><author>Martin Eve</author><guid>https://www-crossref-org.pluma.sjfc.edu/blog/crossref-research-and-development-releasing-our-tools-from-the-ground-up/</guid><description>&lt;p>This is the first post in a series designed to showcase what we do in the Crossref R&amp;amp;D group, also known as &lt;a href="https://www-crossref-org.pluma.sjfc.edu/labs/" target="_blank">Crossref Labs&lt;/a>, which over the last few years has been strengthened, first with Dominika Tkaczyk and Esha Datta, last year with part of Paul Davis’s time, and more recently, yours truly. Research and development are, obviously, crucial for any organisation that doesn’t want to stand still. The R&amp;amp;D group builds prototypes, experimental solutions, and data-mining applications that can help us to understand our member base, in the service of future evolution of the organisation. One of the strategic pillars of Crossref is that we want to contribute to an environment in which the scholarly research community identifies shared problems and co-creates solutions for broad benefit. We do this in all teams through research and engagement with our expanding community.&lt;/p>
&lt;p>&lt;img src="https://www-crossref-org.pluma.sjfc.edu/img/labs/creature3.svg" alt="The Crossref Labs Creature"> &lt;img src="https://www-crossref-org.pluma.sjfc.edu/img/labs/creature2.svg" alt="The Crossref Labs Creature"> &lt;img src="https://www-crossref-org.pluma.sjfc.edu/img/labs/creature1.svg" alt="The Crossref Labs Creature">&lt;/p>
&lt;p>For example, if the metadata team wants to implement a new field in our schema, it helps to have a prototype to show to members. The Labs team would implement such a prototype. If we want to know the answer to a question about the 150m or so metadata records we have – e.g. how many DOIs are duplicates? – it’s the Labs team that will work on this.&lt;/p>
&lt;p>When building such prototypes, which can often seem esoteric and one-off, though, it can be easy to believe that there is no way anybody else would re-use our components. At the same time, we find ourselves consistently working with the same infrastructures, re-using different code blocks across many applications. One of the tasks I have been working on is to extract these duplicated functions and to get them into external code libraries.&lt;/p>
&lt;p>Why is this important? As many readers doubtless know, Crossref is committed to &lt;a href="https://doi-org.pluma.sjfc.edu/10.24343/C34W2H" target="_blank">The Principles of Open Scholarly Infrastructure&lt;/a>. For reasons of insurance, everything we do and newly develop is open source and we want our members to be able to re-use the software that we create. It’s also important because, if we centralize these low-level building blocks, we make it much easier to fix bugs when they occur, which would otherwise be distributed across all of our projects.&lt;/p>
&lt;p>As a result, Crossref Labs has a series of small code libraries that we have released for various service interactions. We often find ourselves needing to interact with AWS services. Indeed, Crossref’s live systems are in the process of transitioning to running in the cloud, rather than our own data centre. It makes sense, therefore, for prototype Labs systems to run on this infrastructure, too. However, the boto3 library is not terribly Pythonic. As a result, many of our low-level tools interact with AWS. These include:&lt;/p>
&lt;ul>
&lt;li>&lt;a href="https://gitlab.com/crossref/labs/claws" target="_blank">CLAWS: the Crossref Labs Amazon Web Services toolkit&lt;/a>. The CLAWS library gives speedy and Pythonic access to functions that we use again and again. This includes downloading files from and pushing data to S3 buckets (often in parallel/asynchronously), fetching secrets from AWS Secrets Manager, generating pre-signed URLs, and more.&lt;/li>
&lt;li>&lt;a href="https://gitlab.com/crossref/labs/longsight" target="_blank">Longsight: A range of common logging functions for the observability of Python AWS cloud applications&lt;/a>. Less mature than CLAWS, this is the starting point for observability across Labs applications. It supports running in AWS Lambda function contexts or pushing your logs to AWS Cloudwatch from anywhere else. It also supports logging metrics in structured forms. Crucially, the logs are all converted into machine-readable JSON format. This allows us to export the metrics into Grafana dashboards to visualize failure and performance.&lt;/li>
&lt;li>&lt;a href="https://gitlab.com/crossref/labs/distrunner" target="_blank">Distrunner: decentralized data processing on AWS services&lt;/a>. Easily the least mature and experimental of these libraries, distrunner is one of the ways that we distribute the workloads of our recurrent data processing. A number of the Labs projects require us to run recurrent data-processing tasks. For instance, my colleague Dominika Tkaczyk has developed the &lt;a href="https://gitlab.com/crossref/labs/sampling-framework" target="_blank">sampling framework&lt;/a> that is regenerated once per week. We use Apache Airflow (and, specifically, Amazon Managed Workflows for Apache Airflow) to host these periodic tasks. This is useful because it gives us quick, visual oversight if tasks fail. However, the Airflow worker instances on AWS are quite severely underpowered and unsuitable for large in-memory activities. Hence, the sampling framework fires up a Spark instance for its processing. Often, though, we do not need the parallelization of Spark and just want to be able to run a generic Python script in a more powerful environment. That’s what distrunner is designed to do. The current version uses &lt;a href="https://www.coiled.io/" target="_blank">Coiled&lt;/a> but this may change in the future.&lt;/li>
&lt;/ul>
&lt;p>While these tools will be useful to nobody except programmers – and this has been quite a technical post – there is a broader philosophical point to be made about this approach, in which everything is available for re-use, “from the ground up”. The point is: we also try, in Labs and in the process of “R&amp;amp;Ding”, to work without privileged access. That is: I don’t get “inside” access to a database that isn’t accessible to external users. I have to work with the same APIs and systems as would an end-user of our services. This means that, when we develop internal libraries, it’s worth releasing them. Because they use systems that are accessible to any of our users.&lt;/p>
&lt;p>I should also say that our openness is more than unidirectional. While we are putting a lot of effort into ensuring that everything new we put out is openly accessible, we are also open to contributions coming in. If we’ve built something and you make changes or improve it, please do get in touch or submit a pull request. Openness has to work both ways if projects are truly to be used by the community.&lt;/p>
&lt;p>Future posts – coming soon! – will introduce some of the technologies and projects that we have been building atop this infrastructure. This includes a Labs API system; new functionality to retrieve unpaginated datasets of whole API routes; a study of the preservation status of DOI-assigned content; and a mechanism for modeling new metadata fields.&lt;/p></description></item></channel></rss>