Automating the Organization and Accessibility of Academic Literature: Bookends and Devonthink Pro

I wrote about Sense-making of the Academic Literature back when I was using Papers. My old workflow required Keyboard Maestro to extract abstract information to Tinderbox. Although Tinderbox is great for visually organizing your notes, it can’t compare to the search capabilities of DevonThink Pro.Since switching to Bookends, I’ve found myself spending less time in Tinderbox and even more time in Devonthink. I’ve made heavy use of a particular built-in Devonthink Template that automates the migration of abstracts from Bookends to DTP (with a link back to Bookends). Once imported into DTP, these abstracts can be parsed using smart folders (enduring searches) that make your literature library a dynamic resource for writing.I have a single DTP database that has indexed the various folders critical in my writing process (e.g. PDFs, extracted notes). In DTP, I’ve organized these folders into a “zz-Inputs Folder”. As you can see below, a sub-folders of zz-Inputs is an “Abstracts Inbox.” When I set up this folder in DTP, I made sure the folder WAS NOT excluded from tagging (this is done by option-clicking on the folder and making sure “exclude from tagging is UNCHECKED).When it’s time to transfer abstracts from Bookends, I first click to select the Abstracts Inbox folder in DTP.I then open up Bookends and sort the fields by the “Added” Column. I like reverse chronological order so I can see the last date I imported abstracts. I highlight all the new manuscripts in Bookends.I then move over to DTP and select the Data:New From Template: Education: Reference (from Bookends) menu item. DTP grabs all the highlighted abstracts from Bookends.The imported information includes Author, Title, Year, Abstract, and Keywords. The template also includes a link back to the publication in Bookends. Finally, there is a field for me to add my own comments to the new DTP file.Once the abstracts are imported into DTP, I make sure to index the folder so the new information is included in search. Now that the information is in DTP I am able to take full advantage of its search functionality and artificial intelligence.I make extensive use of Smart Searches in DTP. Using Smart Searches, The abstracts are automatically sorted into folders.For most topics I have two Smart Searches:

1. as shown below that includes the search criteria AND searching for the tag “Abstracts Inbox” we talked about earlier—this finds only abstracts relevant to the search,

2. only the search criteria—this returns ALL information in the database (including abstracts, PDFs, and extracted text as described in this entry).

In DTP, I bring up the Search Interface (Tools: Search….) and type in my search criteria. By default, in the Search Interface, DTP will search across all databases. Notice on the left hand side I have limited the search to my Writing Database. This returns ALL matching information ONLY in my Writing Database. I’ve color coded information extracted from articles using Highlights (for instance, a blue highlight is a direct quote). I can further limit the search by clicking on the “Advanced…” Button and adding additional criteria (e.g. tags, file types, words such as “blue,” etc.).Once I’ve found an abstract that interests me, I locate semantically similar information by highlighting the abstract in DTP and then clicking on the “See Also” button (the Magic Hat).Using this method, all the information I’ve collected (both read and unread) is easily accessible. I use this method extensively when writing. Please let me know if you find it useful too.

Bookends Adds Floating Citation Window

I’d like to bring the latest update of BookEnds to your attention.This update fixed an error that would crash the Bookends when trying to obtain a reference (or link back to a reference in Bookends using a DOI link) from Highlights.app. The latest update fixes this error. Clicking on the reference link in Highlights now selects the reference in Bookends without having to modify the Highlights markdown file. This obviates the need to modify the Highlights markdown file as mentioned  in this entry.The latest update also added a floating citation window to Bookends. Like Papers, the floating citation window is invoked using customizable key combinations. So far, I’ve cited using the floating window in Scrivener, Ulysses, and Highlights.app. It works flawlessly.Well done Jon and Sonny Software!

Reading, extracting and storing scholarly information to supercharge the writing process

Bookends has completely replaced Papers.app as my citation manager. Despite this change, my method of information gathering has changed little since I covered the topic, except for one major addition. I am now finding the majority of information through qxMD Read.My workflow for extraction of annotations differs a bit from when I was using Papers. I now use the combination of Bookends and Highlights.app to extract each of my annotations and notes as separate files. In addition, after highlighting, I’ve been extracting the entire manuscripts in 1-4 paragraph pieces.The whole purpose of extracting annotations and full text from each manuscript is to gain granular access to the information I’ve read. Through its "see also" feature, Devonthink can identify information I’m looking for. But more importantly Devonthink finds related content based on the context of my original query.After importing a manuscript into Bookends, the document becomes available on all my devices. Bookends outshines Papers in its built-in annotation tools as well as its seamless syncing between iOS and Mac.

My current preference is reading PDFs on a 12.9" iPad Pro along with an Apple Pencil. In a pinch, I’ll annotate on my iPhone. On my first pass through an article, using Bookend's built-in tools, I highlight annotations and jot notes in green. I use blue to highlight quotes I wish to use verbatim. When I’m finished annotating, I add the keyword RFP (ready for processing) to the manuscript in the Bookend database.

I have a Smart Folder set up in Bookends that finds the Keyword RFP. Uzsing this method, I can batch process multiple PDFs when I get to my Mac.When I'm ready to extract the information from the PDF into my Devonthink writing database, I export the PDF to my desktop and append the exported file’s name with FT (for fulltext).On my desktop, I open the file with Highlights. I use Copy Hypertext Link:Copy as Text from Bookends, and then open the “Edit” Tab in Highlights. Using use the text URL copied from Bookends to make a markdown link back to the Bookends Reference.When I export my annotations to Devonthink, the URL link will be appended to each file. Clicking the link within any extracted file takes me directly back to the reference in Bookends.I then turn my attention to Highlights.app. I make sure the annotations color preference is checked in Highlights.app. When checked, this preference will append the color of the highlight in each note.

Within Highlights, I use yellow to mark up the entire document over the top of my previous annotations. I highlight in 1-4 paragraph chunks. I also highlight the text of each table and figure.Once I'm done highlighting the entire paper, from within Highlights.app, I export my annotations to Devonthink from the Highlights.app menu. The annotations are saved as individual files to my Devonthink Inbox. I move the folder from the Devonthink Inbox to my desktop and then use Launchbar to move file from my desktop to my annotations folder. I put the desktop PDF in the trash, replace the RFP keyword in Bookends with DTx to remind me I've extracted the paper.

Since I’ve set up my Devonthink database to index this folder,the next time I launch Devonthink, the new information is catalogued and available for writing. The new information is also transferred to Devonthink ToGo to use while writing on my iPad.Using the Artificial Intelligence of Devonthink. I can quickly find related information. When I find an item in my database I want to include in my paper, clicking the link in the annotation takes me directly to the paper in Bookends.app.Using this method, I'm able to take full advantage of the Devonthink AI. Queries within Devonthink simultaneously find annotations, chunks of text and full manuscripts related to my search. I find this workflow invaluable when writing scholarly information.

***This workflow has been modified from here to the end-Please see Modification to Reading, Extracting And Storing Scholarly Information To Supercharge The Writing Process to see what I do instead.***

Keeping Current with Bookends and QxMD Read

In a previous entry, I covered the ways I add new literature to my scholarly manuscript collection. Technology has evolved since that entry. I’ve migrated to Bookends to keep my literature organized, and I have additional ways I find new literature I’d like to read. That’s what I’d like to cover today.In my MacWorld entry (circa 2012), I mentioned the three ways I collected new reading material:

• as a recipient of an Endnote library (when writing collaboratively)

• through personal searches on the web (and liberal use of Google’s automated search terms)

• as attachments to colleague’s emails

Since then, there are two methods I use even more commonly:

• Following individuals with similar interests and using keywords in Twitter.

• Using QxMD Read

Until recently, I was not familiar with QxMD Read. A colleague at Duke, Dr. Jordan Komisarow, brought the app to my attention. According to QxMD, Read “provides a single place to keep up with new medical & scientific research, read outstanding topic reviews and search PubMed. This iPhone & iPad app provides a simple interface that drives discovery and seamless access to the medical literature by reformatting it into a personalized digital journal.” One comment describes the app as “The Flipboard of Medical Literature.”The app was easy to set up. First I made a QxMD username and password. Then I chose my professions of interest. Finally, I chose specific journals, keywords, and collated collections.Once set up, Read began scouring the medical literature for new articles.In a single screen, I scroll through all the new publications (titles, conclusion, journal, etc.).If I find something of interest, I click on the title to find more information, including the entire abstract. While I’m scanning the abstract, Read attempts to download the full-text PDF.About half the time, Read finds a PDF and downloads it into the app. If the PDF cannot be found, I email the entry to my personalized Omnifocus Mail Drop so the paper is added to my Omnifocus Inbox.If the PDF is downloaded into Read, it is then a simple task to move the document into Bookends. The ease of importing full entries is one of the many places Bookends outshines Papers. I click Open In… and then select the Bookends icon.Bookends works its magic, finding one (or sometimes more) suggestions as matches.I choose the matching metadata and Bookends imports the PDF and metadata into its database. The paper and its metadata are immediately available on all of my devices.The process works pretty seamlessly. The only annoyance is that if you save the PDF to your library, it receives a numerical name. You must later open Bookends on your Mac and rename the PDF to your naming convention. This is a small price to pay for such power.I'm finding even more relevant information these days using QxMD Read. My challenge now is finding the time to read and process it all!

My Migration from Papers to Bookends

I have dedicated a lot of time and energy squeaking every ounce of productivity from the Papers app. I’ve used Papers for more than a decade. Over the years, I built many scripts and work-arounds to address the shortcomings of the application.ReadCube purchased Papers in 2016. Because of the time I’ve spent in Papers, I started having angst when I read of certain changes to the software: a subscription model, the loss of Magic Citations, and the loss of integration with Scrivener.I started exploring alternatives to Papers. When I wrote about my interest in migrating away from Papers, several Wippp Readers suggested I check out Bookends. Several folks raved about the tight integration between Bookends and Devonthink as well as Bookends and Tinderbox—two other programs I use heavily in my academic life. Several people also mentioned the LEGENDARY support of SonnySoft, the company  behind Bookends.My initial evaluation of Bookends was far too cursory. Following publication of the blog entry, I received a very nice email from SonnySoft asking me to take a closer look at Bookends. After digging deep into the software, I decided to migrate from Papers to Bookends. I haven’t looked back. It’s been several months now and the more I use Bookends, the more I like it.Things I like about Bookends:

• The interface, although not as modern as Papers, is cleaner and more organized.

• The iOS and Desktop version are better integrated than Papers.

• A PDF on my phone is drop-dead easy to import into Bookends. The PDF and the metadata I choose is automatically synced with the Mac app (although I have to rename the pdf when I get to my Mac).

• All PDFs are saved to a single folder in iCloud, making them easy to access and for Devonthink to index.

• Bookends integrates well with Scrivener (and many other writing clients)

• Dragging and dropping citations into Tinderbox and Devonthink are both seamless. Tinderbox maintains metadata from Bookends. This obviates the need for the KM script I built to move citations from Papers to Tinderbox.

Things I don’t like:

• It was difficult to import Papers library with PDF into Bookends (Bookends only imported about 1/3 of my PDFs).

• I can't export or customize the format of the exported metadata.

• My smart collections don’t transfer to iOS.

• Although I can designate a “watch” folder to import new PDFs into Bookends, this only works with PDFs that are saved to the folder AFTER Bookends is open. This didn't work well with how I collect information so I decided to modify my Hazel script, changing it to launch Bookends instead of Papers.

• I don't like the way duplicate references and / or PDFs are handled.

Over the next few months, I plan to write several entries about my migration and how Bookends has become a critical part of my augmented writing workflow.

Mourning the loss of Integration Between Papers and Scrivener

Early indications, including personal communication with ReadCube personnel, are that Magic Citations (now called SmartCite) will no longer integrate with Scrivener. The new Papers app will only work with Microsoft Word. Bibliography formatting is not an issue for me. The biggest loss is the ability to add citations on the fly in Scrivener without interupting my writing flow.The news of this impending feature loss (along with the announcement of an annual fee) had me scrambling to invesitgate my options for citation managers. During my search, I came across this Wikipedia article, Comparison of Reference Management Software with a great table that collates the majority of software out there.My needs are the following:

1. Ability to organize and search through metadata and pdfs
2. Integrated citation insertion with Scrivener and Ulysses
3. Ability to insert citations while writing on an iOS device
4. Ability to annotate PDFs and export each comment individually appended with the article's metadata
5. Ability to export metadata (to enable my workflows for sense-making and export of annotations)

I looked at the following:

Zotero

Bookends

Mendeley

Endnote

Readcube

No product currently fills the void left by Papers, although the consensus of users (both those seeking alternatives to Papers and those who are being forced to leave Sente) seems to be to move to Bookends. I tried the demo version of Bookends and was not impressed. I am waiting impatiently for the release of ReadCube Papers. If the majority of features are retained, I will likely bite the bullet and pay the annual fee. I plan to figure out a work-around to add citations to Scrivener / Ulysses.I'd be interested in hearing your plans / thoughts on academic citation managers.