Digital Scholarship Centre

Upload references/articles to a review

• Click on the review name.
• Click Show.

• Files may be added anytime throughout the workflow. If you are adding references to a non-empty review, click New search or Add new in the Search methods box to go to the upload page. Otherwise, continue to the next step.

On the right hand side of the upload screen, there are links to guides to show you how to download your references from various reference managers or databases. This means you will have already completed your searches of various databases and either saved the references in a reference manager (EndNote, Zotero, Mendely, etc.) which you would then export in order to upload into Rayyan, or you may have saved your searches as files downloaded directly from the various databases you searched (e.g. PubMed) and you can upload those files directly to Rayyan.

• On the upload page, click Select files.
• Locate your reference files on your local disk using the Rayyan accepted format, then click on upload.
• Click Continue.

Once the references/articles are imported, Rayyan will automatically populate the filters on the left.

• To delete an import, click the trash icon to the right of the record in the Search methods box, then confirm.

About the reviewer role

The reviewer role can allow review owners to invite reviewers with more limited permissions than Collaborators.

Permissions of the reviewer role

The Reviewer role is limited to performing the tasks required of every reviewer. They can make inclusion decisions, apply labels and reasons, and add notes to references. However, the Reviewer cannot upload references of full texts, copy, deduplicate or export references. The Reviewer cannot change review configuration settings either, such as adding Keywords for inclusion or exclusion.

Purpose of the Reviewer role

The ability to invite Reviewers assures you that once the screening process begins your review will be protected from accidental or undesirable manipulation or handling of data. Protecting user data is a sacred trust and this is just one additional way Rayyan allows you to control access to your reviews and review data.

How to invite a Reviewer

You may invite an unlimited number of Reviewers to a review. This new addition is visible under the "Invite to Review" tab.

The mobile app for reviewers

Manage collaborators in the review

• Expand the review details by clicking on the review name.
• To add collaborators, click Invite.

Collaborators will then receive an email notification. Once they sign into their Rayyan account (using the same email that they were invited with), they can find this review listed under the Collaboration Reviews tab.

• Select Collaborator from the drop-down. A collaborator can do everything on the review except deleting the review, inviting others, changing the blind status, and deleting search methods not created by them.
• Enter emails of the collaborators you want to invite in the Emails box.
• Add a reason/message in the Reason/Message box.
• Click Invite.
• To revoke a collaborator's access and remove their contributions from view, click Delete.
• To revoke a collaborator's access without removing their contributions from view, click Revoke. Note: this is not an option for the free version.

• Once a collaborator's access is revoked, you can restore their access by clicking on Restore Access, or delete their contributions from view by clicking on Delete.

Rayyan automatically detects and resolves 100% duplicate articles from searches. Rayyan then offers duplicate resolution for statistically likely duplicates by comparing the title, author, journal and year. 

Detecting duplicates

• Click Detect duplicates.

• Click OK.

You will see the message: Duplicate detection is running, please wait. Detecting duplicates can be performed once on each unique dataset in a review. If you wish to run it again, you will have to add or delete references/articles from your review first. If you see the message: A search must be added or deleted before running duplicate detection again, this means that you have already run duplicate detection on your dataset. See the next step on how to resolve duplicates.

Once duplicate detection is finished, you will see the message: x duplicates found. You will also see the Possible Duplicates facet in the workbench on the left.

• Under Possible Duplicates, click Unresolved to filter by unresolved duplicates.
• Click on the record you wish to resolve.
• Click Resolve duplicate.

• The detected duplicates will be listed on top of each other in the bottom section. A confidence percentage will be displayed for each. Compare the detected duplicates and click Delete if the references were duplicates, or Not duplicates otherwise. in addition there is the auto-resolver feature for large datasets of articles to de-duplicate. Auto-resolver enables users to resolve duplicates with high confidence percentage in one click, speeding up the process of de-duplication.

If 2 (n) duplicates are found, you can make a decision (Delete/Not a duplicate) on only 1 (n-1) and the other will be moved to be resolved automatically. This is to ensure that you do not delete them all by mistake.

Screen references/articles

• On the reviews page, click on the review name. By default, the Blind is On. Blinding blocks decision, labels and notes from view is part of the effort to minimise risk of bias. If you wish to turn it off, click Blind On button.
• Click Show to open the review page.

• On the review page, you will see a list of your uploaded references. Click on a record to view its full details.

You can see all of the uploaded information at the bottom of the section. It is worth noting that the Topics field will include all the keywords defined in the uploaded files, in addition to others that are added by our machine learning algorithm. This is helpful in cases where not enough keywords (or no keywords at all) are defined in the files.

• To include an article, click Include or press I on your keyboard. Your name will be shown in a green box to the left of the article title.

• To exclude an article, click Exclude or press E on your keyboard. Your name will be shown in a red box to the left of the article title. You can also add an optional exclusion reason by either choosing on of the default reasons, or typing a new one.

• If you are not sure, click Maybe or press U or M on your keyboard. Your name will be shown in a white box to the left of the article title.

You can see that the facets/boxes on the left will be updated to reflect the applied customisations.

You can also add a label or a note to any of the uploaded articles. Labels are short and filterable, while notes are usually longer and non-filterable.

• To add a label, enter the label you want to add in the Label box, then click + ... (new).

The label will be shown in a blue box to the left of the article title.

• To remove a label, click on the article you want to remove it from, then click - (label) in the Label box.

• To add a note, click Add Note to open the Note dialog box. Type in your note then click Save.

• To delete a note, click the trash icon to the left of the note, then confirm.

Rayyan's AI-powered 5 star relevance ranking system helps reviewers accelerate their systematic reviews by learning patterns in decision criteria as reviewers make decisions and using that learning to rate the probability that an article will be included.

To use the 5 star relevance rating 

The 5 star relevance ranking system helps reviewers accelerate their systematic reviews by learning patterns in decision criteria as reviewers make decisions and using that learning to rate the probability that an article will be included.

• Make at least 50 inclusion and/or exclusion decisions in your systematic review, with at least 5 decisions of both types (included and excluded).
• Click Computer Ratings.

You will see the message: New ratings are being calculated while the AI engine calculates the likelihood of each article to be included in your literature review (5 stars is most likely).

Once it is finished, you will see the message: New ratings available. You'll also see star ratings, in the Ratings column at the right, ranging from zero stars - near zero likelihood of inclusion to 5 stars - very high likelihood of inclusion. All decided references by any collaborator at the time of computation won't have ratings.

• You can now sort your review by least or most likely to be included by clicking on the sorting arrow besides the column name. When sorting o the 5 star rating system, the order is precise - in other words, the exact calculation dictates the order while the visible stars show an approximation.

• As you make more decisions, you can compute ratings again at any time. The more decisions you make, the more accurate the ratings will be as Rayyan learns more from each decision.

Use workbench facets/filters

• After uploading your references, the workbench on the left of the review page will be populated with facets/filters based on the important data.
• Selecting multiple terms inside a single facet will yield the OR function.

• Selecting terms across facets will yield an AND function (your search results will be further constrained by adding more filters.

• You can click More>> at the bottom of the facet to see additional terms.
• You can click on [Clear] to the right of the facet title to clear all the selections inside that facet.
• You can add new terms by clicking on [Add new] to the right of the facet title. You can remove terms by clicking on the trash icon to the right of the term count.
• You can see the number of matching references next to each term. The terms in each facet are sorted by that number.
• You can move workbench facets and reorder them by dragging and dropping the title section.
• You can minimise a facet by clicking on - to the right of the facet title. You can open the facet again by clicking + to the right of the facet title.

Facets/filters include:

• Possible Duplicates: filters articles by duplication status (Unresolved/Deleted/Not duplicates/Resolved).
• ​​​​​​​Inclusion decisions: filters articles by the decisions (Undecided/Maybe/Included/Excluded/Conflict).
• Decision by: together with the Inclusion decisions filter, you can filter by decisions made by a specific collaborator (e.g. filter by Included and collaborator1 to show the articles that collaborator1 included). 
• Minimum/Maximum collaborator decisions: Filters articles by number of decisions (At least 1/At least 2/At most )/At most 1 ...). This filter is useful when the blind is on (you can't see collaborators decisions) but would like to limit the number of decisions on your references.
• Search methods: Filters articles by the source of the data.
• Keywords for include: Filters by articles by the listed keywords. These keywords will be highlighted in green while the Highlights are ON.
• Keywords for exclude: Filters articles by the listed keywords. These keywords will be highlighted in red while the Highlights are ON.
• Labels: Filters articles by labels. This filter is added only when there is at least one label.
• Exclusion reasons: Filters articles by exclusion reasons. This filter is added only when there is at least one exclusion reason.
• Topics: Filters articles by topics. This filter is populated by Rayyan's natural language processing and machine learning algorithms.
• Locations: Filters articles by locations. This filter is populated by Rayyan's natural language and machine learning algorithms.
• PDF full-text types: Filters articles by PDF full-text types. This filter is added only when there is at least on uploaded PDF full-text.
• Abstract languages: Filters articles by abstract languages. Rayyan detects the language of abstract(s) and adds this filter only if there are 2 or more detected languages.
• Main language: Filters articles by main language. This filter is added only if there are 2 or more languages in the imported reference file.
• Publication types: Filters articles by publication types. This filter is added only if there are 2 or more publication types in the imported reference file.
• Journal: Filters articles by journal name. This filter is added only if there are 2 or more journals in the imported reference file.
• Authors: Filters articles by authors. This filter is added only if there are 2 or more author combination in the imported reference file.
• Year: Filter articles by year. This filter is added only if there are 2 or more years in the imported reference file.

Copying references or articles from one review to another

• On the review page, select filters from the toolbox on the left to select the subset of the articles you wish to copy. 
• Click Copy.

• In the Copy options box:
  • Select Filtered to copy your filtered dataset, or All to copy all of the review articles.
  • Choose the target review that you want to copy the articles to from the list of all reviews.
• Click Copy.

You will then see the following message:

Your articles will then be copied to the target review. Only the original information will be copied.