RTIPs Submission and Review Process:
A Guide for Program Developers

  1. Intended Audience
  2. Objectives
  3. About RTIPs
  4. Benefits of Participating in an RTIPs Review
  5. Determining Your Program's Submission Readiness
  6. Submitting Your Intervention to RTIPs
  7. The RTIPs Review Process
  8. The RTIPs Rating System
  9. Program Materials
  10. Summary Approval and Publication
  11. Research to Reality (R2R) Promotion
  12. Conclusion
  13. Flowchart: Submission and Review Process

Intended Audience

This guide is intended for program developers who are interested in submitting cancer control and cancer survivorship interventions for review and potential inclusion on the National Cancer Institute (NCI) Research-tested Intervention Programs (RTIPs) website.


Objectives

This guide provides a brief introduction to RTIPs, describes how to submit an intervention to RTIPs, and explains the steps involved in an RTIPs review. This document will help you, as a program developer, to understand:

  • The purpose of RTIPs
  • The benefits of having your program included in RTIPs
  • The criteria RTIPs will use to determine whether your program is eligible for review
  • The steps involved in submitting your program for potential review
  • The documentation you will need to submit for the review, if accepted
  • Your responsibilities and contributions to the review process, as well as those of the RTIPs Development Team, the staff who will coordinate and conduct your review
  • The end product of your review (posting on the RTIPs and R2R websites)

About RTIPs

RTIPs is a searchable database of cancer control interventions. It provides program planners and public health practitioners easy and immediate access to information about interventions and their research base. In addition, it provides access to research-tested program materials that can be used to implement these interventions in community or clinical settings. RTIPs is a voluntary system in which program developers may elect to participate. RTIPs is prominently featured in the Programs section on the Cancer Control P.L.A.N.E.T. (Plan, Link, Act, Network with Evidence-based Tools) Home page at http://ccplanet.cancer.gov. It can also be accessed directly at http://rtips.cancer.gov/rtips/.


Benefits of Participating in an RTIPs Review

RTIPs reviews do require a commitment of time on the part of the program developer to prepare for submission and participate in the review process. However, the effort invested can return a number of benefits:


  • Dissemination and expansion. Participating in a review offers you an opportunity to disseminate the products of your work through the RTIPs website while providing increased visibility and credibility for research-tested approaches to cancer control. While your program materials may not have been developed with dissemination in mind, they can be disseminated directly from the RTIPs website in a downloadable, easy-to-print format or through CD-ROMs requested by interested implementers.

  • Collaboration. You will have the opportunity to collaborate with other cancer control researchers and practitioners through Research to Reality (R2R), an online community with over 1,100 members interested in moving evidence-based programs into practice. Developers of interventions published on RTIPs are engaged by the R2R team to discuss additional opportunities for connecting with others in the online community. To learn more about the R2R community, please visit https://researchtoreality.cancer.gov.

  • Guidance for continued program development and evaluation. Ratings on the criteria assessed during the review process can provide direction for the ongoing development of program materials and future evaluations.

Determining Your Program's Submission Readiness

Before you decide to submit an intervention to RTIPs, you will need to ensure that it addresses one of the RTIPs topic areas and meets the criteria for inclusion. These two important components of submission readiness are described in the next section.


RTIPs Topic Areas

RTIPs accepts interventions in 12 topic areas:

  • Breast Cancer Screening
  • Cervical Cancer Screening
  • Colorectal Cancer Screening
  • Diet/Nutrition
  • HPV Vaccination
  • Informed Decision Making
  • Obesity
  • Physical Activity
  • Public Health Genomics
  • Sun Safety
  • Survivorship/Supportive Care
  • Tobacco Control

Eligibility Criteria for Inclusion

Interventions must meet the following criteria to be eligible for an RTIPs review:


  • Intervention outcome finding(s) must be published in a peer-reviewed journal.

  • The study must have produced one or more positive behavioral and/or psychosocial outcomes (p≤.05) among individuals, communities, or populations.

  • Evidence of these outcomes must be demonstrated in at least one study using an experimental or quasi-experimental design. Experimental designs require random assignment, a control or comparison group, and pre- and post-intervention assessments. Quasi-experimental designs do not require random assignment but do require a comparison or control group and pre- and post-intervention assessments. Studies that are based on single-group, pre-/post-test designs do not meet this requirement.

  • The intervention must have messages, materials, and/or other components in English that can be disseminated in a U.S. community or clinical setting.

  • The intervention must have been evaluated within the past 10 years.

If your intervention meets the above criteria, the RTIPs Development Team welcomes you to submit your program for consideration, following the steps described in the next section. If you have questions about submission or are uncertain about whether your intervention meets the above criteria, please go to the Contact Us page of the RTIPs website and submit your question or concern.


Submitting Your Intervention to RTIPs

There are three steps involved in submitting an intervention to RTIPs:


  • Registering the program,
  • Preparing a submission package of materials, and
  • Participating in a kick-off call.

Step 1: Program Registration

If you are interested in pursuing inclusion in RTIPs, your first step is to complete a Program Registration Form on the RTIPs website at http://rtips.cancer.gov/rtips/register/index.do. This form asks you to confirm whether your intervention meets each of the five eligibility criteria noted above; you will also provide your contact information.


Step 2: Initial Submission

If your program appears to meets the eligibility criteria based on the registration form, the RTIPs Development Team will send you an email inviting you to submit materials so that an official determination can be made about your program's eligibility. The following materials are requested at this stage:


  • One electronic copy of the publication(s) relating to the outcomes of the intervention research project (e.g., article reprints and preprints, final grant reports, chapters,monographs). You must indicate which primary outcomes publication includes the intervention research findings.

  • One complete set of all program materials in the format typically disseminated to the public (electronic, hard copy, or via a web address).

  • Documents that describe the original plan for selecting outcome measure(s) and data analysis (e.g., grant application narratives) and how the intervention/study purpose was presented to participants (e.g., informed consent forms, protocols).

Step 3: Kick-Off Call

The third and final step of the submission process is to participate in a kick-off call with the RTIPs Development Team. The kick-off call provides an opportunity for both parties to ask questions, discuss the RTIPs review process, and identify additional materials, if necessary, that will be needed before the intervention is officially accepted for RTIPs review.

Once it has been determined that both parties wish to proceed with the review, you will be sent an Initial Approval Form. This form is used to finalize important details pertaining to your review (e.g., confirmation of the studies and outcomes that will be reviewed, confirmation of the products to be disseminated, and authorization to initiate program review).


The RTIPs Review Process

After your intervention has been formally accepted for review, it is assigned to one of the Review Coordinators (RCs) on the RTIPs Development Team. The RCs are Ph.D.-level evaluation and behavioral/social science specialists who serve as a technical liaison between developers and the pair of expert reviewers assigned to each review. The expert reviewers are Ph.D.-level professionals who have expertise in one or more of the RTIPs topic areas. The identity of the reviewers assigned to your review will be kept confidential to ensure the integrity of the review process.

Both the RC and the reviewers have a role in rating the intervention. The reviewers rate the intervention's Research Integrity, while the RC rates Intervention Impact, Dissemination Capability, and RE-AIM criteria. More information about the RTIPs Rating System is provided in the next section. To facilitate the reviewers' work, the RC provides the reviewers with a review packet that includes all the study articles and other materials to be reviewed; detailed notes, including any supplemental information you may provide to explain how the study or studies were conducted; and information on the rating anchors. The RC is also responsible for drafting the RTIPs summary for your program, which will be posted on the RTIPs website at the conclusion of the review.


The RTIPs Rating System

Intervention evaluation and program materials are evaluated in four areas for the RTIPs review. These rating areas are described below.


Research Integrity

Research Integrity reflects the overall confidence reviewers can place in the findings of a program's evaluation based on its scientific rigor. The Research Integrity rating system comprises 16 criteria scored by independent experts. Scores on each criterion are given on a 5-point scale ranging from low quality to high quality. The overall integrity score is an average of the 16 criteria reflecting the merits of the science that went into the program evaluation.


Intervention Impact

Intervention Impact describes whether, and to what degree, a program is usable and appropriate for widespread application and dissemination. This rating is determined by the RC. Population Reach and Effect Sizes are separately rated on a 5-point scale; these ratings are then combined using the RTIPs Intervention Impact rating table to determine the impact score.


Dissemination Capability

Dissemination Capability refers to the readiness of program materials for use by others as well as a program's capability to offer services and resources to facilitate dissemination. The rating is given on a 5-point scale ranging from low quality (1.0) to high quality (5.0). Dissemination capability is measured through the assessment of three areas:


  • Quality of implementation materials
  • Training and technical assistance protocols
  • Availability of quality assurance materials to determine whether implementation was done with high fidelity to the original model

RE-AIM

RE-AIM is a five-step framework designed to enhance the quality, speed, and public health impact of efforts to translate research into practice:


  • Reach your intended target population
  • Effectiveness or efficacy
  • Adoption by target staff, settings, or institutions
  • Implementation consistency, costs, and adaptations made during delivery
  • Maintenance of intervention effects in individuals and settings over time

A RE-AIM scoring instrument has been developed to assess the first four steps above. The instrument consists of 22 items within 4 dimensions: Reach (5 items), Effectiveness (3 items), Adoption (6 items), and Implementation (8 items). The Maintenance step is not currently assessed by RTIPs. Additional information about RE-AIM can be found at http://re-aim.org.


Program Materials

As noted in the eligibility criteria, RTIPs programs are required to have messages, materials, and/or other components, in English, available for dissemination to potential implementers in U.S. community or clinical settings. Before your review begins, the RTIPs Development Team will work with you to determine how these materials will be provided to RTIPs users. There are two dissemination options:


  • Materials will be made available to the public through the RTIPs website, at no charge to you. NCI will provide RTIPs users with a CD-ROM of these materials upon request.

  • You disseminate the materials to RTIPs users. If you choose this option, a snapshot of all required program materials will be shown on the Products page of the RTIPsintervention summary. This allows users to preview the materials but does not provide the full documents online. Users are provided with your contact information to obtain cost and ordering information. If your intervention is web-based and currently disseminated to the public, a direct link to your website will be posted instead.

RTIPs programs are also required to have written implementation guidance explaining the steps required to implement the program. If you do not have an existing implementation guide, one will be created for your program using an established NCI template. The Implementation Guide will be developed by the RTIPs Development Team from information you provide, or you may draft one using the NCI template.


Summary Approval and Publication

At the conclusion of your review, you will receive a draft intervention summary containing descriptive information about the program; information about the study reviewed, including key findings; and the final RTIPs ratings. An example of a completed RTIPs summary can be found at http://rtips.cancer.gov/rtips/programDetails.do?programId=892493

You will have the opportunity to request clarification on the ratings and suggest edits to the summary before providing your final consent to post the summary via the Final Approval Forms. NCI considers all edits by developers and may confer with the RTIPs Development Team to resolve any technical questions or concerns.


Research to Reality (R2R) Promotion

For each completed RTIPs intervention summary, an associated R2R Featured Partners segment will be produced for the R2R website. Initiated in 2012, the Featured R2R Partners page provides a profile about the intervention's developer(s). This initiative has been created by NCI to help foster an ongoing dialogue around successes, issues, and challenges associated with the implementation, adaptation, and evaluation of the intervention. You will be notified by the RTIPs Development Team and R2R Team when your intervention summary and R2R Featured Partners profile are published on the RTIPs and R2R websites, respectively.

For an example of a dialogue with an R2R featured partner, please visit visit the Research to Reality Web page of the RTIPs site.


Conclusion

RTIPs is a way for you to share advances in implementation science and improve practices in cancer control and survivorship. Although participating in a review does require some time and effort, the benefits of having your program included in RTIPs should prove to be worth the investment. Below is a flowchart to help illustrate the process.

rtips_process_flowchart