September 04, 2013
Many of us are asked to review both manuscripts and grant proposals. Although there are similarities, there are significant differences that are not always so apparent. As someone who has been both a faculty member reviewing proposals and a recent NSF Program Officer (PO) in the Geography and Spatial Sciences (GSS) program who has had to use reviews to make decisions regarding funding recommendations, I want to share with you some thoughts about writing effective reviews for NSF.
The use of 'ad-hoc' (external) reviewers to assist in proposal evaluation is critical to the review process at NSF and POs take considerable time to select reviewers for their expertise appropriate for the proposal they are being asked to review. A thoughtful and timely review provides input to the advisory panel, POs, as well as to the Principal Investigators (PIs). Below are some specific suggestions for preparing effective reviews.
1. Familiarize yourself with the NSF merit review criteria. NSF is well known for its merit review process. If you are new to reviewing for NSF or have not reviewed in a while, it is worth (re)familiarizing yourself with the NSF merit review criteria as NSF defines these things in a particular way and these have recently been rearticulated. Be sure to pay attention to the request letter the Program Officer emails you as there may be specific requests (some programs such as GSS have additional review criteria above and beyond the standard review criteria at NSF). The letter will also provide you with the most up to date materials you should be referencing. As of this writing the most up to date material on merit review at NSF is found athttp://www.nsf.gov/bfa/dias/policy/merit_review/facts.jsp . In your review you are asked to provide comments on the strengths and weaknesses of the proposal’s Intellectual Merit, Broader Impacts, Data Management Plan, and Post-Doc Mentoring Plan (where applicable). If you are unsure how to tackle all of these parts then I suggest that you answer the specific merit review questions (categories) provided to you as a reviewer. This helps you prepare a complete review.
2. Realize that reviewing for NSF is different from manuscript review. This is because many programs at NSF (GSS included) do not have a formal 'revise and resubmit' category. This means that you need to really focus on the qualityof the specific proposal that you are reading, not one you imagine or wish the PI has prepared. Program Officers cannot send a proposal back and say, "fix these three things and we will fund you" which can happen in journal peer review. PIs can certainly take the feedback and rework their proposal and submit it again (which many do). That new (revised) proposal will go through another full review process and it will be judged again, but not officially against its earlier version as it is treated as a new proposal. Reviewers of the new proposal may or may not be those who reviewed it before. This is the case even with GSS’s new ‘One-Plus’ system.
3. Align your rating and comments. One of the most important components of an effective review is to have your rating reflect your comments. Nothing is more confusing to PIs and frustrating to POs than misalignment between rating and comments. The ratings are as follows:
Excellent: An outstanding proposal in all respects; deserves highest priority for support.
Very Good: High quality proposal in nearly all respects; should be supported if at all possible.
Good: A quality proposal, worthy of support.
Fair: Proposal lacking in one of more critical aspects; key issues need to be addressed.
Poor: Proposal has serious deficiencies.
If you have any concerns about the proposal's research design or its methodology, then it should not be given an 'Excellent' rating. Excellent means that it is a highly innovative idea, with a very complete and well-designed research plan. Too many reviews are submitted that consist of an Excellent rating accompanied with a long list of issues with the proposal. This is very confusing to PIs. Many programs at NSF are extremely competitive, GSS was recently funding at only about the 12% rate, so an Excellent rating should be reserved for the truly exceptional proposal. Reviewers often want to rate something as Excellent for the topic and the potential of the research, and read into the proposal what is not there. The rating needs to reflect the submitted proposal as written and submitted.
4. A good review is not just a summary of the proposal. Most reviewers feel the need to offer a synopsis, which is fine, but I suggest keeping this short and focused. The bulk of the review should constructively address strengths and weaknesses of the intellectual merit and broader impacts of the proposal and should not just describe what the proposal is about. Also avoid simply stating that this is a “good/poor proposal” or it is the “best/worst proposal I have ever read.” Why is it a good/poor proposal? What makes is the best/worst you have ever read? A review is not just an assessment of whether the PI has met the review criteria, but how well (or not) they have done so. Be specific in your comments.
5. An effective review consists of more than a few sentences or bullet points. Avoid one sentence reviews, bulleted lists, and the use of shorthand. A short or incomplete review makes the PIs and POs wonder whether you really read the proposal. If you do not have time to do a full review, then let the Program Officer know. Keep the tone positive and constructive; avoid a brusque, rude or dismissive tone about the topic and/or PI. Don't demand, suggest. The review should be about the submitted proposal, avoid 'cheerleading' a topic or PI. Please avoid typos in your review especially if you note the proposal containing these. Realize that the FastLane system converts accents and auto-formatting features into rogue question marks in your review so avoid using these, turn them off, or review your text before hitting the final submit button.
6. Return your review by the requested date. Ad-hoc review requests are usually solicited before advisory panels meet and are designed to offer expert input to the advisory panelists who are more likely to be generalists on some topics. Therefore reviews are most useful to all in the process if they are received BEFORE the advisory panel meeting, which is why you are provided with a date for returning your review. In some cases a review received after the panel meeting is still helpful, but usually in a more limited way. Please communicate with relevant NSF POs as soon as possible if the specified deadline does not work for you.
7. Be sure not to self-disclose in the review. The review process is highly confidential at NSF which means that PIs do not and should not know who their reviewers were. If you want to provide feedback to the PIs from your own experience (including providing them specific references) use language that is broad and generic and does not flag the comments as coming from you.
8. Review requests go to at least 4-6 reviewers. This means that you are one of a number of reviewers (usually from 3 to 8 total) selected for your expertise, and you should not feel that you have to address everything in the proposal if it goes beyond your expertise and comfort zone. If you feel that there are parts of the proposal that you are not comfortable commenting on, then just focus on the parts that you can comment on and say so in the review. It is up to Program Officers to select a range of reviewers that can provide feedback on all aspects of the project.
9. Never hesitate to communicate with Program Officers at NSF. The request letter is a long computer generated form letter, and it may feel mechanical, but please realize that there are real people running programs at NSF and they are willing to communicate with you. Check the relevant program website; for GSS that is: http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=503621 . All current GSS Program Officers can be emailed at gss-info@nsf.gov .
Program Director, Environmental Studies
Advanced Academic Programs
Johns Hopkins University
No comments:
Post a Comment