First, I consider how the question being addressed fits into the current status of our knowledge.
Second, I ponder how well the work that was conducted actually addresses the central question posed in the paper.
Do the hypotheses follow logically from previous work? To what extent does the Discussion place the findings in a wider context and achieve a balance between interpretation and useful speculation versus tedious waffling? (Then, throughout, if what I am reading is only partly comprehensible, I do not spend a lot of energy trying to make sense of it, but in my review I will relay the ambiguities to the author.) I should also have a good idea of the hypothesis and context within the first few pages, and it matters whether the hypothesis makes sense or is interesting. I do not focus so much on the statistics—a quality journal should have professional statistics review for any accepted manuscript—but I consider all the other logistics of study design where it’s easy to hide a fatal flaw.
Mostly I am concerned with credibility: Could this methodology have answered their question?
I always read the paper sequentially, from start to finish, making comments on the PDF as I go along.
I look for specific indicators of research quality, asking myself questions such as: Are the background literature and study rationale clearly articulated? (I usually pay close attention to the use—and misuse—of frequentist statistics.) Is the presentation of results clear and accessible? That usually becomes apparent by the Methods section.As junior scientists develop their expertise and make names for themselves, they are increasingly likely to receive invitations to review research manuscripts.It’s an important skill and service to the scientific community, but the learning curve can be particularly steep.I read the digital version with an open word processing file, keeping a list of “major items” and “minor items” and making notes as I go.There are a few aspects that I make sure to address, though I cover a lot more ground as well.Then, right in the Introduction, you can often recognize whether the authors considered the full context of their topic.After that, I check whether all the experiments and data make sense, paying particular attention to whether the authors carefully designed and performed the experiments and whether they analyzed and interpreted the results in a comprehensible way.If the answer to all four questions is yes, then I’ll usually agree to review.I am very open-minded when it comes to accepting invitations to review.I see it as a tit-for-tat duty: Since I am an active researcher and I submit papers, hoping for really helpful, constructive comments, it just makes sense that I do the same for others.So accepting an invitation for me is the default, unless a paper is really far from my expertise or my workload doesn’t allow it.