PhD Pitfalls III: Pairing the right questions with the right system

​Each year I teach several courses on social network analysis. These are mainly attended by PhD students, and the courses give me the opportunity to interact and discuss a range of projects and ideas with students. These discussions span well beyond the realm of social network analysis, and often throw up plans that I can find somewhat surprising. What has struck me is the frequency at which there is a mismatch between the types of questions a student wants to ask and the system they are applying the questions to. In my experience, this mismatch can be costly, and ultimately reduces the potential for the student to develop a strong publication record.  

Written by Damien Farine
@DamienFarine

​It often all starts with a recently-published paper. Study X shows this really cool new mechanism or pattern, and suggests that it should be of widespread importance (hence why it is published in a high impact journal). Such studies can influence scientific questions over years to follow, and a truly novel study will undoubtedly inspire at least a generation or two of PhD students to test similar a question in their own study species. However, doing so can lead to quite a lot of disappointment—not because the question is a bad one, nor because the system itself is inherently weak, but simply because the question and study system do not lead to a happy coupling.
 
Better understanding high impact papers
 
The first step to avoiding disappointment is gaining a better understanding of the recipe for a high impact paper. The main ingredient is ecological relevance: the biological phenomenon being studied, whether it is generated in an experimental or observational setting, must actually relate to something real that happens in the system that is focal to the study. Good papers are often described as being ‘elegant’ or ‘simple’, largely because they show the hypothesised patterns with little need to mould or work the data. This is because the phenomenon in question was studied on (one of) the best possible systems to test it. Further, such a study usually requires being able to collect a sufficient sample size to be really convincing. The result is often something which sometimes hardly needs any statistics—they simply show the patterns beyond any doubt (see my thoughts on this in the previous post on data analysis).
 
Identifying whether you’ve fallen for the trap
 
The clarity of the findings in a given study can give the impression that showing the same pattern across other systems should be relatively easy to achieve. However, before setting out to testing this, one should ask at least two key questions. First and foremost is:
 
Do I think that the biological phenomenon occurs and is important? For example, why would a specific behaviour be important? What is it about the system that suggests that this behaviour is selectively adaptive? How could the related implications of this behaviour  impact evolutionary processes? There should be no vague answers here (e.g. ‘it can potentially impact disease spread’). Any time that the answers to such questions are not immediately obvious, it’s worth taking a step back to reassess your approach.
 
Second, and equally importantly:
 
Is the biological phenomenon hypothesised to occur likely to be the most parsimonious one? This question raises a whole bunch of topics. In a given system, will hypothesis A be the most likely to explain pattern A (or are there alternative mechanisms?). The relative importance of different types of behaviours or processes could vary across different systems. For example, while affiliative interactions are likely to be among the most common reason for two dolphins or two baboons to be in close proximity, fighting might be the most common reason for two Tasmanian devils to be in close proximity. Thus, the interpretation of the same dataset (spatial proximity) is likely to be really different across different systems, and therefore it is important to make sure that the behaviour (or pattern or process) of interest is a common one in the focal study system, and unlikely to be generated by other behaviours that have the same data signature.
 
The consequences of the mismatch
 
It pays to be very critical when choosing both study systems and questions. Often circumstances throw up opportunities to work on a given system, and when it does, one should take care to think carefully about whether interests align with opportunity. I’ve written in the past about the importance of scientists being passionate about our topic. If the topic and the system don’t align, forcing them together might seem like a good idea (the passion will be there), but ultimately could be a whole lot of wasted time (the data won’t be there). If you are not critical during the design and implementation stage, it’s likely that your reviewers will be, and unfortunately this will happen after months (at least) of time invested on your part. Poor matching of the system and questions could then ultimately result in publishing fewer (they take longer to publish due to imperfect results) and lower impact (submitting to less critical journals) papers. It is important to remember that having success as a researcher isn’t just being good at designing experiments or meticulously watching animals, it starts with asking the right question for the right system.
 
How to avoid becoming trapped
 
A huge part of the scientific process is critical thinking. It is always important to put your critical thinking hat on when it comes to making decisions about your own research. I find the easiest way of doing this is to put myself in the shoes of a reviewer – one that has no vested interest in the study system. Would I really believe that what the authors suggest happens in this system really happening? Supervisors also play an important role when making decisions about a study’s questions. However, sometimes supervisors are so embedded in their project that it can also be hard to for them to see (or acknowledge) a system’s limitations.
 
Step 1: Applying. Many students will apply for a project that is already well defined. Competition for projects is super-tough, so being successful can often feel like your one chance to make it. However, competition doesn’t get easier after the PhD—and you’ll only progress if your PhD was successful (which requires picking good questions and the best system). Be critical about a project that’s advertised, and if possible seek advice from someone independent about the feasibility of the proposed project to work out. The point here is: don’t assume that because a project is advertised it means that it will be good and/or successful. Also, if you’re competitive to get one position, maybe it’s worth waiting it out for a better one.
 
Step 2: Ask your supervisor to brain-storm or raise potential criticisms that someone might have about your project/ideas. Push them to be realistic here, and ask them ignore the strong desire to defend their body of research just for the sake of it. A good avenue that many universities have is the external advisory committees that oversee each student’s plans and ideas—use these wisely by asking for them to be honest and critical. Most importantly, and I can’t stress this enough, be open to any critical comments that are being made. If someone asks whether you think this behaviour A the most logical explanation for why pattern A might be observed, take the question seriously. (And take note that often those that sit on advisory committees will often not openly state major criticisms in order to avoid upsetting candidates, so such questions are usually understated).