Class 1 - Research I: Principles
Agenda
Introductions (30 minutes)
- Getting to know each other
- Syllabus and materials overview
- Typical class flow
Skills corner: The craft of research (15 minutes)
Break (5 minutes)
Core paper discussion (55 minutes)
Summative lecture (15 minutes)
Introductions
A little about me
A little about me
A little about me
A little about you
Please remember to fill out your introductory Qualtrics survey so that I can learn a bit about you and your goals for the class!
Today, let’s level set on your familiarity with some key ideas: Pollev.com/drfox
Syllabus and materials overview
Dropbox (download a local copy for yourself, feel free to add selective highlights for group discussions)
Typical class flow
Part I: Conceptual grounding and agenda setting
Introduction to topics covered
Skills corner (this will migrate over the semester)
Part II: Core paper discussion
We will discuss the 2-3 papers that all students have been assigned to read in detail
These papers typically will provide a mix of conceptual background and how-to guides
Break
Typical class flow
Part III: Activity period
(Weeks 3 – 7) Compare / contrast: One group tasked with reviewing two additional papers to explain their points of intersection, divergence, and ties to core papers
(Weeks 8 – 12) Replication: One group tasked with using data from one of my current or published papers to replicate analyses and comment on that process and raise questions for general awareness
Typical class flow
Part IV: Summative lecture on concepts
I will make a brief presentation to tie together and highlight key concepts
Elements missed in the general discussion will be given greater focus
Skills corner
Reading, understanding, writing, crafting
Scholarship is more or less developed in that order
First you read (a lot, broadly and narrowly)
Enough reading helps you start to understand (both in terms of content as well as analysis)
When you have understand enough, you can start writing your own thoughts
Those thoughts are put out into the world and receive “feedback” through the review process
Feedback helps you course correct, identify limitations in your understanding or how you are articulating your ideas
After enough cycles, you begin to craft your research
Crafting research, simply put
- Research (at least in our field) is not manufactured, it is crafted:
to make or produce with care, skill, or ingenuity (Merriam-Webster Dictionary)
Skill: Developing “soft” (theory-building) and “hard” (model-building) capabilities to make and defend a thesis
Ingenuity: Having the creativity and base of information to go beyond what is known
Care: Exerting due time and effort in putting together your analyses and arguments
Scholarly contributions
Your task in this program is to advance human knowledge in a non-trivial way
Academics tend to prefer theoretical contributions (we will see why later)
- This means that the web of knowledge for a given theory has been modified
These contributions are often directional, with respect to a target audience
- Example: Nash equilbria (von Neumann versus economists)
- This may run counter to your goals and preferences to make a practical contribution (i.e., explain versus optimize)
A process representation (we will revisit this later)
Discrete skills required to complete that process
Reading articles for multiple purposes
Summarizing your observations
Articulating research questions and associated hypotheses
Selecting appropriate empirical contexts and collecting data
Employing different analytical techniques to examine that data
Drawing inferences and explaining how these conclusions can advance the literature
Motivating the class: Research methods matter!
As a case study, Let’s consider the last paper assigned for today: Delios et al. 2025
Delios et al. (2025)
Analysts used a wide variety of specifications to conduct their analyses. No two analysts adopted the same approach when we consider the variables, method, and sample selec- tion simultaneously. Material 4 on OSF provides a detailed summary table describing the specifications employed by each analyst for each research question. Supplement 2 sum- marizes the control variables used by the analysts.
To achieve a standardized measure of the effect sizes of the independent variables on dependent variables across all analyses, we computed the marginal effect sizes (Breznau et al., 2022; Fey et al., 2023), which represent the increase in a dependent variable for a unit increase in an independent variable. In Fig. 1, we present the distribution graphs of the effect size estimates found in the separate analyses for each research question. For all four research questions, the range of estimates encompasses both negative and positive values and crosses zero.
Beliefs about the research question measured at the end of the study were more strongly related to empirical esti- mates than beliefs measured before receiving the data. This is a similar pattern to that previously captured in Silberzahn et al. (2018), but with more units of observation, lending the results further credibility. It seems that scientists rationally updated their beliefs considering the evidence. We again find no apparent confirmation bias, such that researchers’ precon- ceptions strongly predict their analytic choices and results
Core Readings for Today
Preamble
I have provided some discussion questions for us to consider in case we need to get the ball rolling.
We may or may not discuss those questions depending on the flow of the class.
In general, I would rather talk about your ideas and questions rather than these “canned” items.
Readings
Popper, K. R. (2002). The Logic of Scientific Discovery. Routledge. [Ch .1]
Mantere, S., & Ketokivi, M. 2013. Reasoning in Organization Science. Academy of Management Review, 38(1), 70-89.
Nosek, B. A. & Errington, T. M. 2020. What is replication? PLOS Biology: 1-8.
Rynes, S. L., & Bartunek, J. M. (2017). Evidence-Based Management: Foundations, Development, Controversies and Future. Annual Review of Organizational Psychology and Organizational Behavior, 4(1), 235-261.
Popper (2002)
The Logic of Scientific Discovery. [Ch .1]
According to the view that will be put forward here, the method of critically testing theories, and selecting them according to the results of tests, always proceeds on the following lines. From a new idea, put up tentatively, and not yet justified in any way — an anticipation, a hypothesis, a theoretical system, or what you will—conclusions are drawn by means of logical deduction […]
[Then,] there is the testing of the theory by way of empirical applications of the conclusions which can be derived from it. [p. 9]
Popper (2002)
Discussion Questions
In your view, what is the main point?
Do this worldview currently inform your work? How might it?
Mantere and Ketokivi (2013)
Reasoning in Organization Science. Academy of Management Review, 38(1), 70-89.
Labels aside, a closer look at research practice reveals that researchers across research traditions use all three forms of reasoning. It is hardly surprising to observe that we all make inferences to a case (use deduction), inferences to generalizations (use induction), and inferences to explanations (use abduction). Thus, using reasoning types as labels to describe entire research designs is misleading. Instead, differences between research approaches, whatever they may be, are found not in the types of reasoning used but, rather, in how the three reasoning types are used in conjunction with one an other. (p. 76)
Mantere and Ketokivi (2013)
Discussion Questions
What mode(s) of reasoning do you tend to rely on in your current work?
What concrete practices did you draw from this paper, if any?
Nosek and Errington (2020)
What is replication? PLOS Biology: 1-8.
To be a replication, 2 things must be true: outcomes consistent with a prior claim would increase confidence in the claim, and outcomes inconsistent with a prior claim would decrease confidence in the claim. The symmetry promotes replication as a mechanism for confronting prior claims with new evidence. Therefore, declaring that a study is a replication is a theoretical commitment. Replication provides the opportunity to test whether existing theories, hypotheses, or models are able to predict outcomes that have not yet been observed. (p. 2)
Nosek and Errington (2020)
Discussion Questions
Do you agree with their definition of replication?
How does this fit in with the replication crisis?
Rynes and Bartunek (2017)
Evidence-Based Management: Foundations, Development, Controversies and Future.
Management academics have long noted a large gap between academic research and managerial practice. […] Some have viewed the causes of the gap as lying primarily with academic researchers, who are characterized (perhaps caricatured) as having become overspecialized, self-referential, obsessed with theory, excessively mathematical, jargonladen, unconcerned about practical problems, and dismissive of practitioners […] Others have focused on practitioners, who are sometimes characterized or caricatured as research phobic, anti-intellectual, susceptible to unproven fads and fashions… (p. 236)
Rynes and Bartunek (2017)
Discussion Questions
Are you familiar with evidence-based practice from your current work?
Where might you fit in helping to advance evidence-based management? How might you go about doing it?
Lecture - Principles
Preamble
What follows is my personal, idiosyncratic synthesis of the pieces that we have read to date. To be clear, many interpretations are possible due to these articles’ collective:
- richness
- overlap
- distinctive features
Preamble
Furthermore, there are multiple plausible criteria to judge quality research and a lack of universal consensus given the multiplicity of aims and epistemological orientations.
This is not to say anything goes; rather, I am trying to highlight the limits of my knowledge and that my unique (quantitative, positivist, deductive) lens necessarily abstracts away features from these complex discussions.1
Some key principles of research design
Falsifiability: Can we modify our state of knowledge based on the result of our study?
Defensibility: Do our arguments appropriately employ modes of inference?
Applicability: Does our study actually impact the state of practice?
Replicability: Will our work bear re-examination in a similar or new context?
Falsifiability
Falsifiability provides a basis for to use abductive reasoning to augment pure deductive reasoning.
The latter is true a priori if the premises are valid.
Pure deduction can transform our understanding of the system (think theorems in math).
But deduction cannot uncover truth beyond the system’s universe of premises.
We must use other means (such as abduction) to draw conclusions about what our data is telling us (i.e., concluding a theory is supported or not) to determine whether our deductive premises are valid by comparing our observation to our expectation (i.e., a proof by contradiction or falsification)
Falsifiability
My proposal is based upon an asymmetry between verifiability and falsifiability; an asymmetry which results from the logical form of universal statements. For these are never derivable from singular statements, but can be contradicted by singular statements. - Popper (2002, 19)
Defensibility
If the logic of our arguments are defensible and the evidentiary basis is sound, we are better able to act upon the conclusions with confidence.
Like Mantere and Ketokivi (2013), my coauthor Zeki and I argue that the relative importance of each reasoning mode varies based on the research design employed and intended contribution.
Defensibility
Defensibility
Toulin diagrams provide one way to map out any argument, regardless of the method of inference:
Applicability
But our arguments and conclusions, even if correct, are not that helpful if they aren’t applicable to real-world problems.
There are two corollaries to this observation:
We should pick problems that actually matter, not just intellectual curiosities.
We should not be hamstrung by our ability to tackle important problems.
Applicability
I was recently at a brown-bag seminar where a pair of management colleagues were seeking advice about a preliminary research idea. It took just a few minutes for us all to agree that their research question was fascinating. It addressed an extremely interesting issue that both academics and practicing managers would like to learn more about. The only problem: the presenters had no theory. So, we spent the entire session going through our collective mental catalogues of theories that might be invoked so that the project could proceed and have some prospect of publication. People were men- tioning theories I’d never heard of. We became frenzied, nearly desperate: “Good god, there must be a theory out there that we can latch onto.” - Hambrick (2007)
Replicability
Finally, the structure of our empirical base presumes that the research was performed in good order and that the findings are replicable within their domain of applicability.
Said differently, if we are to make deductive inferences based on extant literature (i.e., to test and build upon that theory), we want to feel comfortable that is well-founded and that we can make inferences about specific cases from the general rules encoded by existing theory.
Replicability
[A]n accumulation of evidence that points to empirical regularities provides us with a much broader and more generalized understanding of the world. Such empirical regularities are known as ‘stylized facts’. - Helfat (2007)
The relative importance of each principle
We can consider four basic “classes” of research in management:
basic disciplinary research (primary studies in AER, AJS)
applied research conducted in a management context (primary studies in AMJ)
data-driven decision making derived from primary studies (systematic reviews in IJMR, JOM)
practitioner-focused outlets (articles in HBR, CMR, popular press)
Where might, for example, applicability be more highly valued? Falsifiability?
Preparation for Next Class
Next class
Research II: Positions
Huff, A. S. (1999). Writing for Scholarly Publication. [Chs. 1, 3]
McGrath, Joseph E. (1981) Dilemmatics: The Study of Research Choices and Dilemmas, American Behavioral Scientist, 25, 2, 179-210.
Simsek, Z., Heavey, C., Fox, B. C., & Yu, T. 2022. Compelling Questions in Research. Journal of Management, 48(6), 1347-1365.
Tushman, M., & O’Reilly, C. (2007). Research and Relevance: Implications of Pasteur’s Quadrant for Doctoral Programs and Faculty Development. AMJ, 50, No. 4, 769-774.
References
Footnotes
My stance is probably better captured with the idea of post-positivism, but I don’t have a great reference to share here - but here is an overview: https://en.wikipedia.org/wiki/Postpositivism.↩︎