Measuring What Matters in Research

• Research impact should be measured by influence on classroom teaching, policy decisions, and organizational change—not citation counts alone.
• Executive summaries and other accessible formats translate dense scholarly research into insights practitioners can actually put to use.
• Strategic partnerships among business schools, academic associations, and policymakers create shared frameworks for advancing research’s real-world impact.

Transcript

Tammy Madsen: [00:12] If we take seriously the idea that impact is about changing how the world thinks and acts. Then we have to move beyond evaluating research on a narrow set of accounts and to understand influence more broadly. This means asking different types of questions.

[00:26] Who is using this research? How does it inform policy? How is it influenced by practice? What’s happening in the classroom? Are we making fundamental changes in organizations based on this research?

[00:35] Citation counts are a signal of academic contribution, but they’re only one part of an article’s impact lifecycle. Business schools have an opportunity to adopt a more balanced approach by taking into consideration both the rigorous publications, contributions, and engagement with the broader societal context. That is really sort of where we’re aiming.

[00:54] If scholarship is expected to matter, then tenure and promotion systems have to align with those expectations, especially for junior faculty. Right now, we send a bit of a mixed signal. We say we value impact, but we reward a very narrow set of outputs. And so, this poses many challenges for junior faculty who might want to engage in practice or with policymakers but feel constrained.

[01:17] What’s needed is a tenure promotion system that takes a multidimensional view of impact and includes a broad portfolio of elements that can inform that view. So, influence on teaching, practice, policymakers, but sort of really extending out the way that we’re thinking about impact that’s coming from any one particular article or piece of research.

Citation counts are a signal of academic contribution, but they’re only one part of an article’s impact lifecycle.

[01:38] One of the persistent challenges we have with business research is that it’s not accessible to managers or policymakers. It sits in these journals. They’re not going to go and read them. How do we get those incredible insights out to them and help them use them? Academic associations have the opportunity to amplify the work published in their journals by using translation-style formats.

[02:00] So, thinking carefully about how we construct those translations, they need to be intentional. We need to make sure there’s language that really resonates with the managerial audience. Being mindful of what kinds of content we’re pushing out. Executive summaries that are accessible to managers, that are readable, and that they’re going to use on a regular basis, or other types of digital content that they see on a regular basis that they’re more comfortable consuming.

[02:25] At the Academy of Management, we have AOM Insights and AOM Today. The goal is to take that research from academic journals and translate it into actionable summaries that can be used by managers, practitioners, and policymakers. To better connect scholarship with practice and policy, we need to think more carefully or broadly.

[02:44] Partnerships, incentives, and other types of mechanisms that can incentivize faculty to engage with policy and practice, and also help them foster those connections and expand their work in different ways. And so, things we can do include opening pathways to partnerships with organizations.

[03:03] I’m in Silicon Valley, and that’s one of the things that we’re trying to do: to connect with a variety of different organizations out there who like someone to come in and help them solve a problem. This is particularly an issue in the context of AI. Can academics come in and help to provide that judgment piece and teach those critical skills, so that AI is an augmentation as opposed to just a substitute for critical thinking?

The goal is to take that research from academic journals and translate it into actionable summaries that can be used by managers.

[03:26] Shifting how we define and evaluate impact requires collaboration and coordination across the wide business ecosystem. Given the interdependencies between universities, academic associations, accrediting organizations, policymakers, businesses, and government, no one actor can go it alone. We really need to work together to move this engine forward.

[03:45] By bringing together a diverse set of actors to collaborate on a task force for impact, AACSB and AOM are helping to shape and develop a multidimensional view of impact. And this collective action is necessary because we all have different points of view, but bringing them together can synthesize how we can move impact forward and how we can advance influence with organizations and policymakers.

[04:07] One question is: can we develop a framework that applies to all types of institutions? Teaching institutions, those that are primarily focused on research, and those that are blending the two together. What we want is a framework with some agility at the edges, so the core remains the same. We’re all aiming toward the same core, but there are opportunities to adjust and flex depending on your institution’s mission and goals.

[04:32] If we all value research and agree on what impact matters, then providing this flexibility can help other schools figure out how to do it on their own paths. When we think about metrics, you know, we have the traditional forms. What are some alternative metrics? And one that I’ve recently learned about is Overton. Where can you connect your articles and identify how these are being used? Where are they being used? Where’s the influence heading? That’s one really excellent example, but I think there’s an opportunity for schools to think beyond that.

[05:04] At [Santa Clara University], we are a tech hub. We have the opportunity to work with a lot of different business organizations. They’ll come to us with a problem, or a scholar will seek data and reach out to them, and then they collectively solve that problem.

[05:18] Then it has an impact not only on the organization but also on the scholar. And that kind of intersection is one of the sort of really optimal places to be. So, can we create those pathways for more of that type of work and capitalize on it going forward?