Research vs. intuition
Based on Andy's excellent comment to my post on my likes and dislikes of 7 Measures, I wanted to clarify that I am not antiresearch. Far from it actually.
Andy makes the point that the ideal scenario is a blending of research and intuition to make decisions. I think that's what I was trying to say, though I think perhaps not well enough. I think the relationship between the two is that research—things like membership surveys, reader surveys, and benchmarking surveys—should absolutely be performed and does provide valuable information, which, in turn, helps people form intuition. Many, many other things help form intuition, too. Some of the important things that come to mind are having conversations with members and other constituents (and then using those conversations to start more conversations with your staff or volunteers to see if they're hearing the same things); looking at the trends in the world around you, staying informed not just of the local, national, and international news (and please, go beyond U.S.-based sources for your international news), but of trends in marketing, technology, diversity and generational issues, economics, and yes (one of my personal weaknesses), even pop culture; and noticing how other organizations are changing.
Having said all that, I don't back down from my assertion that research should not directly inform decision making. Rather, research should be one factor considered in making decisions.
Andy also brings up the notion that research is now faster, more efficient, and less expensive than it used to be, all of which is true—kind of. The other side to that coin is that it is harder than ever to perform classic research that has statistical meaning. The scholarship on research is still coming to terms with the tools that are faster, more efficient, and less expensive. Do the tools introduce a bias into the results based on who is willing to use them? And if so, what is that bias and can you account for it? One small example that most people are familiar with is that phone surveying had been considered a pretty strong method of research assuming the rest of the research design was good. Now, with more and more people giving up landlines and going completely mobile and the prevalence of caller ID systems, there are starting to be questions about what biases are being introduced. These are things I geekily admit to being interested in (at the expense of pop culture). I don't know how it will evolve, but for right now, I treat fast & cheap research with about the same deference as focus groups. It can be a good source to help inform opinions, but as a statistical measure, I'd be skeptical.
| | Permalink |
Comments
I'm going to push back and argue that a blend of research and intuition is not what organizations need to support decision-making. Nine times out of ten, the best organizational decisions will be made based on a healthy dialogue involving diverse points of view, informed by data and other knowledge resources, as well as provocative questions that challenge assumptions, and taken by decision-makers who carefully weigh a variety of factors before reaching a conclusion that is consistent with certain forward-looking, valid and deeply-held principles about what will drive success. If association CEOs and boards make decisions in this way, they will probably do a very good job in stewarding their organizations through uncertainty.
Research is a critical element of the data component of decision-making, but I don't think we should treat it as the Holy Grail. The 7 Measures of Success proves this point. On the one hand, it provides interesting and useful insights into specific factors that ostensibly drive association success. On the other hand, the nature of the matched-pair methodology is such that we get only a retrospective look at those factors, i.e., they created success in the past. Is it a reasonable assumption that they may continue to contribute to success in the future? Possibly, but then there may be other factors not revealed by the research and demanded by the changing environment that associations will need to consider as well. Research informs, but does not control.
My point is simply that decision-making is a context-dependent activity and it is how the decision-makers understand the depth, breadth and richness of that context that will and should always control. It is simply a question of applying good judgment to the combination of what you believe, what you know and what you are able to learn. If you are missing any of these elements, effective decision-making will falter.
Posted by: Jeff De Cagna | September 28, 2006 10:05 AM
Given my comment sparked another post, I should probably respond. In short, I agree with quite a bit of what both Scott and Jeff say here. More specifically, my comments are:
I whole heartedly agree with Scott's statement, "Rather, research should be one factor considered in making decisions." I've said the same thing several times to colleagues. In my comment, I may have been too strong of a proponent for research, but that was in reaction to thinking it was being dismissed a bit too quickly. Generally, Scott, it sounds like we're pretty much on the same page.
Scott's comments about the statistical validity of the newer tools/methods is a good one. Indeed, there may be bias introduced here. But it comes down to how well you know your membership and how comfortable you are in trusting responses to an Internet-based survey, for example.
As for Jeff's opener, "Nine times out of ten, the best organizational decisions will be made based...If association CEOs and boards make decisions in this way, they will probably do a very good job in stewarding their organizations through uncertainty."
Wow, that would be great if CEOs and boards did that. Not sure how often it occurs in reality, and that's not in any way to dismiss the scenario. I wish it occured more frequently.
Here again, I don't think research should be treated as the Holy Grail. As Scott said, and I agree with, it should be but one factor.
And I really like Jeff's last statement, "It is simply a question of applying good judgment to the combination of what you believe, what you know and what you are able to learn. If you are missing any of these elements, effective decision-making will falter."
Posted by: Andy McDaniel | September 28, 2006 11:26 AM
Jeff De Cagna wrote, "Nine times out of ten, the best organizational decisions will be made based on a healthy dialogue involving diverse points of view, informed by data and other knowledge resources, as well as provocative questions that challenge assumptions, and taken by decision-makers who carefully weigh a variety of factors before reaching a conclusion that is consistent with certain forward-looking, valid and deeply-held principles about what will drive success."
Is that based on data Jeff ... or your intuition? Sorry, couldn't resist.
Posted by: Jeffrey Cufaude | September 28, 2006 10:10 PM
Jeffrey, thanks for the comment and question.
The answer to the question, of course, is yes. ;>)
Posted by: Jeff De Cagna | September 29, 2006 2:24 PM