Crowdsourcing without the crowd knowing about it
Professor and author Ian Ayres delivered the Friday morning keynote at Digital Now, urging associations to adopt more statistical regression and randomized testing to better inform their decisions. If you're not, he says, "you're screwing up."
Why? Because, even if you think you're an expert, humans simply aren't good at predicting outcomes in situations that involve multiple driving factors.
In fact, Ayres picked the title of his book, Super Crunchers, via a randomized test through Google AdWords. Some searchers saw an ad for "Super Crunchers," and some saw an ad for "The End of Intuition," (Ayres' personal choice). More people clicked on "Super Crunchers," and so that title won.
I've seen other authors who have openly crowdsourced the titles of their books. Ayres did it blindly. The crowd didn't know it was being measured.
Of course, associations have their own built-in crowds, and Ayres says associations should more actively test anything and everything. "You routinely get a 5 percent to 10 percent lift in whatever numbers you care about when you do randomized testing," he says.
Just a few measurements associations can test:
- Member acquisition or renewal probabilities
- Member lifetime value
- Any and all marketing copy
- Website design
Ayres says the web has made statistical testing much more accessible because making adjustments is easy and cheap, as is gaining a large sample size.
My key takeaway from Ayres' presentation is that associations should trust numbers more (and get their boards to trust them, as well). We often overreact to complaints from members or feedback from evaluations. Broader statistical analysis of how members behave, rather than what a few of them tell you, can let you know whether the ones who are speaking up are representative of the full membership or merely outliers.
| | Permalink |
Comments
Interesting experiment with his book title, but I'm not quite sure it really is crowdsourcing .. unless the professor intents to fit most every act of marketing research
http://en.wikipedia.org/wiki/Crowdsourcing
and
http://en.wikipedia.org/wiki/File:Crowdsourcing_process2.jpg
Posted by: Jeffrey Cufaude | April 9, 2010 3:34 PM
Joe:
That's some good stuff there from Ayres. I so agree that as association professionals we often overreact to complaints from members or feedback from evaluations. We forget it may not be representative of the full membership. It's about balancing the squeaky wheel with the hard data that might say the same thing or something differently.
Thanks for sharing that.
Posted by: Jeff Hurt | April 9, 2010 3:40 PM
Thanks Jeffrey and Jeff.
I'd agree with you that statistical analysis isn't really crowdsourcing, but I wanted to use the term to allude to the idea of tapping into the wisdom of the crowd. (So I guess I should have written that more directly in the post.) You can tap into the wisdom of the crowd by asking people for their input, or you can tap into it by observing how people act. And as Jeff alludes to, interestingly, what people say and what they do may be different.
Posted by: Joe Rominiecki | April 9, 2010 4:34 PM
The thing that Ian Ayres kept (appropriately) hammering on during his session at DigitalNow is something that a lot of us conveniently try to ignore: As Ayres pointed out, any association that isn't doing statistical analysis is flying blind ... but to do any kind of meaningful super crunching, you need a sample size of at least one or two thousand.
Which to me exposes the mistake a lot of associations make when talking about the "wisdom of a crowd." How good is that wisdom if you don't really have a crowd? How many of the crowds we are pointing to and listening to number in the 100s, let alone the thousands? How many of the crowds we listen to are really just a very small and unique (certainly no broadly representative) group of maybe two dozen very unique, energized and engaged members sharing a tweet tag?
Is the crowd we listen to a valid expression of the shared wisdom of the masses, or only a very small, self-selecting group with a unique and valid, but NOT representative point of view?
Too many examples of association crowd sourcing are just as bad as the old bad practice of the Board assuming it knows what the membership is thinking and that the Board's biases and world view and experiences are shared by the membership at large. You are just replacing the Board with a different small, insular, unrepresentative and narrow mindset of people --- the ones who enjoy "playing" in the mobile com world --- and justifying the error by annointing it a wisdom-bearing crowd.
Posted by: Mark J. Golden, CAE | April 12, 2010 9:58 AM
Thanks Mark for the great points. I'm glad you brought up the point that Ayres mentioned about sample size. You're right, he did mention that several times, and it's something we should all remember.
I agree with you that when crowdsourcing is done with the goal of taking the pulse of an association's membership, we often fool ourselves into thinking we've done so even though we only get a "crowd" of 100 people or so. That's a dangerous trap to fall into, and I think that's why Ayres is so passionate about the power of larger statistical testing that can be done by tracking member behavior (not feedback), because it sidesteps the problem of self-selection bias.
However, in other cases, I still think crowdsourcing can be a powerful tool. If you're simply looking to gather more input, get more ideas, and allow more perspectives into any kind of project--more than you'd get with just your staff, just your board, or just your volunteer leaders--then crowdsourcing can be a good thing. It's when we let ourselves think that broader participation equates to full participation that we get into trouble.
Of course, this again brings up the trouble with the term crowdsourcing. It means a lot of different things. And I admit I abused it in this post here.
Posted by: Joe Rominiecki | April 14, 2010 11:45 AM
Mark: Thanks for adding your thinking. It helps me see the more expansive implications of what Ayres is suggesting.
I'm beginning to think crowdsourcing is a term that has lost the original meaning and specificity that was attached to it.
Posted by: Jeffrey Cufaude | April 14, 2010 12:37 PM