Acronym

Katie:

I've always been of the mind that we should examine every interaction (particularly those with groups of members involved) as an opportunity to gather insight and understanding. We have to create systems to then manage the collection and interpretation of that info since it will be more qualitative in nature.

Example: people checking in for a chapter program. What if while giving them their nametag we ask them to take a 3-item survey, share with us their greatest challenge or success, etc.?

The accumulated info gathered from these small ongoing efforts can be a nice complement to the more comprehensive surveys conducted episodically and often reveal things the surveys don't.

Wow, Jeffrey is psychic! I was actually just about to post about a very similar idea--not my own, but one I heard from Peggy Hoffman during an interview for an upcoming article in Associations Now. She suggests asking three brief questions at the end of every e-mail communication with chapters--something along the lines of "Is this communication timely?" "Is it relevant to you?" "Is it actionable?" and asking readers to rate their response to each of those questions on a 1-5 scale.

I thought it was a great idea--especially given that chapters and national organizations sometimes run into issues when they have a mismatch of priorities. Short surveys like that could help you spot those mismatches before they cause real problems.

I've always been skeptical of member needs assessment surveys, and here's why. They are a snapshot in time - if ASAE asked me right now what I needed, I'd tell them what it was right now, but by the time they were able to analyze the data, I will likely be on to something else. Sure, they might be useful for trending, but...

Second, I think, on some level, isn't it our job to know what our members need before they know they need it? I wonder if we tend to rely on needs assessments as a CYA tool...pardon the crass terminology. "Well, that's what the members *said* they wanted..."

I think Jeffrey is on point with his post... every interaction is an opportunity to gather information and insight and understanding. I would add... how can we be sure to share that information with other staff internally in a way that is useful?

Here's some ideas... what about a regular town hall meeting with members - virtual or in-person? What about a blog to collect comments on the association's/chapter's work? What about asking questions on evaluations from current programs? For instance, we ask on every session evaluation - what more would you like to know about this topic? What other questions do you have?

A few thoughts...

I'll agree with Kristi - using social media tools such as blogs and social networks as a way to stay in contact with local members' needs is one way to go. They provide a way to monitor and stay in touch -- but only with the people who use them.

That said, I would offer two caveats. First, you need some good quantitative thinking about how to relate what you learn to the population as a whole. From a statistical standpoint, that means relating responses to known segments of the population. That's nothing new, but it does require some quantitative thinking, otherwise you tend to believe that what the "sqeaky wheels" say represents the membership as a whole.

The second caveat is, before you complain about the lack of representativeness of tools such as blogs, social networks, focus groups, and other "opt in" methods where the participant's selection probability is unknown, make sure you have a handle on the response rates you may already be getting from your mailed and emailed surveys. A poorly responded-to survey that lacks sufficient follow up of nonrespondents, even if based on formal probability sampling techniques, may end up being just as "unrepresentative" as other techniques.

As with life in general, "you get what you buy."

These are great ideas that I will definitely pass along!

These comments jogged my memory regarding promotions of association benefits - oftentimes we bombard members with communications to get our benefits out. If we always include a question regarding member satisfaction related to a product, we could increase awareness and potentially decrease communications.

Establishing a quantitative analysis for informal conversations, I imagine, could be challenging. A potential solution would be to give volunteers and staff the same question to ask members - this way, you could analysis trends.

Great question Katie largely because I think the typical answer is "nothing." There is a huge black hole in data when it comes to the chapter. As an AMC for chapters, I find that much is done "the way we've always done it" and that unfortunately is driven by three forces. (1) Not enough time/expertise to ask the question/collect and analyze basic data; (2) requirements from national organizations that were created too long ago (you must have 8 membership meetings each year) but what we must deliver to; and (3) national doesn't collect or use the local data and so local's don't.

At my level though we do collect data, look at it and act on it. So while we don't do a specific needs assessment, we do conduct ongoing assessment of what members are doing and thinking. Plus we mine the board meetings and gatherings for reoccurring themes. Right now we working with one of our national offices to create a report that will help us understand individual member's needs for certification education so we can create a relevant education calendar for 09.

I'd like to make a plug here for ASAE & The Center's next "decision to" research which will focus on the influence of the local connection on the decision to join because it will help uncover members' need and wants at the local level. Every co-sponsoring association (and you can still sign up and no I'm not on their payroll just passionate about the project) will get critical data about their members and their geographic components. This could be a starting point for really understanding the local needs and an incredible resource for determining the ROI of components. Call Monica Dignam or Clare Enzeo for details (or ask me!)

Bottom line, yes Katie I do agree we need to get the local story! Thanks for blogging about it!

Including chapter related questions in your national level needs assessment is one of the single best things you can do to boost participation and demonstrate real value through what can be an overly lengthy, formal process. Although I am called upon to provide research services, the marketer & association exec in me agrees with Kristi, that yes, you SHOULD know what they need before they do (and you should konw what they're experiencing as well). Of course if you're not careful and focus on only the loudest voices through your normal discovery/intelligence gathering process: otherwise you wind up hearing from the same group of people who may include a few early adopters but also others who are not that influential or representative of the membership.

Anytime we're engaged in survey research design I also try to emphasize the need for periodic & continuous 'data collection' allowing for qualitative (depth interviews, sometimes focus groups), discussion thread analysis, review/analysis of transactional data, and most importantly 'serializing' of your surveys so that they can be themed by topic and not so much time elapses between the conduct of periodic broad surveys of the membership. Successfully? Not so much. RFP'ed projects and Boards are often so attuned to the idea that they need to do a huge project on a regular schedule that it's hard to budge them from that plan.

If you include chapter level questions in your big survey, it's also important to crosstab & report to each chapter (with sufficient participation) their own pertinent results a la a 'report card' (and let them know up-front so they can promote participation. It's also critical to actually share with your overall membership not just choice bits of findings, but also a synopsis of what you are DOING with the results.

In most associations/chapters, members & respondents don't get frustrated if the survey/research process is actually seen as leading to something. You don't have much control at a chapter level if the volunteer pool & culture is still clubby or good ol' boy, but at least you can conduct research in a pragmatic way that helps you learn what you need and to help the chapters who will do something positive with the findings.

I like the spirit of Peggy's email idea (as described by Lisa) for evaluating communication effectiveness, but I'm not sure I would execute it the same way. Two thoughts:

1. We shouldn't need to have every email evaluated for its effectiveness if we are doing our job on the front-end with our communication strategy. A periodic spot-check might be more useful.
2. Asking the same questions on every communication will quickly grow tiresome and predictable on the recipient's part. The redundancy might cause people to just ignore the as they will seem too generic.

I should definitely point out that Peggy shouldn't be held responsible for my summary of what she said; when I spoke with her, she was throwing out a bunch of good ideas for improving national-chapter communication, and this was one of the ideas. We only discussed it very briefly, and I'm sure she would refine it (and describe it) better than I would!

That said, I think Jeffrey's points are good ones--familiarity can breed contempt (or at least a lower response rate). Spot checks on the most important communications vehicles, and more consistent checks on the most important communications, could work better than asking the same questions every time.

Katie,

I appreciate your hesitation to send multiple surveys to members. Over surveying is a sin - and one that can easily lead to reductions in response levels that are unhelpful.

As a former director of a professional services research unit of an online survey software purveyor, let me offer one solution which is technically doable with most commercial survey software packages. Introducing a question that asks members which chapter they belong to (not as an open text response, but a select only one list), allows you to use conditional logic to take the respondent to a sub-set of questions that apply to his/her chapter only. So, in addition to conducting a general membership survey, you can allow each chapter to send a limited series of questions that will be seen and answered ONLY by the local chapter's members, not the general membership.

It's a fairly straightforward solution technically. Of course, the information you get is only going to be as good as the survey you design, but this eliminates the problem of over surveying.

I think it's not so much that member needs surveys are the wrong tool, but that they're used for asking the wrong questions and often ask questions the wrong way.

A simple list of "which of these would you need" is virtually useless. It gets only at top-of-mind issues and breaks the "faster horse" rule. (Henry Ford said that if he had asked his customers what they needed, they'd have told him "a faster horse.") Aside from that, it's very rarely asked in a way that allows for real-world prioritizing based on resource allocation.

On the other hand, social media and focus groups have their own problems. They represent only the people in that group and there's the real possibility of more vocal or more influential members inadvertently (or deliberately) obscuring other needs. Unless you have somebody there who's good at and unafraid of questioning root assumptions, you can end up with the faster horse here, too.

Some alternatives:

Use surveys to ask the big-picture questions about overall need categories. Ask your members what they see as their biggest challenges now and biggest challenges three years from now. I rather like a scale that asks "Major challenge now-Minor challenge now-Minor challenge X years from now - Major challenge X years from now-Not a challenge now or likely to be one X years from now." (X is based on your timeframe for developing tools.) Then use focus groups to get to the root of what resources they will really need.

Use surveys to present scenarios and ask them which are most likely to represent their association in the timeframe you're considering. Then use focus groups to dig into those.

If you think you have a good sense of what they need, show them prototypes of the resource you'd develop. Then ask them specifically when and how they would use it. You can get a lot of insight this way.