Acronym

I'm baffled that more associations don't recruit first-time attendees to be secret shoppers and feedback focus group participants. It immediately moves them from outsiders to insiders, and gets them to provide valuable insights for the organization.

In my past work with retailers, I have seen them use secret shopping, some of the smarter ones also use ethnographic research with anthropologists to drill down into the how and why of things people do subconsciously or at least well beyond their ability to articulate. (What people say and do can be two different things--they're not lying but they omit details and actual behavior often deviates from opinions and doesn't follow the rational norms we often apply to our memories of how we behave.)

Secret shopping is a wonderful way to measure real quality of service delivery but I have noticed the few times we've offered it or used it, it generates a defensive reaction from staff--'oh, the reason your shopper never got someone on the phone is because we're really busy' or 'the reason they sent the wrong thing is because most people just register online anyway.' In contrast, when one includes a few basic questions regarding customer/member service quality, staff responsiveness and articulation rates a 4.8 on a 5-point scale! Sadly these findings tend to reflect the inherent bias toward respondents who like the organization, and members who 'grade on a curve.'

It's often more revealing to test your direct, key points of inbound contact: send test messages to the webmaster, call the main switchboard at various times with legitimate needs. These are the times that someone trying our the association, or who legitimately need that one extra bit of information to make a purchase-type commitment are forming their lasting impressions. Even what would be a great service-failure rate (10% would be quite good based on random observation!) is costing you more than you can measure in memberships, registrations, etc.

Of course the secret shopper can also be an actual member, as was suggested in your IOM course. One of my favorite associations used first-time attendees to critique their show and attendee experiences through focus groups and post-show interviews. We recruited about 40 people total from a variety of disciplines and although much of what we learned was fine-tuning an already well-oiled machine, I think it's a great technique for enlisting fresh eyes; when they chime in, their direct words are far more compelling than what staff or consultants can summarize. The example they cite--of people clustering together and not welcoming newcomers--is one great reminder that in face-to-face settings, even the best and most professional associations with strong senses of community can really still be clubs at heart.

Kevin

This could be a great tool for struggling components .. they are often too close to see the problems but a member from across the country could shed some light. Many assns offer chapter mentoring - maybe chapter secret shoppers could be added!

Jeffrey, Kevin, and Peggy, thank you all so much for your ideas! I really like the idea of recruiting first time attendees as secret shoppers; it's a great way to get fresh eyes and make them feel like part of the action. And Peggy, I think the idea of a national association offering secret shoppers to chapters is a great one--perhaps chapters could even trade secret shoppers with one another?

Peggy's point is a good one. I liken it to the 'report card' approach we try to take with large chapter networks--including a brief assessment of the features/qualities of the chapter(s) that one belongs to. The numbers tend to stimulate competitive attitudes; but the secret shopping would help the experience come alive for the chapter. Perhaps not central HQ: there is nothing like a peer checking you out, while the central office so often can come off sounding clinical or overbearing, like a "Big Brother." Often better to hear and act on non-binding suggestions than react to grades from an authority!