Megan Daniels, a vice president with Phoenix Marketing International, shared four common problems with normative databases at the annual spring NEMRA conference last week. Norms are typically used to compare the results of a survey to: whether for a concept test or an ad test or for employee or customer satisfaction research. Normative databases are expensive to build. As a result, quality can decline over time.

Here are the four common problems Megan identified:

  1. Misinformed – Norms that "do not account for shifting trends and response drivers" may leave you misinformed. As categories develop and evolve, the drivers of consideration shift in importance. For instance, Phoenix has seen the drivers in beer purchase consideration change with a resulting impact on which TV commercials are likely to be successful. "Past champions or long-standing norms may be built on outdated category drivers and could misinform your advertising by not accounting for the current environment," Megan said. Make sure your norms are up-to-date and account for changing category trends.
  2. Misassembled – Norms that provide "an incomplete view of the true competitive landscape" have been misassembled. If you are testing a consumer packaged good, does it look like the aisle where it resides or the entire store? Norms that include all CPG brands have benchmarks that are vastly different from one another. You want to compare to products "in the context of consumers' wallets and the true competitive set." Megan shared a Bank of America ad, "Summer Movie." It scored above average when compared to all financial services, but was only average when compared to credit cards, and it was below average when compared to other cash-back credit cards. Getting the competitive set wrong can lead to the wrong launch decision. Know who the competition is and use them for comparison.
  3. Misaligned – Norms that aren't "tested among the brand’s target consumers" are misaligned. Benchmarks often tout that they use nationally representative sample that matches the U.S. Census. But that might not be your market. For instance, the Esurance ad "Belt Clip Enthusiast" does poorly among those 40 years old and up, but it resonates with its target audience: Millennials. Align your targets to the appropriate market not the overall market to establish the right context.
  4. Misapplied – Norms that are "not reliable for ads in emerging channels" are often misapplied. Sometimes norms really are comparing apples to oranges. The "Luna Corona" YouTube ad [case study] does great when benchmarked against norms of television ads for beer. But it's just average when compared to YouTube videos. Make sure to develop an agile program that readily reacts to changing media.

So that your norms "don't mis- the mark," create normative databases that are broad, deep and up-to-date.