Validating Your Creative Concept(s)

Execute a quantitative survey to identify a winning creative approach and ensure it is equitable in appeal.

During this phase, you’ll test creative concepts to identify a winning approach before moving into production. The test consists of a survey that is executed quantitatively (a type of test that produces statistically significant data on each concept’s performance) among your target audience.

Activities include the development of a research brief and survey questionnaire to inform the execution of the survey, set-up, and fielding of the survey through a research platform, and analysis of survey results to identify the winning concept and the drivers of performance.

The winning concept will be used as the basis for development of all campaign assets, including video, banners, out of home, email, print, and/or radio.

Why it matters

It’s important to measure how your audience reacts to the creative concepts that you’ve created. Don’t make assumptions about what message will resonate with your audience – ask them! And then use the data to drive your decision-making.

Your campaign’s performance will be as strong as possible because it will run with messaging that has been validated as the most effective approach with your target audience, and that the approach is equitable in its appeal.

Creative validation also saves time and money because it helps ensure you won’t invest in creative (or pay for media to run) that won’t move your audience to act.

How it works

A quantitative test of 3-4 different creative concepts is run by survey to identify a “winner.” The concepts the C+E Lab tests typically consist of a script for a 30 second video ad, plus a visual element consisting of a logo with a tagline. This approach focuses respondents on the ideas and the message, and prevents bias caused by liking or disliking of imagery. Here is a sample:

The concept that performs best against key attributes is identified as the winner. We recommend evaluating performance against likability, appeal, believability, and intent.

Performance is calculated by looking for the concept that generates the highest “top 2 box” scores across attributes. The “top 2 box” is calculated by combining the percentage of respondents who selected the top category, typically “Strongly Agree” or “Very Satisfied,” with the percentage of those who chose the second-highest category, such as “Agree” or “Satisfied.”

We also look at the detailed data to ensure winning performance is seen across audiences segments, including across genders, races, and ethnicities.

The survey should be fielded monadically, which is a research methodology where each respondent sees only one concept. Monadic testing is recommended because it is more realistic (audiences will only see one ad when it goes live, not a variety), it reduces bias that would be introduced by competitive concepts, it is more user-friendly for respondents, and ultimately provides more accurate results. A minimum of 150 completed responses per panel is recommended.

Questions should be asked before the test subject sees the creative concept, as well as after the test subject sees the creative concept. This helps identify changes elicited by the creative concept in the test subject’s intent, trust, perception in quality, and belief in self (i.e., I believe I will complete my college degree, I believe my business will be successful, I believe I will find a better job, etc.)

The survey is an opportunity to validate taglines, product benefits, and claims to identify the most effective options. Not sure if it’s better to say “help to put food on the table this summer” or “get $150 to buy groceries this summer”? Test it!

Once the results are in, look to see which concept performed best overall and then analyze attribute performance by demographics (gender, race, ethnicity, age, household income, etc.) to ensure it is activating the audiences you most want to activate in an equitable manner.

We typically use an audience research platform that provides off-the-shelf survey capabilities that are easy to use. Examples of these types of platforms include SurveyMonkey, Qualtrics, Alchemer, and more.

The concept that “wins” then moves into creative production.

How long it takes

2-4 weeks.

The process of validating creative concepts breaks down into three parts:

1. Survey set-up and quality assurance (QA)
2. Fielding
3. Analysis

Set up and QA can happen in as quickly as 1-2 days, and analysis should take no more than a week. It’s the time it can take for the survey to “field” and close that can be a wildcard.

A tightly defined survey audience criteria (i.e., geotargeting to a specific state with very narrow demographic requirements) narrows the pool of qualified respondents so the survey will have to run longer to generate the minimum number of completed responses. Running a survey on a national basis opens up the pool of potential respondents making it easier to run and close quickly - even within a few hours.

If your program serves residents of a specific state, it’s important to geo-target your survey to get accurate feedback. Just be aware it might take a week or two to generate your targeted number of responses and build extra time into your Project Plan.

It can also take more time to field surveys with audiences that are considered hard to reach like minors, undocumented individuals, or unhoused individuals. You may be able to successfully survey these audiences by allotting extra time for the fielding, or you may find alternative research methodologies (such as focus groups or one-on-one interviews) are a more effective solution.

Do it in a day

Simplify each script to a key idea, tagline, and logo. Use a boilerplate survey template from a self-service survey platform and run a “comparative” survey (each respondent sees all 3-4 concepts and compares them). Even with very narrow target audience criteria, it should be possible to reach 200 total completed responses within a day.

Outputs

• Research Brief Template
• Research Discussion Guide Template
• Audience Analysis (coming soon)