ASTM E1958-22

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E1958 − 22
Standard Guide for
Sensory Claim Substantiation
This standard is issued under the fixed designation E1958; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
INTRODUCTION
Formats or standards for testing related to sensory claim substantiation cannot be considered
without a frame of reference of where that format or standard would fit within the legal framework
that surrounds the topic. Product sensory claims tests are performed for three basic reasons: (1)
Comparison of Products—Determines how one product compares to another, usually a competitor or
earlier version of itself. (2) Substantiation of Claims—Enables marketing personnel to use positive
references through advertising or packaging, or both, in the presentation of the product to the
consumer. (3) Test Performance—Ascertains and establishes the tested product performance within
the scope of its intended use.
Theriskassociatedwitheachclaimisassessedwhenconsideringclaimssubstantiation.Compelling
and aggressive claims are sure to be scrutinized closely by competitive firms, and if inconsistencies
are found through competitive test data, the claims could be challenged in one or more of the
following venues: (1) National Advertising Division (NAD) of the Advertising Self-Regulatory
Council (ASRC), (2) one or more media, such as print, broadcast, or electronic media, (3) Consumer
Advocacy Organizations, and (4) Civil or Federal courts. No single test design or standard test will
prevent challenges. The criteria used by each of the potential forums are not identical and are
constantly evolving. With the introduction of new technologies coupled with changing consumer
demands, testing processes and protocols that were sufficient five or ten years ago may not hold up
under today’s criteria and scrutiny. Testing requirements of the future can only be a matter for
speculation.Theoneconstantisthat,asadvocatesoftheirclients’positions,attorneyswilldefendtheir
clients’ testing processes and protocol while questioning with great detail every aspect of their
competitor’sprotocolintheattempttoswaythearbitertoagreethattheirclientsareintheright.Legal
counsel should be part of any team developing claim substantiation.
This guide demonstrates what a group of professionals who are skilled in the science of testing
considerappropriatefromascientificandtechnicalstandpoint,andrepresentsaneffectivemethodfor
both defendant and challenger to determine the viability of a sensory claim. The key word is
“appropriate.” If a particular aspect of a test, or method, is not appropriate for a specific application,
it should not be used. Care should be taken to clearly define the reasons and data supporting a
deviation from the standard, as any departure invites scrutiny. Since departures are inevitable, the
word“should”isusedinthisguidetoindicatewhenothertechniquesmayhaveapplicationsincertain
unusual circumstances. Whenever a test protocol has been completed, it should be critiqued for
weaknesses, including whether experts in the relevant field would consider the research objectively
designed, conducted, and analyzed, using procedures that give accurate and reliable results. If
weaknesses are found, corrective action should be taken, since the competition may point out any
weakness or discrepancy and challenge the study.
While the scientific and technical community identifies the appropriateness of a research method
used to support a sensory claim, the legal community evaluates substantiation for legal claims using
“reasonableness”asthecriterion.Withtheimportanceofhavingalegal“reasonablebasis”foraclaim,
the question remains, “What is reasonable?” Unfortunately, there is no specific answer to that legal
question, as it will depend on the type of claim, product application and use, applicable regulations
where the product is sold, and other factors. These considerations, market pressures (such as timing),
and testing budgets can influence and impact the protocols to support a specific claim. This guide
provides principles and considerations that need to be addressed for good sensory and consumer
testing practices.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E1958 − 22
1. Scope
Test Facility 13
Statistical Analysis 14
1.1 Thisguidecoversreasonablepracticesfordesigningand
Paired-Preference Studies 14.1
implementingsensoryteststhatvalidateclaimspertainingonly Superiority Claims 14.2
Equivalence Claims 14.3
to the sensory or perceptual attributes, or both, of a product.
Unsurpassed Claims 14.4
ThisguidewasdevelopedforuseintheUnitedStatesandmust
Paired Comparison/Difference Studies 14.5
Analysis of Data from Scales 14.6
be adapted to the laws and regulations for advertisement claim
Keywords 15
substantiation for any other country. A claim is a statement
Commonly Asked Questions About ASTM and Claim Ap-
about a product that highlights its advantages, sensory or
Substantiation pen-
dix
perceptual attributes, or product changes or differences com-
X1
pared to other products in order to enhance its marketability.
1.2 This international standard was developed in accor-
Attribute, performance, and hedonic claims, both comparative
dance with internationally recognized principles on standard-
and non-comparative, are covered. This guide includes broad
ization established in the Decision on Principles for the
principles covering selecting and recruiting representative
Development of International Standards, Guides and Recom-
consumersamples,selectingandpreparingproducts,construct-
mendations issued by the World Trade Organization Technical
ing product rating forms, test execution, and statistical han-
Barriers to Trade (TBT) Committee.
dlingofdata.Theobjectiveofthisguideistodisseminategood
sensory and consumer testing practices. Validation of claims
2. Referenced Documents
should be made more defendable if the essence of this guide is
followed.
2.1 ASTM Standards:
Table of Contents
E253Terminology Relating to Sensory Evaluation of Mate-
Introduction
rials and Products
Scope 1
E1885Test Method for Sensory Analysis—Triangle Test
Referenced Documents 2
Terminology 3 E2164Test Method for Directional Difference Test
Basis of Claim Classification 4
2.2 ASTM Publications:
Consumer Based Affective Testing 5
Sampling 5.1
MNL 13Descriptive Analysis Testing for Sensory Evalua-
Sampling Techniques 5.2
tion
Selection of Products 5.3
MNL 26Sensory Testing Methods: Second Edition
Sampling of Products When Both Products Are Currently on 5.4
the Market
STP913 Physical Requirement Guidelines for Sensory
Handling of Products When Both Products Are Currently on 5.5
Evaluation Laboratories
the Market
Sampling of Products Not Yet on the Market 5.6
Sample Preparation/Test Protocol 5.7
3. Terminology
Test Design—Consumer Testing 6
Data Collection Strategies 6.6 3.1 Definitions:
Interviewing Techniques 6.7
3.1.1 Terms used in this guide are in accordance with
Type of Questions 6.8
Terminology E253. Additional terms are listed below.
Questionnaire Design 6.9
Instruction to Respondents 6.10 3.2 Definitions of Terms Specific to This Standard:
Instructions to Interviewers 6.11
3.2.1 α (alpha) risk, n—the probability of concluding that a
General/Overall Questions 6.12
sensory claim is supported when, in reality, the sensory claim
Positioning of the Key Product Rating Questions 6.13
Total Test Context and Presentation Matters 6.14
is false.
Specific Attribute Questions 6.15
3.2.1.1 Discussion—α risk also is known as Type I Error or
Classification or Demographic Questions 6.16
significance level.
Preference Questions 6.17
Test Location 7
3.2.2 β (beta) risk, n—the probability of concluding that a
Test Execution by Way of Test Agencies—Food and Non-food 8
sensory claim is not supported when, in reality, the sensory
Testing
Documents to Retain in Sensory Claims Substantiation Research 9
claim is true.
Laboratory Testing Methods 10
3.2.2.1 Discussion—β risk also is known as Type II Error.
Types of Tests 10.2
Advantages and Limitations of the Use of Trained Descriptive 10.3
3.2.3 central location test (CLT), n—atestwherestimuliare
Panels in Claims Support Research
evaluated by consumers at a common designated location(s);
Test Design—Laboratory Testing 11
the stimuli preparation and presentation are usually controlled.
Product Procurement 11.6
Experimental Design 11.7
3.2.3.1 Discussion—Examples of designated locations in-
Data Collection 11.8
cludemarketresearchfacilities,academiclaboratories,grocery
Data Analysis 11.9
stores, or hotel conference rooms.
Questionnaire Construction 12
1 2
This guide is under the jurisdiction of ASTM Committee E18 on Sensory For referenced ASTM standards, visit the ASTM website, www.astm.org, or
Evaluation and is the direct responsibility of Subcommittee E18.05 on Sensory contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
Applications--General. Standards volume information, refer to the standard’s Document Summary page on
Current edition approved Nov. 15, 2022. Published December 2022. Originally the ASTM website.
approved in 1998. Last previous edition approved in 2021 as E1958–21. DOI: Available fromASTM International Headquarters, 100 Barr Harbor Drive, PO
10.1520/E1958-22. Box C700, West Conshohocken, PA 19428-2959.
E1958 − 22
3.2.4 comparative claim, n—designed to compare similari- criticalinachievingthebestresults.Allinvolvedpartiesshould
ties and differences between two or more products. The basis meet and agree (perhaps several times) prior to implementing
for comparison can be within the same brand, between two the substantiation research.
brands, or between a brand and other products in the category.
4.2 Familiarity with the general classification of advertising
3.2.5 context effect, n—effect upon the perception of a
claims is important in developing clear statements of claims at
stimulus arising from its interrelationship with other stimuli,
an early stage and for developing a rational plan for testing.
such as in a presentation set or in the experiment.
This familiarity also facilitates the process of selecting appro-
priate testing methods, among the many types of methods
3.2.6 directional difference test, n—a paired comparison or
available to the consumer/sensory science professional. Each
2-AFC(AlternativeForcedChoice)methodinwhichassessors
methodanswersspecificquestionsandmaysupportonetypeof
selectthestimulusfromapairofstimulithatisperceivedtobe
claimbutnotanother.Therefore,theconsumer/sensoryscience
higher or lower in intensity of a specified sensory attribute.
function provides an important source of information and
3.2.7 equivalence claim, n—in an equivalence claim, two
experienceinclaimsubstantiationandwillprovidemuchofthe
productsareclaimedtobeequivalentinoneormoreparticular
definition of testing methodology. There are multiple ways to
feature.
support claims depending on the characteristics of the claim.
3.2.8 home use test (HUT), n—a test where stimuli are
Two approaches are consumer based and trained panel based
evaluated by consumers at home or in the environment typical
evaluations.
of the actual use situation; the stimuli preparation and presen-
4.3 Advertisingclaimscanbedividedintotwofundamental
tation are usually self-administered.
classifications: comparative and non-comparative.The distinc-
3.2.8.1 Discussion—Home use test and in-home use test are
tionbetweenthetwoclassificationsiswhetheracomparisonis
used interchangeably.
made relative to an existing product (advertiser’s or competi-
3.2.9 monadic or single product tests, n—product tests
tor’s) or to itself.
where only one product is experienced and rated.
4.4 Comparative Claims—Comparativeclaimsaredesigned
3.2.10 power (statistical), n—the probability that a sensory
to compare similarities and differences between two or more
claim will be supported, when, in reality, the sensory claim is
products. The basis for comparison can be within the same
true.
brand, between two brands, or between a brand and other
3.2.10.1 Discussion—Power=1– β, where β is the Type II
products in the category.
error rate.
4.4.1 Comparativeclaimsgenerallytakeoneofthreeforms:
3.2.11 self-administered questionnaire, n—questionnaires
superiority, equivalence, or unsurpassed. Superiority,
independently completed by the respondent are referred to as
equivalence, or unsurpassed claims are further sub-classified
self-administered.
into two central areas of application: hedonic and attribute/
3.2.12 superiority claim, n—a superiority claim asserts a
perception. Hedonics broadly concern measuring the degree of
higher level of performance or liking or preference relative to
liking and preference; either liking overall or liking that is
another brand; a superiority claim can be opposed to a
limited to one or more specific attributes.Attribute/perception
competitive brand (for example, “cleans better than brand Z”)
claims apply to intensity when measuring one or more specific
or opposed to an earlier formula of the brand (for example,
product attributes.
“nowmorecleaningpowerthanbefore”);asuperiorityclaimis
4.4.2 Equivalence Claims—Equivalence claims are claims
supported if a statistically significant proportion of the respon-
that assert equivalent levels of performance or liking when
dents prefer the advertiser’s product.
comparing a particular product to another product. In general,
3.2.13 unsurpassed claim, n—an unsurpassed claim indi-
equivalence claims are made relative to a market/category
cates that the product(s) selected for comparison is not better/ leader.Twoproductsarenotclaimedtobeexactlyequalinone
higher(orgreaterthan)insomewaytothetargetproduct(s)for
or more particular features; rather, the products are claimed to
which the analysis is executed. be sufficiently similar as to be equivalent from a practical
perspective.
4. Basis of Claim Classification
4.4.2.1 Hedonic:
(1)“Tastes as good as brand X.”
4.1 Afundamentalstepinadvertisingclaimsubstantiationis
4.4.2.2 Attribute/Perception:
creating an explicit statement of the claim prior to actual
(1)“Our product reduces odors as much as brand X.”
testing. The statement is then forwarded to all parties con-
(2)“Our product lasts as long as brand X.”
cerned in the substantiation process. Concerned parties could
(3)“Our cake is as moist as the leading brand.”
includemarketing,marketingresearch,legal,consumertesting,
4.4.2.3 Overall Equivalence:
sensory evaluation, research suppliers, etc. The statement is
(1)“We’re just the same, except for the price.”
essentialasitcanencouragecollaborationintermsofcorporate
(2)“You’ll never know the difference between us and
resources, confirms the selection of appropriate test methods,
brand X.”
and has the potential to maximize the chance of making
reliable business decisions about the proposed claim, pending 4.4.3 Unsurpassed Claims—In unsurpassed claims, the
the results of substantiation research. Collaboration among all claim stated indicates that the product(s) selected for compari-
involved parties prior to executing substantiation research is son is not better/higher (or greater than) in some way to the
E1958 − 22
target product(s) for which the analysis is executed. Examples 4.5.2.2 If one hypothesis is related to an attribute evaluated
of unsurpassed claims include the following types: by a trained sensory panel and another hypothesis is related to
anattributeevaluatedbyaconsumerpanelandnoclaimwillbe
4.4.3.1 Hedonic:
made unless all of the test results support the claim, then a
(1)“No other product is better than our product.”
multiplicity adjustment is not required. This testing procedure
(2)“No other product is more liked for butter flavor.”
must be determined and documented prior to data collection.
4.4.3.2 Attribute/Perception:
4.5.3 An advertiser can use a test result to evaluate hypoth-
(1)“No other cake is more moist than ours.”
eses sequentially without adjustment for having conducted
(2)“No other product has more butter flavor than ours.”
multiple tests if the hypotheses are tested using the same
(3)“No other product reduces odors more than our prod-
approach and differ only in their critical values. Specifically, if
uct.”
one claim is supported, then a subsequent claim can be
(4)“No other product lasts longer than our product.”
evaluated without multiplicity adjustment if all possible test
(5)“No other product is thicker than our product.”
results that support the subsequent claim also support the first
(6)“No other product cleans faster than our product.”
claim. If using the statistical methods in Section 14, then the
4.4.4 SuperiorityClaims—Superiorityclaimsassertahigher
following statements hold:
level of performance or liking relative to another brand.
4.5.3.1 Every superiority claim is also an unsurpassed
Superiority claims can be opposed to competitive brands (for
claim. An advertiser can evaluate an unsurpassed claim as a
example,“cleansbetterthanbrandZ”)oropposedtoanearlier
primary hypothesis and a superiority claim as a secondary
formula of the brand (for example, “now more cleaning power
hypothesis. No claim can be made if the unsurpassed claim is
than before”). Examples of superiority claims include:
not supported. If the unsurpassed claim is supported, then the
4.4.4.1 Hedonic:
unsurpassed claim can be made, regardless of whether the
(1)“Our product tastes better than brand X.”
superiority claim is supported. If the superiority claim is
(2)“Our product tastes better than any other.”
supported, then the superiority claim can be made. The
(3)“Our product is preferred over any other brand.”
unsurpassedtest(14.4)andsuperioritytest(14.2)usethesame
4.4.4.2 Attribute/Perception:
approach and differ only in their critical values. As every test
(1)“Our cake is more moist than any other.”
result that supports a superiority claim also supports an
(2)“Reduces odors more than brand X.”
unsurpassed claim, it is valid to evaluate both tests using the
(3)“Lasts longer than any other product.”
Type I error rate (level α) without adjusting for multiplicity.
(4)“Thicker than brand X.”
4.5.3.2 Every equivalence claim supported by the test in
(5)“Cleans faster than any other product.”
14.3isalsoanunsurpassedclaimthatwillalsobesupportedby
4.4.4.3 In superiority claims, combinations of hedonic
the test in 14.4. These tests use the same critical value for the
claims and attribute/perception claims can sometimes be
lowerbound.Theequivalencetestfrom14.3usesanadditional
found,whensuperiorityclaimsareestablishedbasedonoverall
critical value for the upper bound. An advertiser can evaluate
liking and for specific attributes (for example, “Our hosiery is
an unsurpassed claim as a primary hypothesis and an equiva-
preferred over Brand X for overall liking and it offers more
lence claim as a secondary hypothesis. No claim can be made
support and comfort.”).
if the unsurpassed claim is not supported. If the unsurpassed
claim is supported, then the unsurpassed claim can be made,
4.5 Statistical Considerations for Comparative Claim
regardlessofwhethertheequivalenceclaimissupported.Ifthe
Types:
equivalenceclaimissupported,thentheunsurpassedclaimcan
4.5.1 Fromastatisticalperspective,itmayrequireasmaller
be made. As every test result that supports an equivalence
consumersampletosupportclaimsofunsurpassedthanaclaim
claimalsosupportsanunsurpassedclaim,itisvalidtoevaluate
of superiority or equivalence.
both the unsurpassed and equivalence claims using the Type I
4.5.2 If hypotheses related to multiple attributes are tested
error rate (level α) without adjusting for multiplicity.
independently using the Type I error rate (level α) with the
4.5.3.3 If using statistical methods other than those recom-
intention of making the claims that are supported, then it
mendedinSection14,thenthesestatementsdonotnecessarily
elevates the risk of making one or more false claims above the
hold, so additional justification is required.
nominal Type I error rate. To control such risk, an appropriate
4.5.4 Methods for hypothesis testing can be found in the
statistical procedure must be used to adjust for multiplicity.
sections related to the analysis plan (5.2.10) and to statistical
Typically, each hypothesis needs to be tested using an adjusted
methods (Section 14).
alpha level that is less than the Type I error rate. This testing
procedure must be determined and documented prior to data
4.6 Non-comparative/Communications Claims—The objec-
collection. Consult a statistician if required.
tive of the non-comparative/communications claim is to con-
4.5.2.1 If hypotheses are related to multiple attributes and vey something specific about the product, usually a product
no claim will be made unless all of the test results support the benefit or difference, and in general, does not seek to provide
claim, then a multiplicity adjustment is not required. This comparativeclaimsrelativetootherproducts.Forexample,the
testing procedure must be determined and documented prior to statement “provides long-lasting flavor” or “smells strong for
data collection. A longer questionnaire is often needed to test one month” tells us something about the product, but not in a
multiple claims. For considerations related to test design, see comparative sense relative to an existing product. These types
Section 6 (especially 6.13). of claims are common in new product types, but also are used
E1958 − 22
to bring attention to specific product benefits. Examples of respondents who like and eat strawberry yogurt, not just any
non-comparative/communications claims include the follow- yogurt with fruit. Conversely, respondents typically do not
ing types. need to be restricted to use of only that product within the
4.6.1 Hedonic: category, they may also use alternative products. For the same
(1)“Tastes great.”
example, a claim about strawberry yogurt does not mandate
(2)“Makes your laundry outdoor-fresh.” that they be exclusive to eating strawberry yogurt. In some
(3)“Leaves a long-lasting freshness you will like.”
cases,thescreeningcriterianeedstobebroadened,forexample
4.6.2 Attribute/Performance: a criterion for inclusion may be that all the participants have
(1)“Removes odors for 60 days.”
the same need, even though that need may be met by products
(2)“Leaves glass streak-free.” that are designed very differently or for other purposes (for
(3)“Leaves no residue on surfaces.”
example, widely different products may be used for make-up
(4)“Works fast.”
removal).
NOTE 1—In the above attribute examples, some of these could be
5.1.1.2 If a claim is aimed at drawing future category users
approached either as a non-comparative claim, since no other product is
in addition to present users, future intent to use should be
mentioned, or as a comparative claim versus an appropriate standard
includedasascreeningcriterion.Ifrecentcategoryusageisnot
(streak-free glass, residue-free surface, odor-free room).
applicable (more than the past three to six months), such as
4.7 Selecting an Appropriate Ad Claims Test—Product
with seasonal products or products with long purchase-repeat
claims made in print or on radio, TV, or the Internet require
cycles, or when a product is innovative and part of a newly
valid data that supports the intended claim. As with most
created category, identifying study respondents based on posi-
sensory testing, it is necessary to first identify the project and
tive future category usage intent (for example, concept or
test objectives for the study. The claim statement should
prototype acceptors) may be necessary. More detail than the
indicate whether the claim is based on consumer or laboratory
simple product statement should only be used when it is
sensorymethodsor,infact,someinstrumentalorchemicaltest.
necessarytoclarifywhattheproductisbecauseofitspotential
Sensory claims for preference or liking (“preferred over the
tobiasrespondentsinadvanceoftheclaimssubstantiationtest.
leading brand” or “better than the competition”) require
Whentheproductisnewtothemarketwithoutanycomparable
consumer tests with the preference or liking questions to
product, a formal concept may be used to screen respondents
support the claim. Claims about product attribute(s) or perfor-
for the claims substantiation testing.
mance can be based on data from consumers, who are asked
5.1.2 Hedonic or affective (such as liking, acceptance,
about the specific attribute, or from laboratory sensory tests
preference, or purchase intent) claims should always apply to
designed to measure the specific attribute(s). In some cases,
thetargetuserpopulation.Samplingfromanypopulationother
both types of testing (consumer and laboratory) can be used
than the users to whom the claim is focused, such as
together to support the same claim. The ad claims team needs
purchasers,mayrequireaqualifiedclaimtolimititsgenerality.
to determine the type of claim, the claim statement, the target
The test protocol should state clearly whether a claim is being
population, and the aspect(s) of the product that is the focus of
made for one or more of the following:
the claim. Only then can the test to support the claim generate
data with the right focus and weight to support the claim.
Population Examples
Product recommenders “doctors prefer.”
Purchasers Pet food by pet owners
5. Consumer Testing: Selection of Study Respondents
A
“Choosy moms choose Jif” is a
and Product
claim specific to the purchaser, the
user is the child
5.1 Study Respondent Sampling:
The ultimate consumer of a product “now more refreshing”
A
5.1.1 Representativesamplingofrelevantstudyrespondents
Jif is a registered trademark of the J.M. Smucker Company.
is a key element of a defensible claim study.The type of claim
5.1.3 Perception claims are based on product attribute(s) or
and the target audience of the advertising are two determining
performance, or both, that are detectable in the target user
factors for the appropriate respondent sample, within practical
population. Claims that state or imply detectability (for
considerations of the category and claim. Respondents should
example, ‘more chocolate flavor’), include study participants
be typical users and not be restricted to any specific subset,
from the target user population. Note, trained and validated
suchasheavyorlightusers,unlessitisaqualifiedclaimabout
panel data can be used, if data related to the claim from the
that specific subset. It is also important to distinguish between
target population already exists, and it can be demonstrated
purchasers or users of the product to ensure sampled respon-
thatthetrainedpanel’smeasurementscanbeusedtopredictthe
dents align with the claim.
target population (see 10.3.2 and 10.3.3).
5.1.1.1 Screening based on recent category usage is recom-
mended to identify target respondents. A simple product 5.1.4 For category usage claims, respondents may be re-
statement should be used to describe the product to be tested cruited by screening for brand usage, but care should be taken
withjustenoughinformationsothatitisclearwhattheproduct during screening to ensure respondents cannot determine
is. The product statement should not contain puffery, benefits, whichbrandsaretargetedfortesting.Thiscanbeaccomplished
aesthetic description, or ingredients, unless these are unusual by providing a large list of brands with the brand or brands of
enoughtorequireanexplanation.Sometimesrespondentsneed interest embedded in the questionnaire. Brand usage and
to be screened to a more narrowly defined criterion. For frequency of use data also can be collected to help match the
example, a claim about strawberry yogurt should recruit incidence of these variables in the respondent sample and
E1958 − 22
target population. Target users can be defined by their re- product testing is rarely practical because it would consider all
sponses to several questions, including: consumersinthepopulationofinterestaspotentialparticipants
in the test. In reality, many of these consumers cannot be
5.1.4.1 “What one brand of this product type do you use
most often?” located or are unwilling to participate.
5.1.4.2 “What brands have you used in the last (insert time
5.2.4.1 Screeningcriteriashouldnotberevealedtopotential
period appropriate for the category)?” respondents during recruitment, and the standard security
5.1.4.3 “Whatbrandhaveyouusedmostofteninthelast‘x’ screening questions (for example, whether family members
months?” work in advertising or marketing or other related fields,
5.1.4.4 If the frequency of use is one of the criteria for including that of the test product) should be included.
recruitment, then the respondent also may be asked how often
5.2.5 Sources for Selecting Respondents—Potential respon-
they use the product or how many times they have purchased
dents may be obtained from companies that provide testing
the product within a specific time frame. More questionnaire
services or sell marketing information. These databases are
discussion can be found in 6.9.
built using a variety of techniques, without targeting a particu-
5.1.5 Potential allergies or sensitivities to product ingredi-
lar manufacturer’s consumers. In some cases, a company may
ents should be assessed based on the product being tested and maintain its own branded database of product users. If using
nature of exposure. Participants in product studies need to be
these databases, it is important to consider potential biases, for
screened by self-report for potential allergens, sensitizers, or example, the repeated use of the same individuals, or the
lifestyle factors, or combinations thereof, that may exclude
influence of the known sponsor of the research.
them from the test (for example, tree nuts, wheat, or special
5.2.6 Respondents should be screened specifically for the
diets for religious or ethical reasons for food and beverage
claims substantiation test to ensure they currently meet the
products, nickel, and some perfume components for non-food
inclusion and exclusion criteria in the study design. To mini-
products).
mize over-testing with the same respondents, it is common to
screen for past study participation, excluding those who have
5.2 Study Respondent Sampling Techniques:
tested a specific product in the past three months, or tested in
5.2.1 The type of claim should be kept in mind when
the product category in the past six months. Other approaches
determining the number of respondents. For example, an
to minimize sampling bias may include conducting testing
equivalence claim may require more respondents than a
across a range of days of the week and times of the day, and
superiority claim (see Section 14). Furthermore, the number of
varyingthelocationswherepotentialrespondentsarerecruited.
respondents required to represent the range of relevant respon-
5.2.7 Geographic Area Recruiting—Some claims will re-
dent characteristics may be larger than the minimum that is
quire geographic considerations as a function of the target
required to achieve desired statistical power.
population and the nature of the claim. The study design and
5.2.2 Thecharacteristicsoftherespondentsampleshouldbe
rationale may need to include variables that reflect the geog-
consistent with those of the target population (i.e, about whom
raphy of the claim. Variation in product use may be due to
the claim is being made). The characteristics may include the
geographic variability in demographics. Product performance
population in terms of age, gender, product usage, culture, and
may be impacted by environmental factors such as
language. For example, a constrained demographic sample
temperature, humidity, water hardness, pollution, or hours of
should be employed when it is consistent with the stated claim
sunshine. For example, use of certain sauces and spices may
and normal product usage (for example, a particular gender or
vary geographically, laundry detergent performance and pref-
age).
erences may be impacted by water hardness, hair product
5.2.3 Using quotas on relevant respondent characteristics is
performance may be impacted by humidity, and suntan lotion
important to achieve a match between sampled respondents
use may vary by geographic region.
and the intended target population. Relevant respondent infor-
5.2.7.1 Some preference claims and usage claims may have
mation should be collected to demonstrate the appropriateness
a potential for geographic and demographic dependencies. For
of the sample of the target population. Respondents chosen
example, such a dependency exists if consumer preference or
should reflect the target population. For example, representa-
usage varies by region or by socioeconomic factors, such as
tion of age and gender should reflect the target population.
urbanversussuburbanversusrural.Theevidencefororagainst
5.2.4 Recruiting of the respondent sample must be stated in
such dependencies could come from patterns in product sales,
the test protocol and should be as objective as possible.
or usage, or both. These considerations need to be included in
Develop and document a rationale for both the number of
the respondent sampling rationale when they apply.
respondents and for the inclusion and exclusion criteria. If the
number of respondents is less or more than determined by 5.2.7.2 When geographic region is assumed to be a factor
statistical criteria alone, this should be included in the ratio- relevant to a claim, the geography of respondents should be
nale. Consumer product testing respondents are usually ob- consistent with the scope of the claim. A rationale should be
tained from stratified (not convenience) sampling procedures. developed when testing country-wide or regionally, ensuring a
Quotasareoftenusedbecausetheytendtoproduceasampleof fair representative sample of the defined target population in
respondents that is fit for the purpose of testing with the target multiple markets dispersed across the geography stated or
user population of interest. When there are multiple testing implied in the claim. Various approaches for selection of test
locations (see 5.2.7.2), then these quotas are applied within locationsarepossible;theyneedtobeincludedintherationale.
each of the locations. Probability sampling for consumer Some examples of how test sites could be selected follow.
E1958 − 22
(1) Recruiting Example Based on Geography: National (4) Recruiting Example Based on Brand Development
Claim—An advertiser intends to make a national claim for a Index in Limited Geographies—An advertiser intends to make
product in the United States.Aconsumer test is conducted via a regional claim for a hand cream product that is designed to
protect and soothe hands exposed to cold temperatures. Sales
Central Location Testing (6.6.1) in four major regions (West,
are concentrated in regions in northern U.S. states, mostly in
East, Central, and South) with two cities per region, with a
northern Minnesota, upstate New York and northern Maine.
sufficiently large number of consumers included from each of
Since there are three regions where the Category Development
the cities. In general, it is best to recruit a large enough sample
Index (CDI) is highest, a test is conducted via consumer CLT
of consumers in each city to have the range of relevant
in three cities in these regions, one city per region. The cities
characteristics represented in the consumer sample and to
are selected so that taken together each brand’s Brand Devel-
achieve the desired statistical power (see Section 14). If the
opment Index (BDI) is approximately proportional to that
consumer sampling requirement is met by having 40 consum-
brand’s BDI for the regions in which the claim will be made.
ers per city, this would yield a total sample of 320 consumers
The test includes a sufficiently large number of consumers
(40 consumers × 8 cities). If the consumer sampling require-
from each city to represent target users of this category of
ment is met by having 60 consumers per city, this would yield
products.
a total sample of 480 consumers (60 consumers × 8 cities).
Consumer sampling depends on both the range of relevant 5.2.8 Recruiting Example Based on Market Share—If mar-
characteristicsneededtobeincludedintheconsumersampleto ketshareisacriterion,becautiouswhenselectingmarketsand
ensurethatthetestadequatelyrepresentsthepeopleresidingin
achieve representativeness and the risk of challenge.
the geographic territory on which the claim is based. In
(2) Recruiting Example Based on Geography: National
categories with strong geographic differences in market share,
Claim—An advertiser intends to make a national claim for a
the total market share should be approximated by representing
product in the United States.Aconsumer home use test (HUT;
high, low, and average share markets in the study.The number
6.6.2) is conducted using zip codes to ensure that a sufficiently
of respondents per region may vary, reflecting their contribu-
large number of consumers are included in multiple markets
tion in terms of population density, but not heaviness of usage.
across geographies. In general, it is best to recruit a large
enough sample of consumers to have the range of relevant 5.2.9 Recruiting Examples Based on Product Usage—
characteristics represented in the consumer sample and to Respondent selection may be based on product usage or
achieveadequatestatisticalpower(seeSection14).Toachieve product category usage rather than geographical consider-
thetargetednumberofconsumers,itwilllikelybenecessaryto ations. Examples follow.
recruit some additional consumers. Consumer sampling de-
5.2.9.1 Sensory testing is needed to support a claim that a
pendsonboththerangeofrelevantcharacteristicsneededtobe
product delivers relief from itchy eyes, the primary need for
used in the recruiting/screening of the consumer sample to
selecting a representative sample of respondents is to have
achieve representativeness and the risk of challenge. Relevant usersofthecategoryindependentofgeography.Alargesample
characteristicsrefertothecriteriausedinselectingrespondents of users of products in this category can serve as a represen-
toserveinasensory/consumerproducttest.Forexample,these tative consumer sample in this case.
criteria can pertain to factors such as demographic
5.2.9.2 A product claim pertains to hair coloring products.
characteristics, product usage (“users of automatic dish
Only users of product in the hair coloring category need to be
detergents,” “users of bottled barbeque sauce”), purchasing
included in the respondent sample.
habits or potential to purchase a product or product category,
5.2.10 Analysis Plan—Claims substantiation is confirma-
flavor likers, attitudes, or behaviors relevant to the need for a
torytesting,notexploratorytesting.Onceatargetpopulationis
product(forexample,frequentgymgoersdesiringgymapparel
defined in the study protocol and represented adequately by
that reduces odors; parents with young children who need
sampling,resultsfromthetotalrespondentsampleareusedfor
healthy snacks in portable packaging).
substantiating a claim. Results among some subgroup may not
(3) Recruiting Example Based on Geography: Regional
correspond to overall results because the number of respon-
Claim—An advertiser intends to make a regional claim in the
dents in subgroups are smaller, and therefore may not be as
western United States for a product that is distributed only in
statistically reliable. Moreover, since there is a risk of false
the western United States. Three cities in western U.S. states
positivesandfalsenegativesintestinganyhypothesis,analysis
wheretheadvertiserandcompetitorproductshavecomparable
of multiple subgroups will increase the overall error rate.
market share are selected. The consumers sampled are users
Therefore, the analysis plan reflects the intended claim(s) and
who are representative of consumers with respect to relevant
any subgroup testing should be determined before testing
factors(i.e.,demographic,socio-economicandproductusage).
begins (for example, ensure an adequate number of respon-
AtestisconductedviaCLTwithasufficientlylargenumberof
dents for any analysis and minimize the number of hypotheses
consumers from each of these three cities. For example, a
to test). Any subgroup analyses that were not planned and
comparative taste claim pertaining to a mayonnaise product properly designed may not be a sound analytical practice for
that is distributed only west of the Mississippi River would claims substantiation (see Section 14). If the number of
require mayonnaise consumers to be sampled from cities in respondents who complete the test is lower than what was
states west of the Mississippi River. planned, researchers need to provide a rationale for the
E1958 − 22
shortage or a rationale for why the achieved number of samples match that of local test participants. This way, if
respondentsissufficienttoprovideadequatedatatosupportthe national products manufactured in more than one site have
claim. been formulated differently to appeal to regional differences in
5.2.11 Be prepared to defend the documented rationale for sensorypreferences,appropriateproductswillbetestedagainst
respondent sample criteria if challenged. relevant regional competitors. It is critical that all information
regarding product sourcing be documented.
5.3 Selection of Products:
5.4.4 Competitive products should be purchased in the
5.3.1 If a test is being conducted to support a competitive
standard size package with the highest unit volume or in
claim that is not brand-specific (for example, versus “other
similar size, or both, to the test product. Trial size and
leading brands”), then the competitive brands should be the
club-store oversized product packages should not be used
two brands with the highest national market share. If the
unless the package meets the specific target of the claim.
market is highly fractionated, such that the top two national
5.4.5 Every effort should be made to obtain competitive
brands control less than 50% of the market, then more
products of representative freshness found in the marketplace.
competitors must be included in the test. Either the three
All products in the test should be of typical age. A freshly-
leading national brands or any brand that is among the top two
made product should not be compared against a product
in the four major geographic regions of the country must be
nearing its expiration date.
tested. Unless the product is tested against brands representing
at least 85% of the national market, it is recommended that 5.5 Handling of Products When Both Products are Cur-
claimsshouldbemadeagainstspecificbrandsinlieuofgeneral rently on the Market:
superlative claims. Eighty-five percent (85%) of the market is
5.5.1 After procurement but prior to testing, handling,
defined as all products within said category, including the length of storage, and storage conditions of all products must
brand making the claim.
be identical and consistent with normal consumer practice.
5.3.2 Competitive brands should be in the same market 5.5.2 Competitive samples must not show any signs of
segment as the brand for which the claim is being made. If a
mishandling or abuse. If products become non-homogeneous
brandstraddlesmarketsegments,thenproductsmostsimilarin during handling, in that they cannot be returned to their
a reasonable competitive context should be used.
original state (precipitates may be returned to solution, but
5.3.3 When competing products are sold in more than one fractured pieces cannot be made whole), then test samples
form, the products being tested must be of the same form or in should be remedied for such defects. For example, the last
theformmostrelevanttotheclaim.Ifapowdereddrinkmixis serving or two from a box of cereal that may have a
beingcomparedwithacompetitor’sproductthatalsocomesin disproportionateshareoffinesshouldbediscardedorscreened.
apowdereddrinkmixandasareconstitutedliquid,bothbrands 5.5.3 To minimize the likelihood of product recognition by
would have to be tested in their reconstituted f
...