- FAMRI Center
Shiffman et al paper in Nicotine & Tobacco Research is not a reliable estimate of effects of ecig flavors
On January 7, 2015, Saul Shiffman and colleagues published "The impact of flavor descriptors on nonsmoking teens’ and adult smokers’ interest in electronic cigarettes" in the journal Nicotine and Tobacco Research. This paper, reported work funded by the NJOY e-cigarette company, concluded that "The e-cigarette flavors tested appealed more to adult smokers than to nonsmoking teens, but interest in flavors was low for both groups."
This is an important conclusion for the e-cigarette industry, which has been challenged by evidence that flavors attract kids to e-cigarettes.
Robert Jackler from Stanford has asked me to post this on his behalf about the Shiffman et al paper.
A Comment on the Context of My Critique:
Earlier this year, Stan Glantz submitted a critique of the Shiffman et al paper on flavor preferences in eCigs to the journal which published the original work (Nicotine & Tobacco Research). As it is customary practice for journals to encourage responsible criticism to appear in their pages, I was more than a little surprised to learn that they refused to publish his comments. After Stan’s critique later appeared on his blog, I decided to also submit my own comment to NTR. As I was at the time a consultant to the California Attorney General’s office on negotiations related to the paper’s sponsoring company (NJOY) flavored eCig practices I needed to delay my submission until this matter had resolved.
My commentary (see below) emphasizes both my concerns over the scientific validity of the paper and what I view as compelling evidence of commercial bias. I find it troubling that NTR also rejected my commentary as well. Refusing to publish both Stan and my critiques raises questions about the journal’s commitment to thoughtful dialog on issues of scientific validity and bias. In rejecting my critique, NTR explained that conflict of interest among the authors had been disclosed. However, merely disclosing COI does not excuse any and all sorts of commercial bias contained in the design of the research and/or interpretation of the data. As disclosure is meant to help readers to consider the possibility of bias, it seems most appropriate for the journal to publish letters from responsible academics who perceive commercial interest and seek to share their rationale with the scientific community.
In rejecting my comment, NTR went on to say: “Your letter I think implies that the article has significant flaws that mean it should not have been published in the first place, but it has been published and the journal's view is that the final accepted version dealt adequately with the original reviewer's comments.” By definition, all published papers have passed the journal’s expert peer review process and all subsequent letters to the Editor follow this approval. Having long served as an Editor-in-Chief of a major scientific journal, I am keenly aware of the limitations of the peer review process and the importance of responsible post publication commentary to enhance the value of scientific literature.
My Commentary on Shiffman et. al. study:
The paper ‘Impact of flavor descriptors on non-smoking teens and adult smokers’ interest in electronic cigarettes” by Shiffman and coworkers suffers from flaws in experimental design and data interpretation which call into question their conclusions that flavored e-cigarettes are not of differential appeal to teens.1 As discussed in detail below, their survey methodology has a high likelihood of selection bias, the study groups are too dissimilar to enable the comparisons made, informed consent for minors was not obtained despite evident risks, and commercial interest appears to have influenced the study’s outcome.
In comparing the relative appeal of flavored e-cigarettes to non-smoking teens versus smoking adults, the obvious hypothesis would be that flavors would be of greater attraction to the young. The factual basis for this conjecture is the extensive body of research which found that flavored tobacco products have differential appeal to youth. This led the US Congress to ban flavored cigarettes in 2009. In a broader context, a finding that adolescents have no preference for sweet and fruity flavors would mean that e-cigarettes somehow are a special exception to well established consumption trends throughout the food and beverage industry.
Concerns over survey validity:
The logical way to determine the differences in e-cigarette flavor preferences between adults and teens would be to compare the actual frequency of flavor use by teen and adult e-cigarette users. Rather than use a direct method, Shiffman et. al. compared a small cohort of teen non-smokers with adult smokers via a marketing survey conducted for the sponsoring e-cigarette brand (NJOY). There is reason to question the validity of comparing non-smoking teens with adult smokers as they are notably unequal groups. Nationally only approximately 15% of teens smoke whereas the entire adult survey group smoked. In this survey teen smokers were systematically excluded.
The Shiffman et. al. study consisted of an online survey inquiring about interest in products of the NJOY e-cigarette brand. These included then current flavors (tobacco and menthol) and a list of contemplated flavors. A survey of flavor preferences for bottled water and ice cream was used as a comparison. Invitations were sent to 20,235 adults and 14,151 teens with a mere 432 adults and 216 teens enrolled. This is an under 2% response rate. With such a low response rate, responders may have had a special motivation to participate. This raises the likelihood of selection bias, differences between responders and non-responders, a factor which undermines the value of surveys with a small percentage of responders.2
A sizable fraction of teens are opposed to smoking, some with notable vehemence instilled by parents and schools. It would be expected that a fraction of teen responders felt a special impetus to do so as a means of communicating their opposition to tobacco products. Two sorts of teens are likely to reply to such a survey: those contemplating e-cigarette use who may be attracted to the products being surveyed and those who are vehement in their intention to neither smoke nor vape and are motivated to send a rejectionist message. The motivation of teen responders to oppose smoking is the most probable reason for the low interest in e-cigarettes which the authors erroneously interpreted as a lack of interest in flavored e-cigarettes. That the paper reaches the conclusion: “Nonsmoking teens’ interest in e-cigarettes was very low” is yet another reason to question the survey’s validity. This observation is at variance with 2014 CDC data showing a rapid rise in e-cigarette use among high school students coupled with a reduction in combustible cigarette use.
Not surprisingly, their survey showed non-smoking teens queried on common consumer products (ice cream and flavored water) indicated widespread interest in flavors. In the adult cohort, flavor preferences for e-cigarettes, other than tobacco and menthol, roughly paralleled that of ice cream and water. (Shiffman et al. Figure 3b) That teens expressed a liking for flavored ice cream, but not e-cigarettes, may well be accounted for by responses of teens who had no interest in e-cigarette products. (Shiffman et. al. Figure 3a) It was not that these non-smoking teens did not like the e-cigarette flavors, but rather they categorically were uninterested in NJOY’s products in all flavor variations.
In their flavor analysis, the paper’s data reveals a marked preference among adult smokers for tobacco flavor and menthol flavors with all other flavors (except vanilla) appearing at a fraction of their appeal. (Shiffman et al. Figure 1) Among teen non-smokers, by contrast, interest was essentially equal across flavors with no preference for tobacco or menthol flavors. One interpretation of these data is that, relatively speaking, non-smoking teens prefer a variety of sweet and fruity flavors while adult smokers prefer tobacco flavor. This interpretation of their data is the opposite the authors’ conclusions.
Inappropriate Claim of Exemption from IRB for Survey of Minors
The Shiffman et. al. survey offers no benefit to the minors surveyed and entails significant risks. Exposing teens to a harmful product they may not have considered using, such as use of e-cigarettes and flavored e-cigarettes, could increase the likelihood that they will try the product and develop nicotine addiction. This represents a clear risk to the future health of the minor. The existence of risk, coupled with the lack of benefit to the individual child, necessitates both institutional review and parental informed consent.3
Evidence of Commercial Bias
The authors’ declaration in the Shiffman et. al. paper discloses that the study was sponsored by the NJOY e-cigarette brand. It also reveals that the company was involved in determining the study design. The survey was conducted by a ResearchNow a company which conducts surveys for commercial marketing purposes on subjects such as customer satisfaction and loyalty. All of the study authors were all employees of the Pinney Associates a firm who represents both the tobacco industry (Reynolds American) and e-cigarettes (NJOY - Soterra Inc). Pinney Associates describe their service to companies: “Evaluate, support and develop new commercial opportunities including new products and claims; Manage issues and the scientific and policy environments to gain regulatory approval for new products; and “Enhance the scientific and policy environments to maximize product acceptance.”4
Given the long history of tobacco industry manipulation of scientific research, the fact that the Shiffman et. al. study was designed and sponsored by an e-cigarette brand gives reason for skepticism. That the authors we all employed by a company contracted to NJOY whose goal is to “Enhance the scientific and policy environments to maximize product acceptance” further raises concern over the independence and objectivity of the paper. The circumstances under which the paper arose, as illustrated by the timeline below, indicates that NJOY had a powerful motivation to commission a paper exculpating e-cigarette flavors from preferential youth appeal proximate to the time that Pinney Associates was engaged and the survey was conducted.
In December 2009, NJOY CEO declared that they would halt flavor sales to discourage underage use and to align with the Family Smoking Prevention and Tobacco Control Act recently passed by Congress.5 Between 2012 and 2013 NJOY lost its lead position in market share to Blu (2013 Blu 44.1%, NJOY 24.3%).6 By the second quarter of 2014, Logic overtook NJOY as the number two brand in all tracked channels.7 Lorillard’s purchase of Blu and the 2014 emergence of RJ Reynolds (Vuse) and Altria (Mark Ten) into the e-cigarette market further increased NJOY competition. The New York Times quoted NJOY CEO Craig Weiss as having said in 2013 that other manufacturers use flavors to attract children.8 This New York Times article of July 2014 described NJOY as having chosen to resume flavor sales, after a 5 year hiatus, due to a 50% decline in convenience store sales over the preceding year. In the month following re-introduction of flavors in July 2014, NJOY realized a 65% increase in convenience store sales.9
NJOY leadership was undoubtedly concerned about possible repercussions from their reintroduction of flavors. After all, over a four year period two of their CEOs had publicly linked e-cigarette flavors with youth. Complicating matters further was the 2010 they a consent judgment they agreed to in the nation’s largest state (California) stipulating that they would not market flavored e-cigarettes.10 In an effort to manage anticipated regulatory and public relations difficulties, NJOY engaged Pinney Associates in January 2014. As part of this campaign, it appears that Pinney Associates was commissioned to create a scientific publication with the expectation that it would conclude that flavored e-cigarettes possess no special attractiveness to youth. The work product of this engagement was the scientifically unconvincing Shiffman et. al. study in which the authors seem to have gone to great length to conjure a methodology which would appear to support their sponsors’ desired outcome.
Shortly after the July 2014 re-introduction of flavored NJOY products, the California Attorney General contacted the company to request compliance with the 2010 judgment. In September 2015, the Attorney General’s office agreed to allow flavored NJOY products to remain on the market in California in exchange for certain limitations on youth marketing.11
In summary, it appears that the Shiffman et. al. paper was not hypothesis driven research, but rather a study commissioned by NJOY to create cover for their reversal on flavors driven by dropping market share. Both the circumstances under which it came about, and the scientific content of the work, suggest that the result was not an open ended question but rather attained a result preordained by the sponsor. Fundamentally it was “science” commissioned to support a product launch and to bolster defense against regulatory action.
The paper concludes that adult interest in flavors as modest, with the exception of classic tobacco flavor, which they observed can “ease the transition for smokers from a familiar product to a less-familiar one.” With teens uninterested and adults minimally interested in flavors (other than tobacco and menthol), one wonders why do they bother to market them. If so unappealing to their customers, what accounts for the dramatic rise in NJOY sales after introduction of flavors such as butter crunch, peach tea, and wild berry?
Robert K. Jackler MD
Sewall Professor & Chair
Otolaryngology-Head & Neck Surgery
Stanford Research into the Impact of Tobacco Advertising (SRITA)
Stanford University School of Medicine
Dr. Jackler served as a consultant to the State of California Attorney General’s office on the 2014 motion to enforce the 2010 Soterra (NJOY) consent judgment.
1. Shiffman S, Sembower M, Pillitteri J, Gerlach K, Gitchell J. Impact of electronic cigarette flavor descriptors on non-smoking teens’ and adult smokers’ interest in electronic cigarettes. Nicotine Tob Res. 2015 Jan 7. pii: ntu333.
2. Groves RM, Peytcheva E. The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis The Public Opinion Quarterly Vol. 72, No. 2 (Summer, 2008), pp. 167-189
3. Heath and Human Services. Institutional Review Board Exemptions: Special Classes of Subjects: Children and Minors: (http://www.hhs.gov/ohrp/archive/irb/irb_chapter6.htm)
4. Pinney Associates: About our Company. (http://www.pinneyassociates.com/overview.xml)
5. NJOY to Discontinue Flavors, Takes Additional Steps to Prevent Underage Electronic Cigarette Use. Reuters. Dec 10, 2009
6. Giovenco DP, Hammond D. Corey CG, et. al. E-Cigarette Market Trends in Traditional U.S. Retail Channels, 2012–2013. Nicotine & Tobacco Research, 2015, 1–5
7. Wells Fargo Securities. Equity Research: Vapor - NJOY’s New Product Line Positions it Ahead of the Pack. Wells Fargo Securities; 2014.
8. Richtel M. E-Cigarette Makers Are in an Arms Race for Exotic Vapor Flavor. New York Times, July 15, 2014.
9. NJOY tops US retail Vaping sales. Convenience Store news Nov 11, 2014. http://www.csnews.com/product-categories/tobacco/iri-njoy-tops-us-retail-vaping-sales
10. People of the State of California v. Sottera, Inc. d.b.a. NJOY. Stipulated consent judgment. Case No. RG10528622. July 29, 2010. (http://oag.ca.gov/system/files/attachments/press_releases/n1965_sottera_...)
11. People of the State of California v. Sottera, Inc. d.b.a. NJOY. Stipulation to modify consent judgment of July 30, 2010. August 18, 2015. Case No. RG10528522