Comments on the e-cig model published by Levy et al: Results heavily depend on assuming very low risk

One of the things that make the discussion of e-cigarettes complicated is that no one knows precisely how the market will develop, how the advent of e-cigarettes will interact with cigarette use, and how dangerous e-cigarettes turn out to be.
 
Last year Sara Kalkhoran and I published a model that sought to address these issues by examining how different future scenarios would impact public health as a function of how dangerous e-cigarettes turn out to be.  The modeling approach we took was to model the steady state situation after the new market is fully developed and stable.  The broad conclusion that we drew was that, under the most likely future scenarios, the overall population effect of e-cigarette use would be positive if e-cigarettes are not very dangerous and negative if they are more than about 20-30% as dangerous as cigarettes.
 
David Levy and colleagues just published another model to assess the future that uses a different approach.  Rather than modeling a future stable market, they built a “cohort-based model that follows youth and young adults forward in time to assess health impacts over time.  (As a result, their model does not consider effects on e-cigarettes on long-established smokers.)  Another important difference is that they explicitly distinguish those who would have otherwise smoked from those who would not have otherwise smoked, and distinguish experimental from long-term use.   The overall conclusion that they draw is that “under the most plausible scenarios, [e-cigarette] use generally has a positive public health impact.  However, very high [e-cigarette] use rates could result in net harms.”
 
The primary reason that they reached the conclusion of net benefits is that their primary analysis is based on the evidence-free “expert” estimate that e-cigarettes are only 5% as dangerous as conventional cigarettes.  (Our model also shows net benefits for most scenarios at this very low level of risk.)
 
The important question is what would happen if the risks turn out to be higher than 5% as bad as cigarettes.  Levy and colleagues do alternative analysis with risks as high as 25% (the “high risk” scenario in their Table 1), a risk that they consider (based on no cited evidence) “unlikely”.  At the 25% risk, their model still shows net health benefits, although the results in their Table 1 show the benefits being very small.  This result suggests that the cross-over point into net harm in their model is probably around 30%, which is not all that different from what our model shows. 
 
In addition, because Levy’s model does not include effects on established smokers, one would expect the crossover point to be higher in his model than ours, but the similarity is still encouraging.
 
Another similarity in the results of our two analyses is what their respective sensitivity analyses show are the important parameters in the model.  We found that the dominant determinants are the ongoing health risks following smoking cessation, followed by the e-cigarette risk and the increase in interest in quitting among smokers. Levy and colleagues found that net population health effects from e-cigarettes are especially sensitive to e-cigarette risks and e-cigarette use rates among those likely to smoke cigarettes.
 
So, the message I take away from the Levy model is, at least tentatively, two different modeling approaches lead to similar conclusions, which is encouraging.
 
I say “tentatively,” because the estimate that Levy’s model would cross-over to net negative effects around a 30% risk is a guess that I made based on his Table 1, not a direct calculation from the model.  I emailed Levy and asked him if he would run the higher risks, and he responded, “I'd be glad to re-run the model with the higher levels of risk, but I would not publish or publicize the results at the higher levels of risk, because I do not consider them credible given the current evidence.”
 
This is unfortunate, given that one of the benefits of modeling is that it allows you to explore a broad range of possible futures.  (In particular, Sara Kalkhoran and I included e-cigarette risks all the way down to zero, even though we did not think e-cigarettes were harmless.)  The emerging evidence that e-cigarettes are having similar effects on cardiovascular and non-pulmonary lung disease as smoking cigarettes is why we modeled effects up to at least 50% as bad as cigarettes. 
 
In addition to the fact that we modeled steady state for all users (youth and adults) whereas Levy et al modeled a cohort of youth and young adults going forward in time, another important difference between the two modeling approaches is that, while we structured our model to make (almost) all the transition probabilities directly observable, so that the parameters in the model were based directly on data, Levy et al did not.  As a result, all the probabilities in their model (Figure 1 of their paper and the tables in the online appendix) are their estimates based on considering the evidence in the literature.  (Note that there is not a single direct citation to evidence associated with any of the probabilities in their model.)
 
The reason for this problem is that the first node in their model (in Figure 1 in their paper) is whether a never smoker would transition to “Would have become a smoker in the absence of [e-cigarettes]” or “Would not have become a smoker in the absence of [e-cigarettes].”   As they note near the end of their Discussion section, “The initial branch in Figure 1 leads to hypothetical states which cannot be observed and are inferred from past smoking patterns.”  While theoretically reasonable in a model, this means that the key initial node in the model is unobservable, which means that the probabilities in all the subsequent nodes, which are conditional on the pathway defined by the first node, are unobservable (apologies for technical statistical terms).  That is why Levy and colleagues could not cite specific data for the precise numbers used in the model, which heavily affect the results.  The lack of a direct linkage between the numbers in the model and actual observed data can mask biases and errors in the model.
 
This situation contrasts with the approach that Sara Kalkhoran and I took, which was to base the probabilities in the model as much as possible on directly observable data.  (See Table 1 in our paper and note that there are citations to the data that define almost all the parameters in the model.)  Indeed, it was this desire that led us to do a steady state model with scenarios rather than the time-based approach that Levy and colleagues used; we did not think that there were yet data to support all the additional assumptions that are implicit in including time as a variable.
 
This is more than a small technical detail.  Best practices for developing these kinds of models suggest basing the transition probabilities based on directly observable data.  (See Reporting guidelines for modelling studies by Bennet and Manuel.) 
 
Despite these problems, the tentative similarities in the results of the two models is encouraging in terms of the cross-over risk and sensitivity analysis.  I encourage Levy to publish the results for higher risks for e-cigarettes to advance the discussion.  It would also be better if future versions of their model was based on observable probabilities.  They also need to investigate a wider range of risks.
 
In the meantime, it is important to keep in mind that the Levy paper's overall conclusion the e-cigarettes are likely to provide pubilc health benefit is heavily dependent on the assumption that e-cigarettes have very low risks, an assumption that is likely wrong.
 
Note:  There are two other papers advancing models for future behavior (Cobb et al and Cherng et al), but they do not integrate health risks of e-cigarettes, a key variable according to both our and Levy et al’s sensitivity analyses.