How Likely Is Probably?

When you are asking research participants to estimate how likely something is to happen,  there are several pitfalls to watch out for.  One of them is the lack of consensus about the meaning of words.

From Psychology of Intelligence Analysis published by the CIA:

“The table [above] shows the results of an experiment with 23 NATO military officers accustomed to reading intelligence reports. They were given a number of sentences such as: “It is highly unlikely that...”

All the sentences were the same except that the verbal expressions of probability changed. The officers were asked what percentage probability they would attribute to each statement if they read it in an intelligence report. Each dot in the table represents one officer’s probability assignment. While there was broad consensus about the meaning of “better than even,” there was a wide disparity in interpretation of other probability expressions.

 

plot1.png

Inspired by Sherman's experiment, a Reddit user u/zonination polled fellow redditors with a similar questionnaire, asking them to assign numerical probabilities to different words, and mapped the results.

When you ask someone how likely they are to purchase your product and they answer "unlikely", it could mean a less-than-5% chance, or a greater-than-35% chance. For someone else, "probably" could mean less than even odds.

If you are asking about purchase intent in a survey, be careful how you present answer options, since the probability implied in their order could conflict with what people usually understand each word to mean.  If "probably", on average, means a higher probability than "likely", it could be confusing for respondents to see the options in this order:

  • Definitely

  • Likely

  • Probably

  • About even

  • ...

 

Sherman Kent, the original experiment's author, talks about what sparked his interest in how intelligence professionals communicate different probabilities:

“A few days after the estimate [of Soviet actions in Yugoslavia] appeared, I was in informal conversation with the Policy Planning Staff's chairman. We spoke of Yugoslavia and the estimate. Suddenly he said, "By the way, what did you people mean by the expression `serious possibility'? What kind of odds did you have in mind?" I told him that my personal estimate was on the dark side, namely, that the odds were around 65 to 35 in favor of an attack. He was somewhat jolted by this; he and his colleagues had read "serious possibility" to mean odds very considerably lower.”