I really enjoyed the 2006 Science paper by Dijksterhuis, Bos, Nordgren and van Baaren on deliberation without attention.  Then came Acker (2008) with a meta-review and the suggestion that there is “little evidence” of an advantage of deliberation without attention.

Today, from the latest issue of Judgment and Decision Making: Are complex decisions better left to the unconscious? Further failed replications of the deliberation-without-attention effect by Dustin P. Calvillo and Alan Penaloza.

The summary seems to be that deciding without deliberation, immediately after the stimulus is presented, might sometimes be better than deliberation.  But not with distraction post stimulus.

The moderators of the effect—there seems to be something going on in a few studies!—are still not well understood.

Playing around with rggobi at the moment, the R package which interfaces with the interactive graphics package GGobi.  Installation very easy; just needed the command:

source("http://www.ggobi.org/downloads/install.r")

Followed by a restart of R.

To get the first session going:

require(rggobi)
g = ggobi(mtcars)

What’s the point of Bayes’ rule?  This web page by Eliezer S. Yudkowsky gives a long intuitive explanation (thanks to Keith Frankish for pointing to it).  This blog post is an attempt at a slightly shorter version with a bit more maths, and a bit of rambling about interpretation.

The information in the example problem given there is as follows:

1. 1% of women at age forty who participate in routine screening have breast cancer.
2. 80% of women with breast cancer will get positive mammographies.
3. 9.6% of women without breast cancer will also get positive mammographies.

The task: A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

The general problem solved by Bayes’ rule is that if you know the probability of if A, then B, how do you work out the probability of if B, then A?  More precisely if you know P(B|A), what is P(A|B)?

Here B|A denotes the conditional event, a simultaenously easy and difficult concept.  One way to think of it is as follows.

Consider a fair die with six sides.  It’s thrown.  What’s the probability of a six given that a side showing an even number lands upwards? (Van Frassen, 1976 used an example like this to explain the conditional event interpretation of the natural language if-then.)  This is P(lands six|lands even).  The idea is that you only consider cases where it’s showing an even number (2, 4, or 6). Assuming they’re all equally probable, then P(lands six|lands even) = 1/3.

Interpretation

The first stage of solving problems like that above is interpretating the problem in the language of the mathematical theory you want to use.

Let denote “has cancer”, denote “does not have cancer”, denote “shows a positive test result”, and denote “shows a negative test result”.

Let’s take each item of information individually.

1% of women at age forty who participate in routine screening have breast cancer.

There’s a mix of information here: a percentage of people (1%), from a particular sub-population (women, aged 40, who participate in routine screening), and a property they have.  From the problem it is clear that the interpretation is supposed to be:

But one can imagine a more complicated formalisation, for instance if the population of interest contains women of many different ages, some, but not all, of whom were screened because they had some worry about their health.

Next sentence:

80% of women with breast cancer will get positive mammographies.

This is an instance of

X% of people with property A have property B

The intended interpretation is P(B|A) = X%, but this might not be obvious to all readers.  Take some:

Some people with property A have property B

If this is interpreted as an existential quantifier, then it also follows that some people with property B have property A.  The conditional event, B|A, is in general not reversable in this way, so would not be suitable for the interpretation of an existential “some”.  Consider the following statement:

All people with property A have property B

This is not (in general) reversable. The percentage quantifier (used in the problem description) is also not reversible.  So there’s quite a lot of trickiness involved in interpreting this innocent looking statement. Given some background knowledge (we know the article is about Bayes’ rule, and about conditional probabilities), the intended interpretation of the original information is:

The idea is that if we choose a person at random from the population of interest, who has cancer (i.e., we know for sure she has cancer), then the probability of her having a positive test result is .8.

Then similarly for the last sentence:

9.6% of women without breast cancer will also get positive mammographies.

The formalisation is:

Here is the summary:

Now the problem statement:

A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

We have to infer . Note how this is a reversal of the conditional statements we encounted in the information given about the test.

Calculation

Now comes the calculation. A good place to start when thinking about conditional probability is the ratio formula for the probability of a condititional event:

Take an interpretatation of “If it is raining, then I have an umbrella” as the conditional event expression:

I have an umbrella  |  it is raining

The probability of this is the probability that I have an umbrella and it is raining, divided by the probability that it is raining.

This can easily be rewritten to

So if you know the probability of rain, and the probability that I have an umbrella when it rains, then you can multiply them to infer the probability that it is raining and I have an umbrella.

One step towards Bayes’ rule begins with:

1. 
2. [ in (this) probability theory, so it does not matter what order you write them]

From 2 we can infer , which slots into 1 to give

Now use the same variables as in the original problem

We can already fill in the numerator (top row) with and , but not yet the denominator (bottom row).

Let’s work a bit further then. We can infer as follows:

Which is easily calculated from the rewrite of the conditional probability above:

One more thing: .  So this gives:

Now we have everything we need:

.

I’ve moaned a bit about (what felt at the time to be a religion of) “effect size”. Recently Thom Baguley has published a paper on the topic, comparing standardised effects measures, which involve scaling with respect to the sample variance, with simple effects measures, which are expressed in the original units of measurement.

Baguley reviews some of the problems with standardised measures, all related to factors affecting sample variance. In general he advises reporting simple effect sizes, and preferably with confidence intervals.  If you really want to use standardised measures, for instance to compare conceptually similar measures on different scales, then he advises against reporting absolute and “canned” judgements like “small”, “medium”, and “large”, arguing instead in favour of descriptions about the relative size of effects.

I like his Tukey quote:

“… being so disinterested in our variables that we do not care about their units can hardly be desirable.”

It does seem odd to focus on how much variance is explained rather than actually characterising the nature of relationships between variables.

Reference

Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100, 603–617.

Another brief interlude from the stats and cog psych, but related to individual differences in reasoning I think! Interesting aside in a book by Robert Sutton on dealing with assholes (technical term here) in the workplace. He clarifies his defintion of the particular breed of asshole (technical term) in whom he is interested (pp. 16–17):

My focus is squarely on screening, reforming, and getting rid of people who demean and damage others, especially others with relatively little power. [...] I am a firm believer in the virtues of conflict, even noisy arguments.

Here’s a special case he describes (pp. 18–19) of people who can occasionally appear to be assholes (technical term), but are not:

I also want to put in a good word for socially awkward people [...]. I was struck by how many successful leaders of high-tech companies and creative organizations like advertising agencies, graphic design firms, and Hollywood production companies, had learned to ignore job candidates’ quirks and strange mannerisms, to downplay socially inappropriate remarks, and instead, to focus on what the people could actually do.

Examples he gives include autistic people and those with Tourette’s syndrome.

Reference

Sutton, R  (2007). The No Asshole Rule. NY: Business Plus.

The important details:

“11am, near the Maudsley Hospital in Denmark Hill, Saturday 19th September, to walk to the Tavistock Clinic in the leafy suburb of Belsize Park for about 4-5ish.”

And:

The walk is about 8 miles but I’m planning for a few minor detours for interesting sites (grounds of the old Bedlam Hospital, now the Imperial War Museum, St Thomas’ Hospital and the like) and with stops for lunch and maybe the occasional pint.

More info over here.

Wish I was in London to join this!

(Conflict of interest: I received my copy as a freebie from the publisher.)

Michael and Ellen Kaplan’s book, Bozo Sapiens, begins with the observation that (always other…) people tend to make stupid mistakes, by their own measures of stupidity.  PopSci books related to your research topic can be painful to read (the combination of results not being reported in detail with the realisation that they can’t be reported in detail), but the reward tends to be a reminder of what initially attracted you to the field and the occasional anecdote for teaching. So, off I went.

Where the book really shines is in its many examples of reasoning and decision making in the wild. For instance, how a pilot with too much (but unfortunately just recently out of date) knowledge of the air conditioning system on a 737 contributed to the death of 47 passengers (p. 117). Examples of the way pilots and air traffic control communicate successfully with each other in times of crisis (p. 142). A transcript of the events leading up to the mistaken shelling of friendly forces in Iraq (p. 91) revealing the conflicting decision processes and realisation of the error. The book is packed with great examples.

There were a couple of annoyances. The introduction of (classical) logical validity on p. 7 is wrong. They confuse it with consistency. Take this example.

All dogs have five legs.
Rex is a dog.
Therefore Rex has five legs.

The argument is valid because if the premises are true, then so is the conclusion. Now take a description of people at a party:

Some of the cute girls were tipsy
Some of the tipsy people were German
Therefore some of the cute girls were German

The conclusion is consistent with the premises: it is possible for the conclusion to be true if the premises are true. For instance if there were 50 people at the party, and 7 of them were cute German girls who were tipsy. The argument is, however, not valid: it is not necessary that the conclusion is true if the premises are. Suppose all the cute girls were Austrian, and none of the other girls were cute. A few of the Austrians were tipsy. There were Germans at the party, but none of them were female.

There are some fun psychological things going on with these kinds of sentences related to assumptions of cooperativeness and sensitivity to information ordering (one of which is called… the figural effect).

Wason’s 1966 selection task is introduced very briefly (pp. 115–116). There’s a mass of literature studying the task, and much of it was declared a waste of space by Sperber and Girotto (2002) [Use or misuse of the selection task? Rejoinder to Fiddick, Cosmides and Tooby. Cognition, 85, 277-290] and others. Still, it would have been nice to have a few words on the different interpretations people have. How some of these may be reasonable. How people with high g tend to give the answer Wason expected. The effect denotic content, e.g., about drinking laws, has on people’s performance. And so forth. But I guess the point was, as is often the application of the task, to demonstrate that the reader is stupid.

There are a few similar glosses on lab tasks which don’t really do them justice.  However the endnotes are very detailed so you can follow up references and see what the original papers said.  There are many good choices in there, e.g., a paper by Kemp and Tenenbaum on structure learning.

So overall, the book is great as a collection of examples and anecdotes, and might encourage people to learn more about the details.

As yet I haven’t convinced myself that SEM is a good idea—or at least not in the examples I’ve seen in psychology. Two reasons for starters: (i) fit statistic madness and (ii) weird analyses driven almost entirely by correlations or at best with vague theorizing based on very trival analyses of the tasks (it has a bit of this and bit of that…).

Anyway, recently Andrew Gelman noted:

“… there’s a research paradigm in which you fit a model—maybe a regression, maybe a structural equations model, maybe a multilevel model, whatever—and then you read off the coefficients, with each coefficient telling you something. You gather these together and those are your conclusions.

“My paradigm is a bit different. I sometimes say that each causal inference requires its own analysis and maybe its own experiment. I find it difficult to causally interpret several different coefficients from the same model.”

He goes on in the discussion to add:

“I have no problem with multiple-equation models, including measurement-error models, multilevel models, and instrumental variables. But I’m skeptical of trying to answer several casual questions by fitting a single model to a dataset.”

Interesting discsussion starting over here.

Found some code over here.

There are quite a few holy wars in the psychology of reasoning, the most famous of which was between mental rules and mental models.  This has a negative affect on the community: researchers fight to support their own brand, even when the empirical evidence suggests that major revisions are needed (yes, the data offer challenges to theories, even in psychology!), and mistrust or ignore results interpreted using ideas from competing brands.  It’s a problem even when just chatting with someone about results.  For instance one day I tried, patiently, to explain an empirical result to someone, who got angry about the theoretical framework in which the result was interpreted, and thus refused to look at data which would have been relevant to (and actually supported, once you looked at the numbers) his brand.

Anyway, was nice to see relevant comments from two different corners of the literature lying on my desk.  The first comes from a discussion of (using) the ACT-R theory (Anderson, 2007, p. 19):

“… this book is not about ACT-R; rather I am using ACT-R as a tool to describe the mind. [...] We may be proud of our ACT-R models and think they are better than others [...] but we try not to lose track of the fact that they are just a way of describing what is really of interest.”

Later (p. 239) Anderson cites Herbert Simon, who complained that

“‘brand names’ tend to make difficult the analysis and comparison of [...] mechanisms or the exchange of knowledge between research groups. [...] Physicists did not divide quantum mechanics into the Heisenberg Brand, the Schrodinger Brand, and the Dirac brand.”

Next example, from a book on social class (Wright, 1997, p. 34):

“Readers who are highly skeptical of the Marxist tradition for whatever reasons might feel that there is no point in struggling through the numbers and graphs in the rest of this book. If the conceptual justifications for the categories are unredeemably flawed, it might be thought, the empirical results generated with those categories will be worthless. This would be, I think, a mistake. The empirical categories themselves can be interpreted in a Weberian or hybrid manner. [...] As is usually the case in sociology, the empirical categories are underdetermined by the theoretical frameworks within which they are generated or interpreted.”

All commonsense, really.  But this important faculty of reasoning appears to be lost in practice, due, I suspect, to the very competitive nature of research, e.g., the joys of peer-review, trying to look clever at conferences (and, maybe more importantly, to students), and how tricky it is to continue to appear credible when one changes one’s mind.

Solutions please?

References

Anderson, J. R. (2007). How can the human mind occur in the physical universe? Oxford University Press

Wright, E. O. (1997). Class counts: student edition. Cambridge University Press.