You have to use theory (some of it from intuition) to come up with measures. If the theory’s crap then your measures will be crap. The theory IS mostly crap (psychology is a young social science!)—it doesn’t take a genius to see that. So… where to next…? Exploration? Trying out a few duff measures to see what happens? Use some sort of qualitative process where you talk to people, brainstorm, get them to come up with measures. Hmmmm maybe. But then you’re still relying on more unideal intuition. People don’t actually know at all what we should be measuring.
Cognitive psychology often gets interesting when counterintuitive results are discovered. People who are particularly good at visuospatial reasoning deciding not to use visuospatial reasoning. Discovering that people with poor working memory (well one flavour of) actually doing BETTER in a task that would appear to require working memory.
So there are duff measures around—plenty of them—and somehow they’re picking up something in the noise across a sample of a population. And they often correlate a wee bit with other duff measures. And then there are thought out theories lying around. Progress. Slow. Painful. It’s going somewhere though.
Occasionally people try item-level analyses. LOOK these items of Raven’s are easier for women and THESE are easier for men… Oh look at these particular items in this questionnaire… they seem to be picking up something interesting… why’s that…
There’s heaps of data, people have spent so much of their lives collecting the stuff. And then they just get time to add it up and run it through a correlation or throw it into a structural equation model. The brain is viewed as a collection of correlated Gaussian distributed variables(!). But surely there’s something else that can be done with it. I love the stuff that’s being done with Raven’s matrices. Well those who’ve been brave enough to stay clear of factor analysis.