Voodoo time

Sometimes I worry about the level of statistical expertise required these days in many fields of scientific endeavour – remembering here once reading, more than thirty years ago, about the bad statistics in many of the psychology papers published in very respectable journals – this being when things were a lot simpler, when there were a lot fewer statistical packages and a lot fewer computers to abuse them with than there are now. Not to mention the heavy statistical lifting needed to get a Higgs boson out of the inverse femtobarns of raw stuff they get out of the LHC (see reference 1). Do the teams which knock out all the scientific paper take sufficient care with their statistics in their scramble up the academic ladder? Or their scramble for grants? Do they have enough of the sort of statistical support that they need? Are there enough statisticians of the right sort about – that is to say, not the sort that I used to be? One did need to know stuff, but one did not need to know much about statistics to do population statistics. And sometimes I struggle with the bad writing which often seems to clothe what one supposes – or at least hopes – to be good science.

Then a few days ago, more or less by chance, I came across a paper all about lack of statistical rigour in one particular corner of the neurological field. A paper which was written back in 2008 and made quite a splash for itself at the time by including the word ‘voodoo’ in its title, a word which was removed from the more prosaic title to the second edition, ‘Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition’. A paper which attracted a huge amount of debate, reaching out to the lay press, with one blogger listing near 150 contributions. A paper which led me onto others, and all of which was all the more alarming to me as I had some considerable trouble working out even some of what the authors were on about, despite having some statistical background, and despite some of the authors’ evident concern to keep things simple.

So I tried to help myself out with a story.

Suppose we do a series of paired fMRI scans to investigate the effect on the brain of some bit of behaviour. Perhaps with the first of the pair being neutral – resting state might be the right bit of jargon – and the second of the pair being when the subject is doing the bit of behaviour in question. I think the technical term for such a pair is a contrast. For example, thinking about something important, perhaps the price of red lentils from Ontario, a few months before the next harvest is in. Perhaps being shown a picture of same. Or perhaps being frightened or disgusted by something – but we stick with lentils for the moment. The idea then is that you add all this up, play spot the difference between the summed versions of each of the two elements of the pair and then come up with the bit of the brain which worries about the price of lentils.

Elaborating, let us further suppose that we do twice 5 scans for each of a 100 people, once for neutral and once for lentils. Simplifying, we still further suppose that each scan comprises a sequence (in time) of 50 volumes (a three dimensional image), with each volume made up of 100,000 voxels, roughly 3mm cubes of brain, perhaps arranged as 30 more or less horizontal slices, each of 60 by 60 voxels. For each such voxel we have the value of some tricky variable, probably a floating point number, perhaps a variable called activation, a measure of how busy the neurons in that voxel were at the time of the scan, a period of perhaps 3 seconds. Roughly 5 billion numbers, so quite a lot of data, although still quite a lot less than the raw output from the scanner.

To keep things in proportion, voxels might be quite small, but might contain, on average about 150,000 neurons each – a total of 15 billion. I exclude the cerebellum which has another 75 billion, with both counts being something of a moveable feast. But the important point here is that one could build a very big computer program, doing all manner of clever things, out of 150,000 of the artificial neurons implemented in lots of computer packages. A voxel could be doing a lot of stuff under the covers.

Let us simplify some more and suppose that we are not interested in progression through time, in time series, that we discard the many and varied possibilities for comparison offered by working with a time series for each voxel rather than just a number. We are just doing scans over short periods of time for which we have some bit of behavioural data. To lentil or not to lentil.

Quite a lot of fancy statistical footwork is needed to turn the raw output of the scanner into these numbers, into these volumes, that is to say into pictures of activation that you or the computer can look at.

Quite a lot more fancy statistical footwork is needed to map all those pictures onto a standard brain so that you can compare the one with the other. Or add the one to the other.

And note that despite all the fancy footwork there is still a lot of noise. The brain itself is noisy and the business of collecting data from lots of brain over lots of time is also noisy. You can’t smooth all this noise out without smoothing most of the signal out with it – so there will be plenty of noise left. However, provided it is the right sort of noise, when one adds up lots of scans, the noise should add to zero, leaving pure signal. Provided, that is, that the signals don’t add up to zero too.

Leaving those problems aside, you can then ask Cortana, Watson or whoever to find a region of the brain from which you can reliably predict whether the scan in hand is a lentil one or a not to lentil one. You might simplify things by requiring any such region to be sensible in shape, perhaps something like a small sphere, rather than a torus or, worse still, a collection of bits and pieces taken from all over the brain.

Which all sounds fair enough. But I think the problem now is that the chances are that you will find such regions even when you feed in random data rather than brain data. There are enough small regions that some of them are going to be significant some of the time. Known elsewhere as the multiple comparison problem, well known to wikipedia.

I think an expensive way to deal with this problem – I am thinking here of the considerable expense of all this scanning –  is to use some brand new, clean data to test whatever hypothesis you had come up with. Perhaps to get some rival team to do the test. Does your significant region retain its significance with a new set of data? Or was your fine hypothesis grounded in noise and chance?

The good news is that all this may not as expensive as it would have been as recently as 2008. Lots and lots of fMRI data is now being loaded into public databases, for the use of all. There are lots of data out there to be mined.

So having got that bit of scanning for dummies off my chest, I hope to return to the substance of the voodoo paper in due course. Are the statistics really that dodgy? In the meantime, for the convenience of the keen reader, I list the references – with google well up for the ones that are not already links.

PS: I forget from where I got the illustration from. But thank you, whoever you are.

Reference 1: http://psmv2.blogspot.co.uk/2014/06/and-on-to-physics.html.

Reference 2: Voodoo Correlations in Social Neuroscience – Edward Vul and others – 2008

Reference 3: Circular analysis in systems neuroscience – the dangers of double dipping –Nikolaus Kriegeskorte and others – 2009

Reference 4: Confounds in multivariate pattern analysis: Theory and rule representation case study – Michael T. Todd and others – 2013

Reference 5: http://www.pea-lentil.com/.

Reference 6: http://blogs.discovermagazine.com/neuroskeptic/#.VomFHo_XLIU, the starting point of all this commotion.

Reference 7: http://www.ibm.com/smarterplanet/us/en/ibmwatson/. I don’t think Watson is actually onto all this yet. But it is probably only a matter of time.

More lego

I wonder if one of these would have worked when we were big into Lego. Drawers open, centre stack pulls out for better access to top - or top can be lifted off.

From http://ourweefamily.blogspot.ca/, via Facebook. The 'ca' is presumably for Canada not California, recent posts notwithstanding.

PS: I notice from this blog that Californians can do stately homes too, without bothering to come to the Home Counties. See the pictures of the rather extravagant Hearst Castle, day 6 of the trip to California, March 2014. Puts Polesden Lacey in its place. See reference 1.

Reference 1: http://hearstcastle.org/. Or ask google for 'hearst castle images' if you want some pictures. Synchronised swimming in the pool, rather than dressing up by resting luvvies.

Two dreams

All that is left of the first dream is a fragment which once contained a strong image of a flat bed Lego truck, with an array of Lego soldiers arranged in a hollow rectangle on the flat bed. Perhaps five soldiers long and three soldiers wide, that is to say five plus three plus five plus three less four for double counting equals twelve.

But they were not the sort of soldiers you get with Lego, even though they were in plug-in-able lego format, being more like the figures on the frieze of the Parthenon. I was very clear on that point in the dream, to the point of worrying about the lack of Lego authenticity while still in the dream.

I had originally thought to draw the lorry with its soldiers, but it quickly became apparent that while I could probably have managed this, and done quite a reasonable job, it would have taken me far more time than I cared to spend. So the illustration that I do post, which will, at least, remind me in weeks to come of what I was on about, is a composite image taken from Technic Lego and a bas-relief from Seljuk Turkey. With thanks to google. And with a request to the Powerpoint people to add a feature whereby an image can be rotated in something other than the plane of the screen. Or, if such a feature already exists, to put it somewhere where I can find it. That way, I would be able to do a much better job of patching the images of the soldiers onto the truck.

There is rather more left of the second dream, from last night, but I just post the end of it here. There was a system of underground pipes, perhaps some sort of drainage system, arranged in the form of a branching tree - or a root system - or the streams and small rivers making up the tributary system of a big river. I was crawling out, from larger pipes into smaller pipes.

Eventually I come to a dead end, blocked off by some sort of man-hole. A man-hole, oddly, with a port-hole. But there was no way out and I had to retrace my steps. I soon come to another man-hole, this time more like the sort of bulkhead door you get on ships. A door which also comes with a port-hole. I need to reach through the port-hole in order to be able to unlatch the door - thinking here of the sort of latch you get in old cottages and farm houses; of a metal version of the wooden one at reference 1. There is someone on other side and I ask him to take a screwdriver to the wire strap holding the port-hole in place, in order to get it off - thinking here of the sort of circular wire strap which holds the large, circular door washer onto the front of our washing machine. The strap which I can get off, but cannot, so far anyway, manage to put back on again. This someone gets the strap off easily enough, but manages to break the glass of the port-hole while he is at it. I reach through, open the door and step out, to find that there were actually two windows, both now broken, a large one above and a small one below. The large one was a complicated shape, a fancily shaped & trimmed bowl in some sort of delicately coloured glass. A bit like the sort of thing used as a ceiling lamp shade by an aunt of mine in the fifties of the last century, a glass thing hanging off four short chains. I think you can still buy them in the fancier lighting shops. Maybe, for example, Wray, although a quite peek does not reveal quite the thing I had in mind. See reference 2,

A helpful builder person (an actual person) from TB turns up and after peering at the large broken window for a bit, gives me credit for not trying to hide, to get away with the breakage, and announces that I can get a replacement from Travis Perkins, but that it will set me back £2,000. I could raise this sort of money but don't really want to. I wonder whether my employer will take the loss - while keeping me on.

Wake up.

I clearly had not been thinking very clearly, failing to work out that the someone might better have opened the door for me, rather than breaking the window.

Reference 1: http://psmv2.blogspot.co.uk/2015/05/deal-end.html.

Reference 2: http://www.christopherwray.com/.

A difficult picture 3

The very last picture of the old year. On the landscape screen of my desktop, it comes out a lot bigger (when you click on it) than that of the last post, but it does not capture the strangeness quite as well, at least not to my mind. Focus not quite as sharp either.

Well on the way to becoming a camera nerd!

A difficult picture 2

Another difficult picture, selected for posting from the very last sequence of the old year.

I was just getting back to the house when I was struck by the strange and wonderful appearance of these clouds, as usual a very transient thing. The effect had gone in well under a minute. Rather to my surprise, the telephone had quite a fair stab at it; there is enough of the strange and wonderful appearance left to remind me of what it was like on the day. The snap serves its purpose as a souvenir.

Oddly, another place for striking skyscapes is Tooting Broadway, looking west at the end of the afternoon. Whether that is down to Tooting or to my tending to be there at that time of day, I have no idea. There certainly is plenty of clear sky to the west as one walks that way, northwest to be precise, either down Mitcham Road or over the lights and down Garratt lane.

A difficult picture 1

The putative nipplewort mentioned the other day at reference 1, now transplanted to the bit of rough bed behind the new daffodil bed. It had a surprisingly substantial root ball, substantial while being compact. Looked fine after 24 hours, with one flower coming back out quite nicely, but we shall see what happens next. I don't even know if the thing is an annual or what - but I have staked it, so I should be able to find where it ought to be when I next think of it.

A difficult thing to photograph, it being a fairly small and insignificant plant and there being lots of clutter in the background. One of the scenes on which the eyeball does a much better job than a camera. Perhaps it would have done better had I got the primula bottom left out of view. Perhaps with Photoshop I would have been able to tone down the background after the event, but I have no idea if that package includes the kind of logic needed to discriminate between a nominated, quite complicated plant - including the dying flower stalk shooting out up and left - and the rest. The Lumia must have at least some of what is needed to do that as one can tell it what to focus on by tapping the screen, so perhaps the idea is not that far-fetched.

PS: whatever is the 31 doing in the file name of the post? Surely there are not 31 posts of that name? Not sure how I can easily check, not knowing whether or how I can query on the post title rather than the post body - there will be plenty with horton and clockwise there.

Reference 1: http://www.psmv2.blogspot.co.uk/2015/12/horton-clockwise_31.html.

Still Alice

One of my Christmas presents was the DVD of the film of this name. I had never heard of either the film or of any of the people in it, but it was none the worse for that. One of those emotional family films which the Americans are rather good at. Tear jerking even (for the ladies, that is).

The story of a happy family in which the mother catches something called familial Alzheimer’s disease in middle age. A disaster compounded by it rapidly turning out that at least one of her three children have it. The redeeming good news being that the unborn grandchildren don't.

I thought that the film had quite a fair stab at showing what such a diagnosis might be like, both for the person concerned and for the surrounding family.

Afterwards, I pondered about the merits of being told one has the disease, at a time when one has symptoms, but is still much the same person as one always was. For what it is worth, I decided that it was better to know. The down side is that you have to knowingly live with it, something you might otherwise, just possibly, have been spared. The up side is that you can better make helpful changes, adjustments to your life - and you can make preparations. You can, as they used to say, put your affairs in order, both as regards this world and the next.

The film itself has the down side for people of our age that we go through a hopefully short period of worrying more than we do already about such things as putting the butter in the microwave. Or, in the case of BH, of putting the olive flavoured margarine in the microwave. The up side that we have a bit more awareness and understanding of the whole sorry business. And one never knows which lot one is going to pull out of the urn; which of the deadly trio of heart, cancer and dementia is going to carry one off.

I am sure that I read a quote from Dr. Johnson quite recently which might have illustrated this post, but cannot now track it down, despite the wealth of same turned up by google. An entirely suitable senior moment for the occasion. Maybe I will even get around to reading the great man himself; at least one book of his has been sitting on the shelf unread for many years now. Drifting, I did once have the opportunity of buying his dictionary, which was tempting, but not tempting enough at £200 or so. From a fine secondhand bookshop which used to grace Crewkerne, now deceased. A shop which could also have sold me Holman Hunt's autobiography, also tempting, but not tempting enough. Not seen since. According to streetview, a shop called Gresham Books is still to be found at the right place, that is to say at 50.8833874,-2.7957175 - but it certainly wasn't there last time I looked. Perhaps google don't go round very often out in the provinces of the far west.

With thanks to http://www.alzinfo.org/ for the illustration actually used.

PS: I took a peek at Amazon but we decided to pass on the book from which the film was made.