Stats jargon (and mindset of the field) assumes everything is choice and nature plays no role

[barking_iguana]

I've been learning how to model baseball, where the outcome of events is based on abilities of several people. It was hard, initially, to find the stuff I needed to learn, because the stats work done in the last 80 years or so is almost always presented in terms of a single individual making a choice. Competition and natural events aren't about choice, but they (especially the latter) comprise most of the things one might use the techniques for.

Now that I know what the statisticians' game is, I can think of everything in terms of Nature making a choice, where the abilities correspond to preferences on Natures part. But that's a pretty silly way of looking at everything, if you're not trying to fit into shoes marketed to someone else.

I majored in stats 30 years ago. I don't think this everything-is-a-choice ideology was pervasive in what I learned then. But most of what I learned had been developed before 1925 and most of what I'm learning now has been developed since. The upshot (aside from making it harder for someone now self-teaching to realize that the work I needed had been done) is to reinforce the tendency of the successful (those who need to take advances statistics courses being generally not without privilege, despite lack of income while studying) to pat themselves on the back and see their success as entirely a matter of virtuous choices.

Economists and political scientists, especially, it seems to me, would be better educated in a true sense if their feet were not always crammed into 'choice' shoes.

Comments

[chemoelectric.livejournal.com]

Read here about the Mind Projection Fallacy: bayes.wustl.edu/etj/articles/cmystery.pdf (Do not be misled by the reference to ‘Bayesian’, which is a catch-all term.)

[barking-iguana.livejournal.com]

You've pointed me to this before. I don't buy it. I think his hypothesis contains the essence of the wrong-thinking I described. Is this the same Jaynes who wrote how the bicameral mind is the recent origin of modern consciousness? It has the same tone of dense but promising theorizing that turns out to be bullshit, so I suppose it must be him.

[chemoelectric.livejournal.com]

This isn’t even within a megaparsec of being the same person. I don’t think you understand the paper.

I’m completely serious. I think you ought to reread it several times.

[chemoelectric.livejournal.com]

Can I ask you much physics you have had? I have a masters in EE and barely can follow all the details; one needs to know it somewhat to see what he’s saying, though.

[barking-iguana.livejournal.com]

I can't follow the physics. I can, for the most part, follow his more philosophical statements about the nature of probability..

[chemoelectric.livejournal.com]

Without the examples I think it would be easy to misunderstand what he is saying as having something to do with psychology rather than technical mathematical matters.

In a way the use of the EPR paradox as his main example is a bad one, because in that case the criticism is of a silly math error that no school of probability theory would disagree with. Where there is not actually a math error, it is a technical matter of which methods of thinking about a problem are the most productive. There is more than one way to skin a cat, though; it is probably not hard for skilled people to phrase everything Jaynes said in terms of the measure-theoretic approach that I think is the dominant model for probability these days.

OTOH Jaynes was a physicist and fundamental physics was and remains in such a sad state of muddled thinking that it is impossible not to get ‘philosophic’ about it. There is widespread discontent in the field, I think. It snuck into ‘Science’ a while back in which a reviewer of a book by Hawking took mocked string theory as ‘epicycles’. But mostly the ‘prestigious’ journals part of the sickness in sciences these days. (Recent press pertains mostly to the prevalence of cooked results in biomed papers in ‘prestige’ journals.)

[barking-iguana.livejournal.com]

Physicists over the past few decades have been working toward a perspective in which time can be factored out, in which all that has happened and all that will happen are simultaneously (to use a word without real meaning in that perspective) valid.

That is directly at odds with the traditional perspective (to use a word laden with meaning) which states that any phenomenon occurring at a time and place where the speed of light prevents it from affecting you is not, for scientific purposes, part of the universe in which you live.

It is in this traditional perspective that Heisenberg proved (as I understand it) that the universe at any given point in space-time contains insufficient information to fully describe itself while going about the business of existing. In such a perspective, properties must be taken, from a scientific perspective, to be undetermined, not merely unknown. It is an unsatisfying conclusion based on an unsatisfying conclusion stemming from an unsatisfying perspective, but it follows inexorably from the premise that what cannot even in a theoretical way be tested does not, in effect, exist.

If the newer outlook shows that someone living outside time as we experience it could indeed see all of space-time as a deterministic whole, that would be very satisfying to abstract curiosity and very troubling to human experience. But it would not change the constraints in which we live, which are those of the traditional perspective.

So when someone talks of uncertainty never containing indeterminacy, it looks to me like grinding a philosophical ax rather than describing the word in which I live.

[chemoelectric.livejournal.com]

Heisenberg uncertainty isn’t something proven; it is one of the postulates making up what we call quantum mechanics. AFAIK the ‘proof’ is merely that quantum theory makes correct physical predictions, but that in no way precludes a deeper theory that had no uncertainty principle. The arguments that quantum theory is the be all and end all are based on optical experiments and Bell’s ‘theorem’; it is the latter that is bogus: http://en.wikipedia.org/wiki/Bell%27s_theorem

[chemoelectric.livejournal.com]

As for a static world in which everything is simultaneous and ‘unmoving’, I am pretty sure we have had that since Minkowski.

[chemoelectric.livejournal.com]

For reference:

E T Jaynes: http://en.wikipedia.org/wiki/Edwin_Thompson_Jaynes

Julian Jaynes: http://en.wikipedia.org/wiki/Julian_Jaynes

[barking-iguana.livejournal.com]

That's a relief, anyway.

[chemoelectric.livejournal.com]

I’m not familiar with how statisticians talk, but I’ve reread the above and think more than before that you are arguing along similar lines to Jaynes. The basic idea to get around this is (a) determine all the relevant data that is known, then (b) write a mathematical description that reveals facts not otherwise obvious. These have to be facts, so no ‘degrees of belief’ can matter; and the idea that nature makes ‘choices’ is preposterous. Jaynes criticizes physicists for believing that nature makes choices, and elsewhere criticizes statisticians for discounting relevant data if it doesn’t have a place in their models. (I can’t say much about that.)

What happened in physics is that somehow the notion got out there that the apparent discrete ‘randomness’ in quantum theory could not be explained by recourse to a deeper theory in which, perhaps, all motions were determinate, like in Newton’s or Einstein’s theory. Einstein and friends pointed out that if this were the case then you had to have a form of instantaneous action at a distance. Since then physicists have found ways to argue that this instantanous action at a distance is ‘real’; the argument goes like this: (a) We perform an experiment with correlated particles; (b) once the particles have been shot out -- that is, once nature has made its choice -- they can’t affect each other; (c) therefore we can write P(a) times P(b) instead of using conditional probabilities. Step (c) is mere handwaving, without any basis whatsoever. The problem is that physicists really think of a probability expression for particles in terms of nature making a ‘choice’ and so they keep coming back to (c).

We don’t actually know that the electromagnetic activity in these experiments consists of particles, by the way; the discretization could be taking place in the detectors. Maybe it’s really a continuous field. The statistical results in the experiments are actually those predicted by classical electrodynamics, assuming a field and discretization in the detectors. I would have thought this would settle the matter, but great verbal effort goes into justifying (c) above on the basis of (b).

(I think that, ironically, it was Einstein who came up with the notion of a photon to explain the discretization in photodetectors.)

[barking-iguana.livejournal.com]

Choice is implies volition, which we have no way of discerning and good reason to discount, when it comes to nature. But when you say that nature making choices is preposterous, I wonder if you mean something different, and likely something that I see no reason to call preposterous.

[chemoelectric.livejournal.com]

Don’t think of it as volition. I need to explain better.

The scientists do two thought experiments for the actual experiment, once using quantum theory and once using classical theory. The quantum theory reasoning comes up with the correct result. There is no need for ‘Copenhagen interpretation’ or anything like that, because the theory will give the same result no matter what metaphor you use. Then they do a classical thought experiment. In the classical thought experiment, the moment of ‘choice’ occurs when the two particles are ejected away from each other. After that, of course, neither can have an effect on the other; so, the scientists argue, we can from now on assume they are statistically independent. This step is what I call ‘nature making a choice’ -- nature chose that the correlated particles become uncorrelated particles.

The conventional view of quantum mechanics, then, is that these now uncorrelated, statistically independent particles become ‘correlated again’ through an instantaneous action at a distance at ‘wave collapse’. This is how fundamental physicists actually think, though if you state it outright the silliness is evident and so instead they say it is a mystery, like the Trinity.

The classical argument from that point on assumes no correlation and comes up with results that are inconsistent with the physical experiment, proving nothing.

[chemoelectric.livejournal.com]

Ugh, when I wrote that I hadn’t even realized how much like the Trinity the mystery is. I had forgotten that they explain it with a Duality, by saying, well, it’s a particle, but at the same time it is a wave. That’s how they can avoid stating that they think the particles are uncorrelated once they leave the source; but they prove that’s what they think, by how they formulate the classical solution.

[chemoelectric.livejournal.com]

If you ever get a chance, read John Bell’s ‘Bertlmann’s socks’ article (in his ‘Speakable and Unspeakable’ or whatever it’s called book) and think about his reasoning with the socks. I haven’t lately but when I did it became clear he was wrong about the socks, and at the time I was able to explain why very simply and succinctly. I’m too lazy to redo that now, being busy writing geometric algebra code and studying the Psalms. :)

I suspect statisticians are not too different in their afflictions from Bell’s and others’, because I once had an argument with a statistician about predicting paths of airplanes, in which I argued that past motion of the plane could in principle be relevant and the statistician argued it could not be. I think the problem with Bertlmann’s socks was something like that -- Bell assumes we can’t learn anything about Prof. Bertlmann from his past behavior.

[chemoelectric.livejournal.com]

I must have been thinking about this in my sleep (i.e., forming new synapse strengths, etc.), because I figured out how to explain this as physicists thinking of nature as making choices. They treat the polarity of emitted light ‘particles’ as a problem of choosing a polarity from an urn (which can be done by an automaton, so volition does not come into it). In fact we have no idea how the polarity of these ‘particles’ comes about and we should be completely agnostic about it.

Certain light sources are such that if you measure the light emitted in opposite directions then there are correlations between their polarizations, which you can measure by sticking polarizing filters in front of photodetectors. This is true whether you use quantum theory, assuming whatever you want, or Maxwellian electrodynamics, assuming electromagnetic waves. Photodetectors respond with clicks occurring at different rates, rather than steady currents, however (which is what caused Einstein to get a Nobel Prize). So, Maxwell getting the right result tends not to register as evidence that there is nothing especially bizarre about the correlations. The reasoners go right ahead and assume that, because the measurement at one photodetector cannot classically affect the measurement at the other photodetector, they can drop the ‘|condition’ in P(x|condition). It is the legendary confusion of causation and correlation, except with lots of verbiage to justify it.

There is much money and immense vats of reputation tied up in maintaining this situation. ‘Quantum’ computing, assuming it makes sense at all (which I have no reason to doubt), might be doable with macroscopic devices; how to justify all the effort that has gone into ‘quantum’ devices? Quantum cryptography, supposedly unbreakable in principle, might be a complete joke; at any point some smart person might come up with a deeper physical theory that lets you break the cipher with ease. And how are career scientists, in the modern atmosphere of citation churning in ‘prestige’ journals, to explain that they taught their students never to look for a deeper theory and shunned those few who dared to do so anyway? It’s just not likely to happen until there are enough apostates to form a critical mass, or they manage to sneak some papers into the prestige press. (One paper did make it into the Proceedings of the National Academy of Sciences, but it wasn’t very good. I had no idea what the authors were trying to say, and I’m not sure anyone else did, either.)

[barking-iguana.livejournal.com]

Thank you for the explanation. It is clear. You must realize that though I give considerable weight to your depiction and that such collective stupidity of scientists does happen, belief that it does happen, even by very smart people such as yourself, is much more common than the actual phenomenon. So I am forced into agnosticism myself, rather than completely taking your word for it. But your word counts for more than the vast majority of others' would.

[chemoelectric.livejournal.com]

I am going to post a page from the Bertlmann’s socks article, in which the analogy is made to epidemiological studies, and in which you can see the point at which Bell resorts to handwaving and, separately, how he confuses correlation and causation.

I looked back and the actual ‘Bertlmann socks’ analogy is to an eccentric professor who always wears mismatched socks. If you see his left sock first and it is pink, you can know already that the other sock is not-pink. I think this is supposed to be a strawman of the arguments of opponents of ‘non-locality’, and the handwaving is supposed to show that where there is not an intelligent being behind it all you can make ‘simplifying assumptions’.