Log in

Think Bayes - De Horror Vacui [entries|archive|friends|userinfo]
De Horror Vacui

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Think Bayes [Nov. 22nd, 2016|12:21 pm]
De Horror Vacui
Think Bayes, Allen B. Downey:

This is really the most accessible book on Bayesianism that I've seen. Which is strange, since it's a programming book. I had been trying to get through Savage's Foundations of Statistics, but there's a reason why it's been cited by more people than have read it, so I went ahead with this book which takes the same approach to a statistical valuation of knowledge as Savage had, but is more focused on practical problems. In fact, in being so practical, he discusses many of the problems I'd run into with a Bayesian view of induction in the past and some work-arounds.

The general idea around the Bayesian view of knowledge is that from whatever your initial views (sans a completely irrational view), by evaluating more evidence as it comes in sequentially in the proper Bayesian manner, your views will change to be closer to the truth. The second half of that is easier. The evaluation of evidence in this idea of rationality is the use of Bayes theorem which describes the relationship between dependent events:

The product of the probability of one event and the probability that you'll see the second given you see the first is equal to the product of the probability of the other event times the probability that you'll see the first given you see the first: P(A)P(B|A) = P(B)P(A|B)

This can be used to update your beliefs about the probability of a rule given some evidence for or against that rule. To do so, you need to interpret B (say) as evidence and A as the rule. Then P(A) is your initial evaluation of the likelihood of your rule (or hypothesis) and P(B|A) is the probability that the evidence occurs given rule is in effect. The first is called the "prior" and the second is called the "likelihood". The probability of the evidence _whether or not the rule is true_ is P(D) and is called the "normalizing constant." Finally, the updated probability is P(A|B) called the "posterior."

When new evidence comes in, you then take the postierior from the previous evaluation and use it as the prior for the new evidence. And continue until the differences between priors and posteriors becomes too small to care about.

And that's how induction works under Bayesian epistemology.

As far as it goes, this makes for a rational way to change your beliefs, rather than the haphazard random way that people really do. It's a very enticing view of the meaning of probability. It's fairly simple to extended this to multiple possible hypotheses and many kinds of evidence, and it's not too much harder to move to continuous variables.

The probelms come about because your beliefs have to be amenable to this interpretation. Most importantly, you can't be really, really attached to one particular rule -- you have to start with non-zero probabilities for every rule. You should probably keep them non-zero over time, which can really only happen if your likelihoods are never zero. The latter is less of a problem, but if it is true that your likelihood for a particular case is zero you could probably use stronger experimental methods than statistical ones.

The former issue, though, is critical. If you come into a situation saying that the probability of a particular rule is zero, even if it happens to be a combination of two other rules with high probabilities, it can never be improved, no matter what the evidence would say if you gave it even a 0.01% chance.

The book also has a lot of nice examples of particular situations. Data analysis, observer bias, and so on. It also goes into how to work around some of the issues I discussed. But mostly, it's just a very clear and concise description of what Bayesian statistics can do for you. It is not deeply philosophical or mathematical like Savage, but I think you get a better idea of what Bayesianism is about through this book than from more analytical treatments (I would say technical, but this is a very technical book -- very practical, just not deep).

Other books, 2016:

82. Functional Thinking, Neal Ford
81. Three Act Trajedy, Agatha Christie
80. Think Bayes, Allen B. Downey
79. Two-Sided Matching, Alvin E. Roth and Marilda A Oliveira Sotomayor
78. The Physics of Sailing Explained, Bryon D. Anderson
77. The Unpleasantness at the Bellona Club, Dorothy L. Sayers
76. The Evolution of Culture in Animals, John T. Bonner
75. The Psychopath Test, Jon Ronsson
74. Behind Closed Doors, Johanna Rothman and Esther Derby
73. Aegean Art and Architecture, Donald Preziosi and Louise A. Hitchcock
72. The Hollow, Agatha Christie
71. The Design of Design, Frederick P. Brooks, Jr.
70. Artificial Intelligence for Humans, Vol 1: Fundamental Algorithms, Jeff Heaton
69. Eric, Terry Pratchett
68. This Immortal, Roger Zelazny
67. One, Two, Buckle My Shoe, Agatha Christie
66. From Special Relativity to Feynman Diagrams, Riccardo D'Auria and Mario Trigiante
65. The Vanishing Tower, Michael Moorcock
64. Who Gets What -- and Why, Alvin E. Roth
63. Emperor and Clown, Dave Duncan
62. What Every Body is Saying, Joe Navarro
61. Linear Algebra and Geometry, Irving Kaplansky
60. Feynman's Tips of Physics, Richard P. Feynman
59. Perilous Sea, Dave Duncan
58. American Frontier Lawmen 1850 - 1930, Charles M. Robinson III
57. The Weird of the White Wolf, Michael Moorcock
56. How We THing, Alan H. Schoenfeld
55. Sailor on the Seas of Fate, Michael Moorcock
54. Creativity, Inc. Ed Catmull
53. Stable Marriage and Its Relation to Other Combinatorial Problems, Donald E. Knuth
52. Language Implementation Patterns, Terence Parr
51. The Myth of the Magus, E.M. Butler
50. Elric of Melibone, Michael Moorcock
49. Physlets: Teaching Physics with Interactive Curricular Material, Wolfgang Christian and Mario Belloni
48. Elephants Can Remember, Agatha Christie
47. Faery Lands Forlorn, Dave Duncan
46. Inevitable Illusions, Massimo Piattelli-Palmarini
45. Don't Make Me Think: A Common Sense Approach to Usability, Steve Krug
44. Matrix and Tensor Calculus, Aristotle D. Michal
43. The Magic Casement, Dave Duncan
42. A Mind for Numbers, Barbara Oakley
41. Hogfather, Terry Pratchett
40. Five Easy Lessons: Strategies for Successful Physics Teaching, Randall D. Knight
39. The Living God, Dave Duncan
38. The Art of Game Design: A Book of Lenses, Jesse Schell
37. The Maker of Universes, Philip Jose Farmer
36. Javascript Web Applications, Alex MacCaw
35. The Stricken Field, Dave Duncan
34. JavaScript: The Good Parts: Douglas Crockford
33. A Theory of Fun for Game Design, Raph Koster
32. An Introduction to Hilbert Space and Quantum Logic, David W. Cohen
31. Magic in the Middle Ages, Richard Kieckhefer
30. The Silver Warriors, Michael Moorcock.
29. Engines of Creation, K. Eric Drexler
28. Prince of Chaos, Roger Zelazny
27. Thinking Fast and Slow, Daniel Kahneman
26. Sparkling Cyanide, Agatha Christie
25. Knight of Shadows, Roger Zelazny
24. Death on the Nile, Agatha Christie
23. Feynman Lectures on Computation, Richard Feynman
22. Effective Computation in Physics, Anthony Scopatz and Kathryn D. Huff
21. How to Fail at Everything and Still Win Big, Scott Adams
20. Sign of Chaos, Roger Zelazny
19. Murder Must Advertise, Dorothy Sayers
18. The Mythical Man-Month, Fredrick Brooks
17. Blood of Amber, Roger Zelazny
16. Understanding Computation, Tom Stuart
15. Social Class in the 21st Century, Mike Savage
14. Design for Great-Day, Alan Dean Foster and Eric Frank Russell
13. QED: The Strange Theory of Light and Matter, Richart Feynman
12. SciPy and NumPy, Eli Bressert
11. Elementary Quantum Mechanics in One Dimension, Robert Gilmore
10. The Trumps of Doom, Roger Zelazny
9. Your Code as a Crime Scene, Adam Tornhill
8. Upland Outlaws, Dave Duncan
7. Identity Economics, George Akerlof and Rachel Kranton
6. The Courts of Chaos, Roger Zelazny
5. Nudge, Richard Thaler and Cass Sunstein
4. The Cutting Edge, Dave Duncan
3. The Nature of Software Development, Ron Jeffers
2. The Death of Chaos, L.E. Modesitt, Jr.
1. Kivy -- Interactive Applications and Games in Python, Roberto Ulloa