Think Bayes 
[Nov. 22nd, 201612:21 pm]
De Horror Vacui

Think Bayes, Allen B. Downey:
This is really the most accessible book on Bayesianism that I've seen. Which is strange, since it's a programming book. I had been trying to get through Savage's Foundations of Statistics, but there's a reason why it's been cited by more people than have read it, so I went ahead with this book which takes the same approach to a statistical valuation of knowledge as Savage had, but is more focused on practical problems. In fact, in being so practical, he discusses many of the problems I'd run into with a Bayesian view of induction in the past and some workarounds.
The general idea around the Bayesian view of knowledge is that from whatever your initial views (sans a completely irrational view), by evaluating more evidence as it comes in sequentially in the proper Bayesian manner, your views will change to be closer to the truth. The second half of that is easier. The evaluation of evidence in this idea of rationality is the use of Bayes theorem which describes the relationship between dependent events:
The product of the probability of one event and the probability that you'll see the second given you see the first is equal to the product of the probability of the other event times the probability that you'll see the first given you see the first: P(A)P(BA) = P(B)P(AB)
This can be used to update your beliefs about the probability of a rule given some evidence for or against that rule. To do so, you need to interpret B (say) as evidence and A as the rule. Then P(A) is your initial evaluation of the likelihood of your rule (or hypothesis) and P(BA) is the probability that the evidence occurs given rule is in effect. The first is called the "prior" and the second is called the "likelihood". The probability of the evidence _whether or not the rule is true_ is P(D) and is called the "normalizing constant." Finally, the updated probability is P(AB) called the "posterior."
When new evidence comes in, you then take the postierior from the previous evaluation and use it as the prior for the new evidence. And continue until the differences between priors and posteriors becomes too small to care about.
And that's how induction works under Bayesian epistemology.
As far as it goes, this makes for a rational way to change your beliefs, rather than the haphazard random way that people really do. It's a very enticing view of the meaning of probability. It's fairly simple to extended this to multiple possible hypotheses and many kinds of evidence, and it's not too much harder to move to continuous variables.
The probelms come about because your beliefs have to be amenable to this interpretation. Most importantly, you can't be really, really attached to one particular rule  you have to start with nonzero probabilities for every rule. You should probably keep them nonzero over time, which can really only happen if your likelihoods are never zero. The latter is less of a problem, but if it is true that your likelihood for a particular case is zero you could probably use stronger experimental methods than statistical ones.
The former issue, though, is critical. If you come into a situation saying that the probability of a particular rule is zero, even if it happens to be a combination of two other rules with high probabilities, it can never be improved, no matter what the evidence would say if you gave it even a 0.01% chance.
The book also has a lot of nice examples of particular situations. Data analysis, observer bias, and so on. It also goes into how to work around some of the issues I discussed. But mostly, it's just a very clear and concise description of what Bayesian statistics can do for you. It is not deeply philosophical or mathematical like Savage, but I think you get a better idea of what Bayesianism is about through this book than from more analytical treatments (I would say technical, but this is a very technical book  very practical, just not deep).
Other books, 2016:
82. Functional Thinking, Neal Ford 81. Three Act Trajedy, Agatha Christie 80. Think Bayes, Allen B. Downey 79. TwoSided Matching, Alvin E. Roth and Marilda A Oliveira Sotomayor 78. The Physics of Sailing Explained, Bryon D. Anderson 77. The Unpleasantness at the Bellona Club, Dorothy L. Sayers 76. The Evolution of Culture in Animals, John T. Bonner 75. The Psychopath Test, Jon Ronsson 74. Behind Closed Doors, Johanna Rothman and Esther Derby 73. Aegean Art and Architecture, Donald Preziosi and Louise A. Hitchcock 72. The Hollow, Agatha Christie 71. The Design of Design, Frederick P. Brooks, Jr. 70. Artificial Intelligence for Humans, Vol 1: Fundamental Algorithms, Jeff Heaton 69. Eric, Terry Pratchett 68. This Immortal, Roger Zelazny 67. One, Two, Buckle My Shoe, Agatha Christie 66. From Special Relativity to Feynman Diagrams, Riccardo D'Auria and Mario Trigiante 65. The Vanishing Tower, Michael Moorcock 64. Who Gets What  and Why, Alvin E. Roth 63. Emperor and Clown, Dave Duncan 62. What Every Body is Saying, Joe Navarro 61. Linear Algebra and Geometry, Irving Kaplansky 60. Feynman's Tips of Physics, Richard P. Feynman 59. Perilous Sea, Dave Duncan 58. American Frontier Lawmen 1850  1930, Charles M. Robinson III 57. The Weird of the White Wolf, Michael Moorcock 56. How We THing, Alan H. Schoenfeld 55. Sailor on the Seas of Fate, Michael Moorcock 54. Creativity, Inc. Ed Catmull 53. Stable Marriage and Its Relation to Other Combinatorial Problems, Donald E. Knuth 52. Language Implementation Patterns, Terence Parr 51. The Myth of the Magus, E.M. Butler 50. Elric of Melibone, Michael Moorcock 49. Physlets: Teaching Physics with Interactive Curricular Material, Wolfgang Christian and Mario Belloni 48. Elephants Can Remember, Agatha Christie 47. Faery Lands Forlorn, Dave Duncan 46. Inevitable Illusions, Massimo PiattelliPalmarini 45. Don't Make Me Think: A Common Sense Approach to Usability, Steve Krug 44. Matrix and Tensor Calculus, Aristotle D. Michal 43. The Magic Casement, Dave Duncan 42. A Mind for Numbers, Barbara Oakley 41. Hogfather, Terry Pratchett 40. Five Easy Lessons: Strategies for Successful Physics Teaching, Randall D. Knight 39. The Living God, Dave Duncan 38. The Art of Game Design: A Book of Lenses, Jesse Schell 37. The Maker of Universes, Philip Jose Farmer 36. Javascript Web Applications, Alex MacCaw 35. The Stricken Field, Dave Duncan 34. JavaScript: The Good Parts: Douglas Crockford 33. A Theory of Fun for Game Design, Raph Koster 32. An Introduction to Hilbert Space and Quantum Logic, David W. Cohen 31. Magic in the Middle Ages, Richard Kieckhefer 30. The Silver Warriors, Michael Moorcock. 29. Engines of Creation, K. Eric Drexler 28. Prince of Chaos, Roger Zelazny 27. Thinking Fast and Slow, Daniel Kahneman 26. Sparkling Cyanide, Agatha Christie 25. Knight of Shadows, Roger Zelazny 24. Death on the Nile, Agatha Christie 23. Feynman Lectures on Computation, Richard Feynman 22. Effective Computation in Physics, Anthony Scopatz and Kathryn D. Huff 21. How to Fail at Everything and Still Win Big, Scott Adams 20. Sign of Chaos, Roger Zelazny 19. Murder Must Advertise, Dorothy Sayers 18. The Mythical ManMonth, Fredrick Brooks 17. Blood of Amber, Roger Zelazny 16. Understanding Computation, Tom Stuart 15. Social Class in the 21st Century, Mike Savage 14. Design for GreatDay, Alan Dean Foster and Eric Frank Russell 13. QED: The Strange Theory of Light and Matter, Richart Feynman 12. SciPy and NumPy, Eli Bressert 11. Elementary Quantum Mechanics in One Dimension, Robert Gilmore 10. The Trumps of Doom, Roger Zelazny 9. Your Code as a Crime Scene, Adam Tornhill 8. Upland Outlaws, Dave Duncan 7. Identity Economics, George Akerlof and Rachel Kranton 6. The Courts of Chaos, Roger Zelazny 5. Nudge, Richard Thaler and Cass Sunstein 4. The Cutting Edge, Dave Duncan 3. The Nature of Software Development, Ron Jeffers 2. The Death of Chaos, L.E. Modesitt, Jr. 1. Kivy  Interactive Applications and Games in Python, Roberto Ulloa 

