Statistics is presently basically a mathematical discipline (if properly used, as social scientists may often not do, even if they apply statistical methods), though it is interesting to reflect that originally 'statistics' derives from the Italian, and describes the collection of numbers - relating to: inhabitants, imports, exports, incomes, taxable commodities - that are useful for the state and for government, which is a practice that originated in the Italian cities of the Renaissance.

Some knowledge and understanding of statistics and its foundations is mandatory both for scientists and philosophers, and for whomever else is aware how often one generalizes from data or makes guesses about the future based on information in the past.

The interested reader should be warned that a thorough understanding of statistics requires a thorough understanding of mathematics, including the mathematics of probability. Also, there are many subtle and at best partially resolved issues in both probability and statistics - though skeptics about probability and statistics should realize that gambling houses and insurance firms base firm profits on these fundamentally mathematical disciplines, so it does work in practice, mostly and usually.

Useful elementary introductions are Crowe et al., Edwards, Hodges and Lehman or Moroney. Kendall & Stuart is a thorough 3-volume set of advanced statistics that includes and explains a lot of the mathematics of statistics. Hacking, Kyburg and Pollock treat some of the problems of statistical inference. Hirsch Ed. gives a useful and clear summary of the principles of statistics and probability (and of much of mathematics). Hoel is a good introduction to the mathematics of statistics. Halmos provides the mathematical basis for much of both statistics and probability. Stegmüller provides clear expositions of the foundations of statistics and probability and of many of its problems, issues and (attempts of) solutions and recent extensions. Burks is a good introduction to the mathematics and philosophy of probability, causality and chance. Freudenthal is a good, clear, concise and elegant explanation of the foundations of probability and statistics.