Ch. 2 Probability Theory 1 Descriptive Study of Data 1.1 Histograms and Their Numerical Characteristics By descriptive study of data we refer to the summarization and exposition(tab- ulation, grouping, graphical representation) of observed data as well as the derivation of numerical characteristics such as measures of location, dispersion and shape Although the descriptive study of data is an important facet of modeling with real data itself, in the present study it is mainly used to motivate the need for probability theory and statistical inference proper In order to make the discussion more specific let us consider the after-tax ersonal income data of 23000 household for 1999-2000 in the uS. There data in row form constitute 23000 numbers between $5000 and $100000. This presents us with a formidable task in attempting to understand how income is distributed among the 23000 households represented in the data. The purpose of descriptive catistics is to help us make some sense of such data. A natural way to proceed to summarize the data by allocating the numbers into classes(intervals). The number of intervals is chosen a priori and it depends on the degree of summa- rization needed. Then we have the Table of the personal income in the US The first column of the table shows the income intervals. the second column the second column shows the number of income falling into each interval and the third column the relative frequency for each interval. The relative frequency is calculated by dividing the number of observations in each interval by the total number of observations. The fourth column is the cumulative frequency. Sum- marizing the data in this Table enables us to get some idea of how income is distributed among various class. If we plot the relative(cumulative)frequencies in a bar graph we get what is known as the histogram(cumulative) For further information on the distribution of income we could calculate vari- ous numerical characteristics describing the histogram's location, dispersion and shape. Such measure can be calculate directly in terms of the raw data. How- in the expositional purpose grouped data. The main reason for this is to introduce various concept which ill b ted in the context of probability
Ch. 2 Probability Theory 1 Descriptive Study of Data 1.1 Histograms and Their Numerical Characteristics By descriptive study of data we refer to the summarization and exposition (tabulation, grouping, graphical representation) of observed data as well as the derivation of numerical characteristics such as measures of location, dispersion and shape. Although the descriptive study of data is an important facet of modeling with real data itself, in the present study it is mainly used to motivate the need for probability theory and statistical inference proper. In order to make the discussion more specific let us consider the after-tax personal income data of 23000 household for 1999-2000 in the US. There data in row form constitute 23000 numbers between $5000 and $100000. This presents us with a formidable task in attempting to understand how income is distributed among the 23000 households represented in the data. The purpose of descriptive statistics is to help us make some sense of such data. A natural way to proceed is to summarize the data by allocating the numbers into classes (intervals). The number of intervals is chosen a priori and it depends on the degree of summarization needed. Then we have the ” Table of the personal income in the US”. The first column of the table shows the income intervals, the second column the second column shows the number of income falling into each interval and the third column the relative frequency for each interval. The relative frequency is calculated by dividing the number of observations in each interval by the total number of observations. The fourth column is the cumulative frequency. Summarizing the data in this Table enables us to get some idea of how income is distributed among various class. If we plot the relative (cumulative) frequencies in a bar graph we get what is known as the histogram (cumulative). For further information on the distribution of income we could calculate various numerical characteristics describing the histogram’s location, dispersion and shape. Such measure can be calculate directly in terms of the raw data. However, in the present case it is more convenient for expositional purpose to use the grouped data. The main reason for this is to introduce various concept which will be reinterpreted in the context of probability. 1
The mean as measure of location takes the form z=∑ where o i and zi refer to the relatively frequency and the midpoint of interval i The mode as a measure of location refers to the value of income that occurs most frequently in the data set. Another measure of location is the median referring to the value of income in the middle when income are arranged in an ascending order according to the size of income. The best way to calculate the median is to plot the cumulative frequency grap Another important feature of the histogram is the dispersion of the relative frequency around a measure of central tendency. The most frequently used mea- sured of dispersion is the variance defined by ∑(a1-2) which is a measure of dispersion around the mean; v is known as the standard deviation We can extend the concept of the variance to i=1 defining what are known as higher central moments. These higher moments can be used to get a better idea of the shape of the histogram. For example, the standardized form of the third and fourth moments defined by known as the skewness and kurtosis coef ficients, measure the asymmetry and the peakedness of the histogram, respectively. In the case of a symmetric his- togram, SK=0 and the less d the histogram the 1.2 Looking Ahead The most important drawback of descriptive statistics is that the study of the observed data enables us to draw certain conclusions which relate only to the
The mean as measure of location takes the form z¯ = Xn i=1 φizi , where φi and zi refer to the relatively frequency and the midpoint of interval i. The mode as a measure of location refers to the value of income that occurs most frequently in the data set. Another measure of location is the median referring to the value of income in the middle when income are arranged in an ascending order according to the size of income. The best way to calculate the median is to plot the cumulative frequency graph. Another important feature of the histogram is the dispersion of the relative frequency around a measure of central tendency. The most frequently used measured of dispersion is the variance defined by v 2 = Xn i=1 (zi − z¯) 2φi , which is a measure of dispersion around the mean; v is known as the standard deviation. We can extend the concept of the variance to mk = Xn i=1 (zi − z¯) kφi , k = 3, 4, ... defining what are known as higher central moments. These higher moments can be used to get a better idea of the shape of the histogram. For example, the standardized form of the third and fourth moments defined by SK = m3 v 3 and K = m4 v 4 , known as the skewness and kurtosis coefficients, measure the asymmetry and the peakedness of the histogram, respectively. In the case of a symmetric histogram, SK = 0 and the less peaked the histogram the greater value of K. 1.2 Looking Ahead The most important drawback of descriptive statistics is that the study of the observed data enables us to draw certain conclusions which relate only to the 2
data in hand. The temptation in analyzing the above income data is to at tempt to make generalizations beyond the data in hand, in particular about the distribution of income in the US (not just 23000 households in US). This, how- ever, is not possible in the descriptive statistics framework. In order to be able to generalize beyond the data in hand we need to model"the distribution of income in the US and not just describe the observed data in hand. Such a general model is provided by probability theory to be considered in Section 2 It turns out that the model provided by probability theory owns a lot to the earlier developed descriptive statistics. In particular, most of the concepts which form the basis of the probability theory were motivated by the descriptive statistic concept considered above. The concepts of measures of location, dispersion and shape, as well as the frequency curve, were transplanted into probability theory with renewed interpretations. The frequency curve when reinterpreted becomes a density function purporting to model observable real world phenomena, As for the various measures, they will now be reinterpreted in terms of the density function
data in hand. The temptation in analyzing the above income data is to attempt to make generalizations beyond the data in hand, in particular about the distribution of income in the US (not just 23000 households in US). This, however, is not possible in the descriptive statistics framework. In order to be able to generalize beyond the data in hand we need to ”model” the distribution of income in the US and not just describe the observed data in hand. Such a general model is provided by probability theory to be considered in Section 2. It turns out that the model provided by probability theory owns a lot to the earlier developed descriptive statistics. In particular, most of the concepts which form the basis of the probability theory were motivated by the descriptive statistic concept considered above. The concepts of measures of location, dispersion and shape, as well as the frequency curve, were transplanted into probability theory with renewed interpretations. The frequency curve when reinterpreted becomes a density function purporting to model observable real world phenomena, As for the various measures, they will now be reinterpreted in terms of the density function. 3
2 Probability Why we need the probability theory in analyzing observed data? In the descrip- tive study of data considered in the last section, it is emphasized that the result cannot be generalized outside the observed data under consideration. Any ques- tion relating to the population from which the observed data were from cannot be answered within the descriptive statistics framework. In order to be able to do that we need the theoretical framework offered by probability theory. In ef- fect probability theory develops a mathematical model which provides the logical foundation of statistical inference procedures for analyzing observed dat In developing a mathematical model we must first identify the important fea- tures, relations and entities in the real world phenomena and then devise the concepts and choose the assumptions with which to project a generalized de- scription of there phenomena; an idealized pictures of these phenomena. The model as a consistent mathematical system has" a life of its own"and can be analyzed and studied without direct reference to real world phenomena.(Thinks of analyzing the population, we do not have to refer to the information in the sample By the 1920s there was a wealth of results and probability began to grow into a systematic body of knowledge. Although various people attempted a systemati zation of probability it was the work of the Russian mathematician Kolmogorov hich provided to be the cornerstone for a systematic approach to probability theory. Kolmogorov managed to relate the concept of the probability to that of a measure in integration theory and exploited to the full the analogies between set theory and the theory of functions on the one hand and the concept of random variable on the other. In a monumental monograph in 1933 he proposed an axiomatization of probability theory establishing it once and for all as part of mathematical proper. There is no doubt that this monograph provided to be the watershed for the later development of probability theory growing enormously in importance and applicability
2 Probability Why we need the probability theory in analyzing observed data ? In the descriptive study of data considered in the last section, it is emphasized that the results cannot be generalized outside the observed data under consideration. Any question relating to the population from which the observed data were from cannot be answered within the descriptive statistics framework. In order to be able to do that we need the theoretical framework offered by probability theory. In effect probability theory develops a mathematical model which provides the logical foundation of statistical inference procedures for analyzing observed data. In developing a mathematical model we must first identify the important features, relations and entities in the real world phenomena and then devise the concepts and choose the assumptions with which to project a generalized description of there phenomena; an idealized pictures of these phenomena. The model as a consistent mathematical system has ”a life of its own” and can be analyzed and studied without direct reference to real world phenomena. (Thinks of analyzing the population, we do not have to refer to the information in the sample.) By the 1920s there was a wealth of results and probability began to grow into a systematic body of knowledge. Although various people attempted a systematization of probability it was the work of the Russian mathematician Kolmogorov which provided to be the cornerstone for a systematic approach to probability theory. Kolmogorov managed to relate the concept of the probability to that of a measure in integration theory and exploited to the full the analogies between set theory and the theory of functions on the one hand and the concept of a random variable on the other. In a monumental monograph in 1933 he proposed an axiomatization of probability theory establishing it once and for all as part of mathematical proper. There is no doubt that this monograph provided to be the watershed for the later development of probability theory growing enormously in importance and applicability. 4
2.1 The Axiomatic Approach The axiomatic approach to probability proceeds from a set of axioms(accepted without questioning as obvious), which are based on many centuries of human experience, and the subsequent development is built deductively using formal logical arguments, like any other part of mathematics such as geometry or linear algebra. In mathematics an axiomatic system is required to be complete, non redundant and consistent. By complete we mean that the set of axioms postu- lated should enables us to prove every other theorem in the theory in question using the axioms and mathematical logic. The notion of non-redundancy refers to the impossibility of deriving any axiom of the system from the other axioms Consistency refers to the non- contradictory nature of the axioms a probability model is by construction intended to be a description of a chand mechanism giving rise to observed data. The starting point of such a model is provided by the concept of a random experiment describing a simplistic and idealized process giving rise to observed data. The starting point of such a model is provided by the concept of a random erperiment describing a simplistic and idealized process giving rise to the observed data Definition 1 A random experiment, denoted by &, is an experiment which satisfies the fol- wing conditions (a) all possible distinct outcomes are known a priori (b) in any particular trial the outcomes is not known a priori; and (c)it can be repeated under identical conditions The axiomatic approach to probability theory can be viewed as a formalization of the concept of a random experiment. In an attempt to formalize condition(a) all possible distinct outcome are known a priori, Kolmogorov devised the set s which included "all possible distinct outcome" and has to be postulated before the experiment is performed Definition 2 The sample space, denoted by S, is defined to be the set of all possible outcome of the experiment 8. The elements of S are called elementary events
2.1 The Axiomatic Approach The axiomatic approach to probability proceeds from a set of axioms (accepted without questioning as obvious), which are based on many centuries of human experience, and the subsequent development is built deductively using formal logical arguments, like any other part of mathematics such as geometry or linear algebra. In mathematics an axiomatic system is required to be complete, non − redundant and consistent. By complete we mean that the set of axioms postulated should enables us to prove every other theorem in the theory in question using the axioms and mathematical logic. The notion of non-redundancy refers to the impossibility of deriving any axiom of the system from the other axioms. Consistency refers to the non- contradictory nature of the axioms. A probability model is by construction intended to be a description of a chance mechanism giving rise to observed data. The starting point of such a model is provided by the concept of a random experiment describing a simplistic and idealized process giving rise to observed data. The starting point of such a model is provided by the concept of a random experiment describing a simplistic and idealized process giving rise to the observed data. Definition 1: A random experiment, denoted by E, is an experiment which satisfies the following conditions: (a) all possible distinct outcomes are known a priori; (b) in any particular trial the outcomes is not known a priori; and (c) it can be repeated under identical conditions. The axiomatic approach to probability theory can be viewed as a formalization of the concept of a random experiment. In an attempt to formalize condition (a) all possible distinct outcome are known a priori, Kolmogorov devised the set S which included ”all possible distinct outcome” and has to be postulated before the experiment is performed. Definition 2: The sample space, denoted by S, is defined to be the set of all possible outcome of the experiment E. The elements of S are called elementary events. 5