MA/STAT 519: Introduction to Probability Theory
Spring 2018, Purdue University

http://www.math.purdue.edu/~yip/519

Course Description:

Algebra of sets, sample spaces, combinatorial problems, conditional probability and independence, discrete and continuous random variables, distribution and joint distribution functions, expectation of random variables, moment generating functions, limit theorems and applications.

Instructor:

Aaron Nung Kwan Yip
Department of Mathematics
Purdue University

Contact Information:

Office: MATH 432
Email: click here

Lecture Time and Place:

T, Th 10:30 - 11:45, UNIV 117

Office Hours:

Tue, 4:30-5:30pm, Wed, 3:00-4:00pm, or by appointment.

Textbook:

(All of the following are either available online or on reserve in the math library.)

Main Text:
[CD] Probability with Applications in Engineering, Science, and Technology,
by Matthew A. Carlton, Jay L. Devore, 2nd edition (available online from Purdue library page)

Reference:
[R] A First Course in Probability, by Sheldon Ross (I am using the 9th edition, but older editions are OK for reading and reference)
[HPS] Introduction to Probability Theory, by Hoel, Port and Stone
[D1] Fundamentals of Probability: a First Course, by Anirban DasGupta (available online from Purdue library page)
[D2] Probability for Statistics and Machine Learning: Fundamentals and Advanced Topics, by Anirban DasGupta (available online from Purdue library page)
[F1, F2] An Introduction to Probability Theory and Its Applications, Volume 1 and 2, by William Feller

Course Outline:

The course will cover most of the sections of [CD] Chapters 1 to 4, and part of Chapter 5.

Prerequisites:

Good "working" knowledge of:
linear algebra (e.g. MA 265, 351, 511),
vector calculus (e.g. MA 261, 362, 510),
and mathematical analysis (e.g. MA 341, 440, 504).

Homework:

Homeworks will be assigned roughly bi-weekly. They will be gradually assigned as the course progresses. Please refer to the course announcement below.

  • Steps must be shown to explain your answers. No credit will be given for just writing down the answers, even if it is correct.

  • Please staple all loose sheets of your homework to prevent 5% penalty.

  • Please resolve any error in the grading (hws and tests) WINTHIN ONE WEEK after the return of each homework and exam.

  • No late homeworks are accepted (in principle).

  • You are encouraged to discuss the homework problems with your classmates but all your handed-in homeworks must be your own work.
  • Examinations:

    Midterm Test: Date to be determined
    Final Exam: During Final Exam Week

    Grading Policy:

    Homeworks (50%)
    Midterm Test (20%)
    Final Exam (30%)

    You are expected to observe academic honesty to the highest standard. Any form of cheating will automatically lead to an F grade, plus any other disciplinary action, deemed appropriate.

    Accommodations for Students with Disabilities and Academic Adjustment:

    University policy will be followed.

    Course Progress and Announcement:

    (You should consult this section regularly, for homework assignments, additional materials and announcements.)


    WEEK 1

    Jan 9 (Tue): [CD 1.1-1.2][R 2.1-2.4][D1, Chapter 1]
    Concept of statistical regularity, Law of Large Numbers (LLN), Central Limit Theorem (CLT),
    Probability space, outcome, events, logical operations between events, complement, unions, intersections, de Morgan's Law.

    Jan 11 (Thur): [CD 1.1-1.2][R 2.1-2.4][D1, Chapter 1]
    Probability measure, countably-additivity,
    inclusion-exclusion principle [R p. 30, Proposition 4.4][D1, 1.5],
    objective vs subjective approaches to probability measures.


    WEEK 2

    Jan 16 (Tue):
    Some theoretical remarks about probability measures: countably-addivitiy, (non-)measureable sets, typical outcomes,
    equally-likely-outcome probability space;
    [CD 1.3][R 1.1-1.4][D1 1.4] counting techniques: permutations and combinations.

    Jan 18 (Thur) [R 1.1-1.6]:
    binomial and multinomial coefficients and expansions, number of integer solutions.

    Homework 1, due Thursday, Jan 25, in class


    WEEK 3

    Jan 23 (Tue): [D1 6.5, Chapter 2][R 2.5][CD 1.3.3, 2.6.1]
    Sampling (partition, hypergeometric distribition), birthday and matching problems,
    asymptotics of formula when some number (e.g. population size) is large.

    Jan 25 (Thur):
    [D1 Chapter 15] Urn and occupancy problems, Maxwell-Boltzmann, Bose-Einstein and Fermi-Dirac Distributions;
    [R 3.1-3.3][D1 Chapter 3][CD, 1.4]
    Conditional probability, Bayes' forward (formula of total probability) and backward (updating prior) formula,
    prior and posterior probabilities.


    WEEK 4

    Jan 30 (Tue):
    (cont.) Bayes' backward and forward formula, prior and posterior probabilities, sensitivity and specificity of medical tests.
    [R, 3.1-3.4][D1 Chapter 3][CD 1.5] independence between two, three, and multiple events.

    Feb 1 (Thur):
    [R, 3.1-3.4][D1 Chapter 3][CD 1.5] (cont.) (jointly) indepedence between multiple events.
    "Probability paradoxes": Monty Hall

    Homework 2, due Thursday, Feb 8, in class


    WEEK 5

    Feb 6 (Tue):
    "Probability paradoxes": (Mr. Smith's) Boy vs Girl.
    computation of probability by iteration/recursion/induction,
    an example [R, e.g. 5d, p. 93]: matching problem.

    Feb 8 (Thur):
    iterations/recursion/induction, Gambler's Ruin Problem [R, p. 84],
    trials with random (unknown) parameters, conditional independence.


    WEEK 6

    Feb 13 (Tue):
    [R 4.1-4.6][D1 Chapter 4][DC Chapter 2.1-2.4]
    (Discrete) Random variables (numerical observations of experimental outcomes),
    probability mass (density) function (pmf, pdf) and cumulative density function (cdf), expectation, variance, Cauchy-Schwarz Inequality,
    Bernoulli, Binomial, and Poisson random variables.
    Note on random variables and expectation.

    Feb 15 (Thur):
    Variance of Binomial and Poisson random variables
    Analytical computation of expectations and variance of Binomial and Poisson random variables
    [R Chapter 4/6.2-6.3/7.2-7.3] Joint random variables
    Joint (bivariate and multivariate) random variables, joint pdf, marginal pdfs,
    independent random variables, computations of Ef(X,Y), E(X+Y), E(XY)

    An example of uncorrelated but dependent random variables.

    Homework 3, due Thursday, Feb 22, in class


    WEEK 7

    Feb 20 (Tue):
    use of indicator functions to compute expectations and variances, applied to, Binomial r.v., matching, exclusion-inclusion principle.
    geometric random variable-arrival time of first success, negative Binomial random variables-arrival time of k-th success

    Feb 22 (Thur):
    reinterpretation of negative Binomial as sum of geometric r.v. (interarrival times), convolution between discrete pdfs,
    Hypergeometric random variable.


    WEEK 8

    Feb 27 (Tue):
    convergence of Hypergeometric to Binomial random variables,
    Poisson random variables, as a result of a large number of weakly dependent Bernoulli trials each with very small success probability,
    convergence of Binomial random variable to Poisson random variable.
    Two properties of Poisson random variables: merging and splitting of Poisson variables.

    Mar 1 (Thur):
    [R p. 140][D1 p. 110] "Poisson Paradigm", application of Poisson approximation to weakly dependent examples: hat matching problem, birthday problem,
    Le Cam Theorem on error estimate for Poisson approximation [D1, p. 110],
    A review paper on Poisson approximation by Serfling (1978),
    Continuum limit of Binomial random variables, and first success time: converge of geometric r.v. to exponential r.v..

    Homework 4, due Thursday, Mar 8, in class


    WEEK 9

    Mar 6 (Tue):
    Converge of negative Binomial r.v. to Gamma r.v..
    Note on negative hypergeometric random variables.
    [R, Chapter 5][D1, Chapter 7] Continuous random variables, probability density function (pdf) f(x), and cumulative density function (cdf) F(x).
    Note on continuous random variables, their pdfs and cdfs.

    Mar 8 (Thur):
    Change of variable formula for pdf of continuous random variables (one dimensional case), Y = aX + b, Y = g(X), Y = X^2,
    expectation of continuous random variables.


    WEEK 10

    Midterm, Thursday Mar 22, in class
    (review on Mar 20)
    Materials covered: Ross Chapters 1-4, Homeworks 1-4.
    A past exam
    No electronic devices. A one page (two-sided, 8x11) formula sheet is allowed.

    Solution of midterm
    Statistics of midterm:
    65 <= A; no. of students (out of 29): 7;
    40 <= B <= 59; no. of students: 17;
    C <= 39; no. of students: 5;


    WEEK 11

    Mar 27 (Tue):
    Examples of continuous random variables: uniform, Bertrand's paradox (check it out), exponential, gamma distributions,
    their relation to discrete random variables, recurrence relation for Gamma function.
    Note on examples of continuous random variables.

    Mar 29 (Thur):
    memoryless property of geometric and exponential random variables and their equivalence,
    Beta random variables, their applications in Bayesian estimate in coin tossing with random (unknown) probability of success.

    Homework 5, due Thursday, Apr 5, in class


    WEEK 12

    Apr 3 (Tue):
    Normal (Gaussian) random variables, standard normal (Z, N(0,1)), "error" function,
    some calculus formula concerning integration w.r.t. normal random variables, general Guassian integration,
    chi^2-distribution with 1-degree of freedom (Y=Z^2), linear change of variable of N(0,1), N(mu, sigma^2)
    use of normal table(s), percentages of A, B, C, D, F in a "normal grade distribution".

    Apr 5 (Thur):
    [R, Chatper 8] Limit Theorems:
    Law of Large Numbers (LLN) (law of the average) vs Central Limit Theorems (CLT) (fluctuation around the average),
    proof of the Weak LLN, Chebychev and Markov inequalities, graphical representation in terms of binomial r.v.,
    statement of CLT and graphical repressentation in terms of binomial r.v.,
    compare and contrast LLN and CLT.


    WEEK 13

    Apr 10 (Tue):
    Applications of CLT: approximations of Bin using normal, estimation of p in Bin, Hypothesis testing.
    Note on Applications of CLT.

    Apr 12 (Thur):
    Proof of CLT for Binomials (Laplace-DeMoivre Theorem).
    [R 7.7] Moment generating functions (MGF) (Laplace transform of random variables).
    [R 8.3] Proof of CLT Using MGF

    Homework 6, due Thursday, Apr 19, in class


    WEEK 14

    Apr 17 (Tue):
    [R 7.7] Moment generating functions (MGF) (Laplace transform of random variables).
    [R 8.3] Proof of CLT Using MGF
    [R, 6.1, 6.2, 6.4, 6.5] Joint cdfs, pdfs, and independence, and conditional pdfs

    Apr 19 (Thur):
    [R, 6.7] change of variable formula of joint pdfs, Cartesian to polar coordinates,
    [R, 6.3] sums of independent random variables, convolution of pdfs (discrete and continuous).


    WEEK 15

    Apr 24 (Tue):
    [R, 6.3] sums of independent random variables,
    convolution of pdfs (discrete and continuous), relation to moment generating function, infinite divisibility.
    Note on sums of independent random variables,

    [R, Sec 6.5, p. 253] Bi-variate normal random variables, rho=correlation coefficient,
    [R, Sec 7.8, p. 345] Multi-variate normal random variables,
    Note on bivariate normal random variables.
    Note on multivariate normal random variables.

    Homework 7, Practice Problems, no need to hand in.


    Final Exam: Wed, May 2nd, 8am-10am, BRWN 1154

    Materials covered: comprehensive, all homeworks and lecture materials.
    No electronic devices. One page (two-sided, 8x11) of formula sheet is allowed.
    A past final
    Office hours: Tuesday, May 1, 2-4pm

    Solution of final