MA/STAT 519: Introduction to Probability Theory
Fall 2021, Purdue University
http://www.math.purdue.edu/~yipn/519

Course Description:

(i) Algebra of sets, sample spaces, probability measures, combinatorial problems,
(ii) conditional probability and independence,
(iii) discrete and continuous random variables,
(iv) distribution and joint distribution functions, expectation of random variables,
(v) moment generating functions,
(vi) limit theorems and applications,
(vii) conditional expectation,
(viii) Gaussian random variables.

Instructor:

Aaron Nung Kwan Yip
Department of Mathematics
Purdue University

Contact Information:

Office: MATH 432
Email and Phone: click here

Lecture Times and Places:

Section 004 (CRN 60148): T, Th 12:00 - 1:15, UNIV 117

Office Hours:

Tue: 2:00-3:15pm, Wed: 3:00-4:15pm, or by appointment.

Textbook:

Main Text:
[R] A First Course in Probability, by Sheldon Ross
(The most current version is the 10th edition. But older editions such as 8th or 9th editions are fine for reading and reference. I will send you the actual assignment sheets.)

References:
[HPS] Introduction to Probability Theory, by Hoel, Port and Stone
[D1] Fundamentals of Probability: a First Course, by Anirban DasGupta (available online from Purdue library page)
[D2] Probability for Statistics and Machine Learning: Fundamentals and Advanced Topics, by Anirban DasGupta (available online from Purdue library page)
[F1, F2] An Introduction to Probability Theory and Its Applications, Volume 1 and 2, by William Feller

Course Outline:

The course will cover most of the sections of [R] Chapters 1 to 8.

Prerequisites:

Good "working" knowledge of:
linear algebra (e.g. MA 265, 351, 511),
vector calculus (e.g. MA 261, 362, 510),
and mathematical analysis (e.g. MA 341, 440, 504).

Homework:

Homeworks will be assigned roughly bi-weekly. They will be gradually assigned as the course progresses. Please refer to the course announcement below.

Guidelines for homework:

  • Steps must be shown to explain your answers. No credit will be given for just writing down the answers, even if it is correct.

  • Please resolve any error in the grading (hws and tests) WINTHIN ONE WEEK after the return of each homework and exam.

  • No late homeworks are accepted (in principle).

  • You are encouraged to discuss the homework problems with your classmates but all your handed-in homeworks must be your own work.

    • Examinations:

    Midterm Test: Date to be determined
    Final Exam: During Final Exam Week

    Grading Policy:

    Homeworks (50%)
    Midterm Test (20%)
    Final Exam (30%)

    The following is departmental policy for the grade cut-offs:
    97% of the total points in this course are guaranteed an A+,
    93% an A,
    90% an A-,
    87% a B+,
    83% a B
    80% a B-,
    77% a C+,
    73% a C,
    70% a C-,
    67% a D+,
    63% a D, and
    60% a D-.
    For each of these grades, it's possible that at the end of the semester a lower percentage will be enough to achieve that grade.

    You are expected to observe academic honesty to the highest standard. Any form of cheating will automatically lead to an F grade, plus any other disciplinary action, deemed appropriate.

    Accommodations for Students with Disabilities and Academic Adjustment:

    Purdue University strives to make learning experiences accessible to all participants. If you anticipate or experience physical or academic barriers based on disability, you are welcome to let me know so that we can discuss options. You are also encouraged to contact the Disability Resource Center at: drc@purdue.edu or by phone at 765-494-1247.

    If you have been certified by the Disability Resource Center (DRC) as eligible for accommodations, you should contact me to discuss your accommodations as soon as possible. Click here for instructions for sending your Course Accessibility Letter to me.

    Nondiscrimination Statement:

    This class, as part of Purdue University's educational endeavor, is committed to maintaining a community which recognizes and values the inherent worth and dignity of every person; fosters tolerance, sensitivity, understanding, and mutual respect among its members; and encourages each individual to strive to reach his or her own potential.

    Protect Purdue Plan:

    Protect Purdue Plan / (Pledge)
    Essential Protect Purdue Guidance (and frequently asked questions)
    Compliance Plan: What to do in the presence of violations: Ask, Offer, Leave, Report

    Lack of compliance
    Students who are not engaging in behaviors established in the standard operating procedures (e.g., properly wearing a mask when required) will be asked to comply and offered any assistance they need in order to comply. If non-compliance continues, possible results include instructors asking students to leave the class, potentially followed by instructors dismissing the whole class. Students who do not comply with the required health and Protect Purdue Pledge behaviors are violating the University Code of Conduct and will be reported to the Dean of Students Office, with sanctions ranging from educational requirements to dismissal from the university. For additional guidance, please see the Dean of Students guidance on Managing Classroom Behavior and Expectations.

    Student rights
    Any student who has substantial reason to believe that another person in the room is threatening class safety by not wearing a face covering or following other safety guidelines for public health considerations may leave the class without consequence. The student is encouraged to report the observed behavior to the course instructor or to the Office of Student Rights and Responsibilities (OSRR), as well as discuss next steps with the instructor.

    Course Progress and Announcement:

    You should consult this section regularly, for homework assignments, additional materials and announcements.
    You can also access this page through BrightSpace.

    Week 1 (Aug 24, 25):

    [R 2.1-2.4][D1, Chapter 1]
    Concept of statistical regularity, Law of Large Numbers (LLN), Central Limit Theorem (CLT);
    Sample space, outcome, events, logical operations between events;
    Probability measure, its axioms, and properties;
    Objective (frequentist) vs subjective (Bayesian) approaches to find/assign probability measures;
    Inclusion-exclusion principle [R p. 31, Proposition 4.4][D1, 1.5];
    Bonferroni Inequality [R p. 33][D1 p. 15]

    Note: Probability space and probability measure
    Note: A simple remark about the objective (frequentist) vs subjective (degree of belief) approaches to the assignment of probabilities (Trosset, Intro. Stat. Inf.)
    Note: Another statement about frequentist vs subjective interpretation of probability [D1]

    Week 2 (Aug 31, Sept 2):

    [R 1.1-1.6, 2.5][D1 1.4, Chapter 2]
    Counting techniques: permutations and combinations,
    Binomial and multinomial coefficients.
    Number of integer solutions, matching problems.
    Equally-likely-outcome probability space.
    For your curiosity, check out:
    Maxwell-Boltzmann, Bose-Einstein, and Fermi-Diract distributions from statistical and quantum mechaincs.

    Note: Concepts and Formulas from Combinatorics

    Homework 1, due Thursday, Sept. 9, in class.

    Week 3 (Sept 7, 9):

    [R 3.1-3.3][D1 Chapter 3]
    Conditional probability,
    Multiplicative rule, Bayes' forward (Law of Total Probability) and backward (updating prior) formula,
    Prior and posterior probabilities.
    Applications of Bayes Formulas:
  • Sensitivity and Specificity of medical tests, false positive.
    Paradoxes in probability:
  • Monty Hall Problem, Box Paradox, Boy vs Girl.
    For your curiosity: search for more paradoxes in probability theory:
    envelope paradox, base rate fallacy, prosecutor's fallacy and many more.

    Note: Conditional Probability and Independence
    Note: Three bewitching paradoxes (Snell-Vanderbei)
    Note: Paradoxes (Grinstead-Snell, Probability)
    Note: Sensitivity and specificity of COVID-19 test
    Note: Some real life examples (Anderson-Seppalainen-Valko, Intro. Prob.)

    Week 4 (Sept 14, 16):

    [R, 3.1-3.4][D1 Chapter 3]
    Independence between two, three, and n events.
    Computation of probability by conditioning:
  • 5 happens before 7 [R, p.82 Example 4h].
    Computation of probability by iteration/recursion/induction:
  • matching problem [R, p.99, Example 5d]
  • Gambler's Ruin Problem [R, p.87, Example 4m]
    Trials with random (unknown) parameters,
    Conditional independence: P(AB|C) = P(A|C)P(B|C)

    Homework 2, due Thursday, Sept. 23, in class.

    Week 5 (Sept 21, 23):

    [R 4.1-4.7][D1 Chapter 4]
    (Discrete) Random variables: numerical observations of experimental outcomes,
    Probability mass (density) function (pmf, pdf) and cumulative density function (cdf),
    Hypergeometric, Bernoulli, Binomial, and Poisson random variables.
    Expectation and variance of random variables.
    [R Prop 4.1] E(X) vs E(f(X)),
    Var(X) = E(X^2) - (EX)^2; E(X^2) >= (EX)^2; Cauchy-Schwarz Inequality.

    Note: Random variables, their expectations and variances
    Note: Analytical computations of expectations and variances of Binomial and Poisson RVs
    (idea of generating functions)

    Week 6 (Sept 28, 30):

    [R Chapter 4/6.2-6.3/7.2-7.3][D1 Chapters 4, 11]
    Joint (bivariate and multivariate) random variables, joint and marginal pdfs;
    Independent random variables, joint pdf = product of marginals.
    Computations of E(X+Y), E(f(X,Y)), E(XY), Var(X+Y),
    Independence, (un)correlation, covariance, correlation coefficients.
    Use of S=X_1 + X_2 + ... + X_n and indicator functions to compute expectations and variances.
    Applied to Binomial r.v., matching number, inclusion-exclusion principle.
    Conditional Expection: Ef(X,Y) = E[E[f(X,Y)|Y]], Ef(X)g(Y) = E[g(Y)E[f(X)|Y]], Ef(X) = E[E[f(X)|Y]].
    Infinitely many identical and independent coin tossing (trials),

    Geometric random variable - arrival time of first success,
    Negative Binomial random variables - arrival time of k-th success.

    Note: Multivariate (joint) and independent random variables

    Homework 3, due Thursday, Oct. 7, in class.

    Week 7 (Oct 5, 7):

    Reinterpretation of Negative Binomial as sum of geometric r.v. (interarrival times);
    Sum of independent rvs. and convolution between discrete pdfs.
    Memoryless property of geometric random variables.

    [R 4.7][D1 6.6, 6.7] Poisson random variables, as a result of:
    a large number of weakly dependent Bernoulli trials, each with very small success probability.
    Poisson convergence/approximation: convergence of Binomial random variable to Poisson random variable.
    Two properties of Poisson random variables: merging and splitting of Poisson variables.
    [R p. 140][D1 p. 110] "Poisson Paradigm"
    Application of Poisson approximation to weakly dependent examples:
    Hat matching and birthday problems.
    Error estimate for Poisson approximation

    Note: Geometric and Negative Binomials
    Note: Analytical computations of expectations and variances of Negative Binomials
    (idea of generating functions)
    Note: Note: Poisson Random Variables - I (Basic Properties)
    Note: Note: Poisson Random Variables - II (Approximation)
    A review paper on Poisson approximation by Serfling (1978).

    Week 9 (Oct 19, 21):

    Midterm: Tuesday, Oct 19, in class.
    Materials covered: everything discrete.
    Homeworks 1-3, all lecture materials and relevant sections of [R].
    It is closed book and closed note. A one page (two-sided, 8x11) formula sheet is allowed but no electronic devices.

         Past midterm (Fall 2018)
    Past midterm (Fall 2020)

    Midterm Statistics:
    Average: 37pts/(80pts)
    A (45pts <= scores <= 80pts): No. of students = 6
    B (30pts <= scores <= 44pts): No. of students = 7
    C (scores <= 29pts): No. of students = 8
    (Note: the above cut-offs are very rough and simple cut-offs, purely based on the test scores. I have not considered the hws.)
    Midterm Solution

    [R, 4.7, p. 155-157]
    Continuum limit of Binomial random variables, and success times:
    convergence of geometric and negative binomial r.v. to exponential and gamma r.v.s.

    [R, Chapter 5][D1, Chapter 7]
    Continuous random variables
    Cumulative density function (cdf) F(x)
    Probability density function (pdf) f(x)

    Note: Continuous random variables, their pdfs and cdfs.

    Week 10 (Oct 26, 28):

    [R, Chapter 5][D1, Chapter 7]
    Change of variable formula for pdf of continuous random variables (one dimensional case),
    Y = aX + b, Y = g(X), Y = X^2,
    Expectation of continuous random variables.
    [R, Chapter 5][D1, Chapter 8]
    Examples of continuous random variables:
    Uniform, exponential, and gamma distributions, and their relation to discrete random variables,
    Recurrence relation for the Gamma function,
    Memoryless property of geometric and exponential random variables and their equivalence.

    Note: Examples of continuous random variables
    Note: Some integration property concerning Gamma function

    Homework 4, due Thursday, Nov. 4, in class.

    Week 11 (Nov 2, 4):

    [R, Chapter 5][D1, Chapters 8, 9]
    Beta random variables, their applications in Bayesian estimate in coin tossing with random (unknown) probability of success.
    Normal (Gaussian) random variables, standard normal (Z, N(0,1)),
    Some calculus formula concerning integration w.r.t. normal random variables, general Guassian integration,
    Y=Z^2: chi^2-distribution with 1-degree of freedom.

    Week 12 (Nov 9, 11):

    [R, Chatper 8][D1 Chapter 10]
    Law of Large Numbers (LLN) (law of the average) vs
    Central Limit Theorems (CLT) (fluctuation around the average).
    Compare and contrast between LLN and CLT:
    Graphical representation of LLN and CLT in terms of binomial r.v.,
    Proof of the Weak LLN: Markov and Chebychev inequalities.

    Homework 5, due Thursday, Nov. 18, in class.

    Week 13 (Nov 16, 18):

    [R, Chatper 8][D1 Chapter 10]
    Proof of the CLT (Binomial case): De Moivre-Laplace Theorem (Wiki).
    Applications of CLT:
    (1) Approximations of Bin using normal distribution;
    (2) Estimation of p in Bin and confidence interval;
    (3) Hypothesis testing.

    Note: Proof of CLT - Binomial Case
    Note: Limit Theorems and Applications of CLT
    An excellent introduction and discussion about inference and hypothesis testing (Trosset, Intro to Stat. Inf. and Its Appls., Ch. 9)

    Week 14 (Nov 23):

    [R, Chapter 6.1, 6.2]
    Joint probability distributions, marginals (for continuous rvs)
    Independent random variables: joint cdf/pdf are given by product of marginals.
    [R Chapter 6.7]
    Change of variable formula of joint pdfs, formula and computation of Jacobians.

    Note: Joint Probability Distributions, marginals and independence

    Homework 6, due Thursday, Dec. 2, in class.

    Week 15 (Nov 30, Dec 2):

    [R Chapter 6.7] (Cont.)
    Change of variable formula of joint pdfs, formula and computation of Jacobians.
    [R, 6.3]
    Sum of independent random variables
    convolution of pdfs (discrete and continuous).
    Infinite divisibility.

    Note: Transformation (change of variable formula)
    Note: Sums of independent random variables
    Note: Infinite Divisibility

    (Some materials on moment generating functions (MGF):
    Note: Proof of CLT using MGF
    Note: Infinite Divisibility using MGF.)

    Week 16 (Dec 7, 9):

    [R, Sec 6.5, p. 253]
    Multidimensional (jointly) normal random variables,
    Bi-variate normal random variables, rho=correlation coefficient
    Conditional probability distribution.

    Note: Conditional probability distribution.

    Final Exam: Tue 12/14, 7:00pm-9:00pm, UNIV 117

    Materials covered: comprehensive, all homeworks and lecture materials.
    This formula sheet (from Ross) will be provided for you.
    In addition, you can bring in one page (two-sided, 8x11) of formula sheet.
    However, no electronic devices are allowed.

    Past final (Fall 2018)
    Past final (Fall 2020)