Sample Space, Independence and Bayes' Theorem
CSIR NET probability: sample spaces, sigma-fields, Kolmogorov axioms, inclusion-exclusion, conditional probability, independence and Bayes' theorem.
- 3 min
On this page · 12 sections
Prerequisites
- Counting from Combinatorics and Counting or, at a lighter level, Counting, Probability and Number Properties. Classical probability is counting with a denominator.
- Set operations, countable unions and limits of sets from Real Numbers, Sets and Countability.
- Lebesgue measure from Riemann, Improper and Lebesgue Integration. A probability is a measure of total mass 1, and “geometric probability” is Lebesgue measure on a region.
The Hook: The Test That Is 95% Accurate
Sample Space, Events and the Axioms
A random experiment has a set \( \Omega \) of possible outcomes, the sample space. Events are subsets of \( \Omega \), but when \( \Omega \) is uncountable not every subset can be assigned a probability consistently, so we fix a collection \( \mathcal{F} \) of events.
\( \sigma \)-field. \( \mathcal{F} \subseteq \mathcal{P}(\Omega) \) is a \( \sigma \)-field if (i) \( \Omega \in \mathcal{F} \), (ii) \( A \in \mathcal{F} \Rightarrow A^c \in \mathcal{F} \), (iii) \( A_1, A_2, \dots \in \mathcal{F} \Rightarrow \bigcup_n A_n \in \mathcal{F} \). It is then closed under countable intersections, differences and limsup/liminf of sequences. The Borel \( \sigma \)-field on \( \mathbb{R} \) is the smallest one containing all intervals.
Kolmogorov’s axioms. A probability \( P : \mathcal{F} \to [0, 1] \) satisfies \( P(\Omega) = 1 \) and countable additivity: for pairwise disjoint \( A_1, A_2, \dots \), \( P\left(\bigcup A_n\right) = \sum P(A_n) \). The triple \( (\Omega, \mathcal{F}, P) \) is a probability space.
Consequences of the axioms
| Rule | Statement |
|---|---|
| Complement | \( P(A^c) = 1 - P(A) \); \( P(\emptyset) = 0 \) |
| Monotonicity | \( A \subseteq B \Rightarrow P(A) \le P(B) \), and \( P(B \setminus A) = P(B) - P(A) \) |
| Addition | \( P(A \cup B) = P(A) + P(B) - P(A \cap B) \) |
| Boole (union bound) | \( P\left(\bigcup A_n\right) \le \sum P(A_n) \) |
| Bonferroni | \( P\left(\bigcap_{i=1}^{n} A_i\right) \ge \sum P(A_i) - (n - 1) \) |
| Continuity | \( A_n \uparrow A \Rightarrow P(A_n) \uparrow P(A) \); \( A_n \downarrow A \Rightarrow P(A_n) \downarrow P(A) \) |
Inclusion–exclusion.
$$P\left(\bigcup_{i=1}^{n} A_i\right) = \sum_i P(A_i) - \sum_{iThe probability that exactly \( m \) of the \( n \) events occur is \( \sum_{k=m}^{n} (-1)^{k-m}\binom{k}{m} S_k \), where \( S_k \) is the \( k \)-th inclusion–exclusion sum.
Three ways to assign probabilities
| Model | Rule | Typical question |
|---|---|---|
| Classical | \( P(A) = \frac{\lvert A \rvert}{\lvert \Omega \rvert} \), equally likely outcomes | dice, cards, committees, arrangements |
| Geometric | \( P(A) = \frac{\text{measure}(A)}{\text{measure}(\Omega)} \) | random points on a segment, square, disc |
| Axiomatic / given | any \( P \) obeying the axioms | discrete models, data-based models |
Three classical results worth carrying:
- Birthday problem. With 365 equally likely birthdays, the probability that \( n \) people all differ is \( \prod_{k=0}^{n-1}\left(1 - \frac{k}{365}\right) \); it drops below \( \frac{1}{2} \) at \( n = 23 \).
- Broken stick. Break a unit stick at two uniformly chosen points. The three pieces form a triangle (each piece \( < \frac{1}{2} \)) with probability \( \frac{1}{4} \).
- Buffon’s needle. A needle of length \( l \) dropped on parallel lines at spacing \( d \ge l \) crosses a line with probability \( \frac{2l}{\pi d} \).
Conditional Probability
For \( P(B) > 0 \),
$$\boxed{P(A \mid B) = \frac{P(A \cap B)}{P(B)}}$$For fixed \( B \), \( A \mapsto P(A \mid B) \) is itself a probability on \( \mathcal{F} \), so every rule above holds conditionally: \( P(A^c \mid B) = 1 - P(A \mid B) \), and so on. Conditioning does not work in the other slot: \( P(A \mid B^c) \ne 1 - P(A \mid B) \) in general.
Multiplication rule. \( P(A_1 A_2 \cdots A_n) = P(A_1)\,P(A_2 \mid A_1)\,P(A_3 \mid A_1A_2)\cdots P(A_n \mid A_1 \cdots A_{n-1}) \).
Law of total probability. If \( B_1, \dots, B_k \) partition \( \Omega \) with \( P(B_i) > 0 \),
$$P(A) = \sum_{i=1}^{k} P(A \mid B_i)\,P(B_i).$$Bayes’ theorem.
$$\boxed{P(B_j \mid A) = \frac{P(A \mid B_j)\,P(B_j)}{\sum_i P(A \mid B_i)\,P(B_i)}}$$Read it as posterior \( \propto \) likelihood \( \times \) prior. In odds form for two hypotheses:
$$\frac{P(B_1 \mid A)}{P(B_2 \mid A)} = \frac{P(A \mid B_1)}{P(A \mid B_2)} \cdot \frac{P(B_1)}{P(B_2)}.$$graph LR
S[Start] -->|P of B1| B1[Cause B1]
S -->|P of B2| B2[Cause B2]
S -->|P of B3| B3[Cause B3]
B1 -->|P of A given B1| A1[Effect A]
B2 -->|P of A given B2| A2[Effect A]
B3 -->|P of A given B3| A3[Effect A]Total probability adds the products along every path that ends in \( A \). Bayes takes one such path product and divides by that total.
Pólya’s urn: dependence without asymmetry
An urn holds \( r \) red and \( b \) black balls. Draw one, return it together with \( c \) more of the same colour, repeat. Then:
- \( P(\text{red on draw } n) = \frac{r}{r + b} \) for every \( n \) (the draws are exchangeable);
- the draws are not independent: \( P(R_1 R_2) = \frac{r}{r+b}\cdot\frac{r+c}{r+b+c} \ne \left(\frac{r}{r+b}\right)^2 \) when \( c > 0 \).
Exchangeability lets you compute \( P(R_1 \mid R_2) = P(R_2 \mid R_1) \).
Independence
Two events. \( A \) and \( B \) are independent if \( P(A \cap B) = P(A)\,P(B) \). If \( P(B) > 0 \) this says \( P(A \mid B) = P(A) \).
If \( A, B \) are independent, so are \( A, B^c \); \( A^c, B \); and \( A^c, B^c \). An event of probability 0 or 1 is independent of every event, including itself; conversely \( A \) is independent of itself iff \( P(A) \in \{0, 1\} \).
Many events. \( A_1, \dots, A_n \) are mutually independent if for every subcollection \( \{i_1 < \dots < i_k\} \), \( k \ge 2 \),
$$P(A_{i_1} \cdots A_{i_k}) = P(A_{i_1}) \cdots P(A_{i_k}).$$That is \( 2^n - n - 1 \) equations. Pairwise independence asks only for the \( \binom{n}{2} \) equations with \( k = 2 \).
Independence versus disjointness. If \( A \cap B = \emptyset \) and both have positive probability, then \( P(AB) = 0 \ne P(A)P(B) \): disjoint events with positive probability are always dependent. Disjointness is a property of sets; independence is a property of \( P \).
Independent trials. Repeating an experiment independently gives product probabilities; this is the setting of the binomial and geometric distributions and of the laws of large numbers in Inequalities, Convergence and Limit Theorems.
Conditional independence (\( P(AB \mid C) = P(A \mid C)P(B \mid C) \)) neither implies nor follows from independence. In the Pólya urn, the colours are conditionally independent given the limiting proportion but not independent; in the two-coin example above, \( A \) and \( B \) are independent but not conditionally independent given \( C \).
Memory Tricks
Don’t Confuse
| This | vs | That |
|---|---|---|
| \( P(A \mid B) \) | \( P(B \mid A) \) (Bayes converts one to the other) | |
| mutually exclusive: \( P(AB) = 0 \) | independent: \( P(AB) = P(A)P(B) \) | |
| pairwise independent | mutually independent | |
| \( P(A^c \mid B) = 1 - P(A \mid B) \) (true) | \( P(A \mid B^c) = 1 - P(A \mid B) \) (false) | |
| exchangeable draws (Pólya) | independent draws | |
| finitely additive | countably additive (the axiom) |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper. In Level 3, as in Part C of Mathematical Sciences, one or more options may be correct.
Level 1: Foundation
- \( A \) and \( B \) are independent with \( P(A) = 0.5 \), \( P(B) = 0.4 \). Find \( P(A \cup B) \).
- Can two events with positive probabilities be both mutually exclusive and independent?
- How many product conditions define mutual independence of 5 events?
Answers
- \( 0.5 + 0.4 - 0.2 = 0.7 \).
- No. Disjointness gives \( P(AB) = 0 \), independence gives \( P(AB) = P(A)P(B) > 0 \).
- \( 2^5 - 5 - 1 = 26 \).
Level 2: Part B Style
Practice Q1. A condition has prevalence 1%. A test has sensitivity 95% and false-positive rate 5%. Given a positive test, the probability of the condition is closest to:
(a) 0.95 (b) 0.50 (c) 0.16 (d) 0.05
Answer: (c)
\( \frac{0.0095}{0.0095 + 0.0495} = \frac{0.0095}{0.059} \approx 0.161 \). (a) confuses \( P(+ \mid D) \) with \( P(D \mid +) \).
Practice Q2. Box I has 3 red and 2 white balls; box II has 1 red and 4 white. A box is chosen at random and one ball is drawn; it is red. The probability that it came from box I is:
(a) \( \frac{3}{5} \) (b) \( \frac{3}{4} \) (c) \( \frac{2}{5} \) (d) \( \frac{1}{2} \)
Answer: (b)
\( P(\text{red}) = \frac{1}{2}\cdot\frac{3}{5} + \frac{1}{2}\cdot\frac{1}{5} = \frac{2}{5} \). Then \( P(\text{I} \mid \text{red}) = \frac{3/10}{2/5} = \frac{3}{4} \). (c) is \( P(\text{red}) \) itself.
Practice Q3. A fair die is rolled three times. The probability that all three faces are different is:
(a) \( \frac{1}{2} \) (b) \( \frac{5}{6} \) (c) \( \frac{5}{9} \) (d) \( \frac{4}{9} \)
Answer: (c)
\( \frac{6 \cdot 5 \cdot 4}{6^3} = \frac{120}{216} = \frac{5}{9} \).
Practice Q4. \( X \) and \( Y \) are independent and uniform on \( [0, 1] \). Then \( P(\lvert X - Y \rvert < \frac{1}{3}) \) equals:
(a) \( \frac{1}{3} \) (b) \( \frac{5}{9} \) (c) \( \frac{2}{3} \) (d) \( \frac{4}{9} \)
Answer: (b)
\( 1 - \left(\frac{2}{3}\right)^2 = \frac{5}{9} \): the complement is two right triangles with legs \( \frac{2}{3} \). (d) is the complement.
Level 3: Part C Style
Practice Q5. Let \( \Omega = \{1, 2, 3, 4\} \) with \( P(\{i\}) = \frac{1}{4} \), and \( A = \{1, 2\} \), \( B = \{1, 3\} \), \( C = \{1, 4\} \). Which statements are true?
(a) \( A \) and \( B \) are independent. (b) \( A \), \( B \), \( C \) are mutually independent. (c) \( A \) and \( B^c \) are independent. (d) \( A \) and \( B \cap C \) are independent.
Answer: (a) and (c)
- (a) True. \( P(AB) = P(\{1\}) = \frac{1}{4} = \frac{1}{2}\cdot\frac{1}{2} \).
- (b) False. \( P(ABC) = \frac{1}{4} \ne \frac{1}{8} \).
- (c) True. Complements of independent events are independent: \( P(AB^c) = P(\{2\}) = \frac{1}{4} = \frac{1}{2}\cdot\frac{1}{2} \).
- (d) False. \( B \cap C = \{1\} \subset A \), so \( P(A \cap \{1\}) = \frac{1}{4} \ne \frac{1}{2}\cdot\frac{1}{4} \). This is exactly the failure of mutual independence.
Practice Q6. An urn contains 2 red and 3 black balls. A ball is drawn, then returned together with one more ball of the same colour, and a second ball is drawn. Let \( R_i \) = red on draw \( i \). Which statements are true?
(a) \( P(R_2) = \frac{2}{5} \). (b) \( P(R_1 \mid R_2) = \frac{1}{2} \). (c) \( R_1 \) and \( R_2 \) are independent. (d) \( P(R_1 R_2) = \frac{1}{5} \).
Answer: (a), (b) and (d)
- (d) True. \( P(R_1R_2) = \frac{2}{5}\cdot\frac{3}{6} = \frac{1}{5} \).
- (a) True. \( P(R_2) = \frac{2}{5}\cdot\frac{3}{6} + \frac{3}{5}\cdot\frac{2}{6} = \frac{6 + 6}{30} = \frac{2}{5} \), the same as for the first draw.
- (b) True. \( P(R_1 \mid R_2) = \frac{1/5}{2/5} = \frac{1}{2} \).
- (c) False. \( \frac{1}{5} \ne \left(\frac{2}{5}\right)^2 = \frac{4}{25} \).
Practice Q7. Four letters are placed at random, one each, into four addressed envelopes. Which statements are true?
(a) The probability that no letter is in its correct envelope is \( \frac{3}{8} \). (b) The probability that exactly three letters are correct is 0. (c) The probability that exactly one letter is correct is \( \frac{1}{3} \). (d) The probability that at least one letter is correct exceeds \( \frac{2}{3} \).
Answer: (a), (b) and (c)
- (a) True. There are 9 derangements of 4 objects: \( \frac{9}{24} = \frac{3}{8} \).
- (b) True. Three correct forces the fourth to be correct.
- (c) True. Choose the correct letter (4 ways) and derange the other three (2 ways): \( \frac{8}{24} = \frac{1}{3} \).
- (d) False. \( 1 - \frac{3}{8} = \frac{5}{8} = 0.625 < \frac{2}{3} \).
Check: none 9, exactly one 8, exactly two \( \binom{4}{2} \cdot 1 = 6 \), exactly four 1; total 24.
Quick Revision Table
| Idea | Fact |
|---|---|
| Probability space | \( (\Omega, \mathcal{F}, P) \); \( \sigma \)-field closed under complements and countable unions |
| Axioms | \( P(\Omega) = 1 \), countable additivity |
| Inclusion–exclusion | alternating sums \( S_1 - S_2 + S_3 - \dots \); truncations are bounds |
| Matching problem | \( P(\text{no match}) = \sum_{k=0}^{n} \frac{(-1)^k}{k!} \to \frac{1}{e} \) |
| Conditional | \( P(A \mid B) = P(AB)/P(B) \), a probability in \( A \) |
| Total probability | \( P(A) = \sum P(A \mid B_i)P(B_i) \) |
| Bayes | \( P(B_j \mid A) \propto P(A \mid B_j)P(B_j) \) |
| Independence | \( P(AB) = P(A)P(B) \); passes to complements |
| Mutual, \( n \) events | \( 2^n - n - 1 \) conditions; pairwise is weaker |
| Disjoint + positive probability | dependent |
| Geometric | \( P(\lvert X - Y \rvert < a) = 1 - (1 - a)^2 \); broken stick \( \frac{1}{4} \); Buffon \( \frac{2l}{\pi d} \) |
Key Takeaways
What’s Next
- Random Variables, Distribution Functions and Moments: attaching numbers to outcomes and summarising them by \( F \), \( E \), moments and generating functions.
- Joint, Marginal and Conditional Distributions; Characteristic Functions: independence and conditioning for random variables rather than events.
- Descriptive Statistics and Exploratory Data Analysis if you skipped the data side of the chapter.
- Probability Theory: Quick Revision.