Markov Chains and Stochastic Processes
CSIR NET Mathematical Sciences Unit 4: Markov chains, classification of states, stationary distributions, limits of P^n, Poisson and birth-death processes.
- 3
- Topics
- 61
- Min read
- 01 Markov Chains and Classification of StatesStart here CSIR NET statistics: discrete-time Markov chains on finite and countable spaces, Chapman–Kolmogorov, communicating classes, recurrence, period, absorption.
- 02 n-Step Transition Limits and Stationary Distributions CSIR NET statistics: stationary distributions, mean recurrence times, the convergence theorem for P^n, periodic and reducible chains, detailed balance.
- 03 Poisson and Birth-and-Death Processes CSIR NET statistics: Poisson process properties, continuous-time Markov chains, Kolmogorov equations, birth-and-death processes, Yule and M/M/1 queues.
- Markov Chains and Stochastic Processes Formula Sheet and Short Notes Revise The whole chapter on one page.
Chapter Overview
Markov chains are the part of Unit 4 (statistics) of the CSIR NET Mathematical Sciences syllabus where probability meets linear algebra. A chain is a stochastic matrix; its \( n \)-step behaviour is the matrix power \( P^n \); its long-run behaviour is a left eigenvector for the eigenvalue 1. The continuous-time half of the chapter, the Poisson process and birth-and-death processes, replaces the matrix by a generator and the powers by an exponential, and gives the standard models of queues and populations.
The notes assume an MSc first course in probability and a first look at stochastic processes (texts such as Ross’s Introduction to Probability Models, Karlin and Taylor’s A First Course in Stochastic Processes, Norris’s Markov Chains or Feller’s An Introduction to Probability Theory and Its Applications, Volume 1). They do not re-prove the limit theorems. They collect the exact statements, the shortcuts that turn a \( 3 \times 3 \) system into two lines, and the counter-examples (periodic, reducible and countable chains) that decide multiple-correct Part C questions.
What You’ll Learn
By the end of this chapter you will be able to:
- Write the transition matrix of a verbal model and compute \( n \)-step probabilities with Chapman–Kolmogorov
- Split any finite chain into communicating classes and decide, in seconds, which are closed, recurrent or transient, and their periods
- Decide recurrence of countable chains (random walks on \( \mathbb{Z} \) and \( \mathbb{Z}^d \), reflecting walks) from \( \sum_n p_{ii}^{(n)} \)
- Compute absorption probabilities and expected absorption times by first-step analysis or the fundamental matrix
- Find stationary distributions with the two-state formula, double stochasticity, detailed balance or the degree formula, and convert them to mean recurrence times
- State exactly when \( p_{ij}^{(n)} \) converges, to what, and what happens for periodic and reducible chains
- Use the Poisson process (exponential gaps, conditioning on \( N(t) \), superposition, thinning) and solve birth-and-death models: Yule, linear birth–death, M/M/1, M/M/\( \infty \)
Prerequisites
- Basic probability: conditional probability, discrete and continuous distributions (Poisson, binomial, geometric, exponential, gamma). Part A’s Counting, Probability and Number Properties is a warm-up.
- Eigenvalues and the Cayley–Hamilton Theorem and diagonalisation, for \( P^n \).
- Convergence of series from Real Analysis, for the recurrence criterion.
Learning Path
graph TD
A["1. Markov Chains and Classification of States"] --> B["2. n-Step Transition Limits and Stationary Distributions"]
A --> C["3. Poisson and Birth-and-Death Processes"]
B --> C
B --> D["Quick Revision"]
C --> DTopics in This Chapter
| # | Topic | What it covers | Time |
|---|---|---|---|
| 1 | Markov Chains and Classification of States | Markov property, transition matrices, Chapman–Kolmogorov, communicating and closed classes, recurrence and transience, random walks on \( \mathbb{Z}^d \), period, absorption, gambler’s ruin | 40 min |
| 2 | n-Step Transition Limits and Stationary Distributions | stationary distributions and mean recurrence times, detailed balance, random walks on graphs, birth–death chains, the convergence theorem, periodic and reducible limits, \( P^n \) in closed form | 40 min |
| 3 | Poisson and Birth-and-Death Processes | Poisson process and its properties, continuous-time chains and Kolmogorov equations, birth-and-death processes, Yule, linear birth–death, M/M/1 and other queues | 40 min |
| Quick Revision | every formula and counter-example on one page | 10 min |
Official syllabus lines and where they are covered
| Official topic line | Covered in |
|---|---|
| Markov chains (finite/countable), classification of states | Markov Chains and Classification of States (finite chains, random walks and countable chains); reflecting walks and birth–death chains on the non-negative integers in n-Step Transition Limits and Stationary Distributions |
| n-step transition limits, stationary distribution | n-Step Transition Limits and Stationary Distributions; Chapman–Kolmogorov and \( P^n \) are introduced in Markov Chains and Classification of States |
| Poisson and birth-and-death processes | Poisson and Birth-and-Death Processes |
How CSIR NET Asks It
Per the NTA information bulletin for December 2026, the Mathematical Sciences paper (code 704) is marked as follows:
| Part B | Part C | |
|---|---|---|
| Questions given | 40 | 60 |
| Questions to attempt | 25 | 20 |
| Marks per correct answer | 3 | 4.75 |
| Marks per wrong answer | −0.75 | 0 |
| Maximum | 75 | 95 |
NTA publishes no topic-wise weightage, so we make no claim about how many questions come from this chapter. What the format tells you is how to prepare:
- Part B (single correct) rewards fast computation: the number of classes of a given matrix, a period, an absorption probability, a stationary distribution, a mean recurrence time, \( \lim p_{ij}^{(n)} \), a Poisson or conditional binomial probability, an M/M/1 measure.
- Part C (one or more correct) rewards structure plus counter-examples. A typical stem fixes a chain (a random walk, a reducible matrix, a birth–death model) and lists four statements about recurrence, periodicity, uniqueness of \( \pi \), convergence of \( P^n \) or extinction. Each is either forced by a theorem or broken by a small chain. Every topic page ends with Level 3 practice in exactly this format, with every option justified.
For the marking scheme and the other units, see the Mathematical Sciences overview. The matrix side of this chapter (eigenvalues, diagonalisation) is in Linear Algebra; the ODE side of Kolmogorov’s equations is in Linear ODEs and Variation of Parameters. Part A (General Aptitude) is common to all subjects. Found an error? See About for how to report it.
Start Learning
Begin with the first topic: