Experimental Design, Sampling and Data Analysis
CSIR NET Part A research aptitude: controls, replication, randomisation, blinding, sampling, bias, accuracy vs precision, p-values and test errors.
- 22 min
On this page · 14 sections
Prerequisites
- Variables, hypotheses and study types from Research Process, Hypotheses and Scientific Reasoning.
- Averages and weighted averages from Percentages, Ratios, Averages and Mixtures.
- Probability rules from Counting, Probability and Number Properties, used for test errors and diagnostic tests.
The Hook: Three Lanes on One Blot
Controls
A control is a condition that differs from the test condition in only the factor being studied, or that checks the method itself. It is what lets you attribute a difference to the treatment.
| Control | What it receives | What it rules out | Example |
|---|---|---|---|
| Negative control | No treatment, or a treatment known to have no effect | A positive result arising from contamination or background | No-template control in PCR |
| Positive control | A treatment known to give the effect | A negative result arising because the assay failed | A known antibiotic in a growth-inhibition assay |
| Vehicle (solvent) control | The solvent or carrier alone, same volume | An effect of the solvent | DMSO-only wells when the drug is dissolved in DMSO |
| Sham control | The procedure without the active step | An effect of handling or surgery itself | Sham-operated animals in a lesion study |
| Placebo control | An inert preparation that looks identical | Effects of expectation (the placebo effect) | Sugar pill in a drug trial |
| Loading or internal control | A reference measured in the same sample | Differences in the amount of material | β-actin band, or a reference gene in qPCR |
| Wild-type or isogenic control | The same strain without the mutation | Differences in genetic background | Parental strain beside a knockout |
The Principles of Experimental Design
Three principles, associated with R. A. Fisher’s work on agricultural field trials, are the backbone of design:
| Principle | What it means | What it protects against |
|---|---|---|
| Replication | Applying each treatment to several independent experimental units | Mistaking chance variation for an effect; it lets you estimate error |
| Randomisation | Assigning units to treatments by a random process | Systematic differences between groups (confounding, selection by the researcher) |
| Local control (blocking) | Grouping similar units into blocks and comparing treatments within each block | Known nuisance variation inflating the error |
Biological vs technical replicates
| Biological replicate | Technical replicate | |
|---|---|---|
| What is repeated | Independent biological samples (separate animals, plants, cultures grown separately) | Repeated measurements of the same sample |
| What it measures | Biological variation, which a conclusion about the population needs | Measurement (assay) variation |
| Supports a claim about | The biology | The precision of the method |
The experimental unit is the smallest unit to which a treatment is independently applied. If a drug is added to a flask and you count 100 cells from that flask, n is 1 flask, not 100 cells. Treating sub-samples as independent replicates is called pseudoreplication, and it makes an effect look far more certain than it is.
Common designs
| Design | Layout | When to use |
|---|---|---|
| Completely randomised design (CRD) | Every unit assigned at random to any treatment | Units are uniform (e.g. a growth chamber) |
| Randomised block design (RBD) | Units grouped into blocks of similar units; each block gets every treatment once, at random | A known gradient (field slope, batch, day) |
| Latin square | Treatments arranged so each appears once in every row and every column | Two nuisance gradients at right angles |
| Factorial design | Every combination of the levels of two or more factors | To find interactions between factors |
| Paired / before–after / crossover | Each subject serves as its own control | Large differences between subjects |
Blinding
| Level | Who does not know the allocation |
|---|---|
| Single-blind | The subjects |
| Double-blind | The subjects and the people giving the treatment or assessing the outcome |
| Triple-blind | The above and the people analysing the data (usage varies between sources) |
In laboratory work, “blind scoring” means the person counting cells or scoring phenotypes does not know which sample is which, usually by coding the samples. It protects against observer bias.
Effects that mimic a treatment effect
| Effect | What happens | Defence |
|---|---|---|
| Placebo effect | Subjects improve because they expect to | Placebo control, blinding |
| Hawthorne effect | Subjects change behaviour because they know they are being observed | A control group observed in the same way |
| Regression to the mean | Units chosen because they were extreme tend to be less extreme when measured again, with no treatment at all | A randomised control group selected the same way |
| Observer (experimenter) bias | The researcher’s expectations shift the measurement | Blinding, objective measures |
Population and Sampling
The population is the whole set the conclusion is meant to apply to; the sample is the part actually studied; the sampling frame is the list from which the sample is drawn. A parameter describes the population (for example \( \mu \), \( \sigma \)); a statistic describes the sample (\( \bar{x} \), \( s \)) and is used to estimate the parameter.
Sampling methods
graph TD
A["Sampling methods"] --> B["Probability sampling
every unit has a known,
non-zero chance"]
A --> C["Non-probability sampling
chance of selection unknown"]
B --> B1["Simple random"]
B --> B2["Systematic
every k-th unit"]
B --> B3["Stratified
random within subgroups"]
B --> B4["Cluster
random whole groups"]
B --> B5["Multistage"]
C --> C1["Convenience"]
C --> C2["Purposive / judgement"]
C --> C3["Quota"]
C --> C4["Snowball"]| Method | How it works | Strength | Weakness |
|---|---|---|---|
| Simple random | Every possible sample of size n is equally likely (lottery, random numbers) | Unbiased; basis of most statistics | Needs a complete list; may miss small subgroups by chance |
| Systematic | Random start, then every k-th unit, \( k = N/n \) | Easy in the field | Biased if the list has a cycle matching k |
| Stratified | Divide into non-overlapping strata (e.g. by year of study); random sample within each | Ensures every subgroup is represented; more precise | Needs stratum information in advance |
| Cluster | Divide into clusters (e.g. villages); randomly select whole clusters | Cheap when the population is spread out | Less precise than a simple random sample of the same size |
| Multistage | Random clusters, then random units within them | Practical for large surveys | Complex analysis |
| Convenience | Whoever is easy to reach | Quick | Usually unrepresentative |
| Purposive | Researcher picks typical or information-rich cases | Useful in qualitative work | Depends on the researcher’s judgement |
| Quota | Fill fixed numbers per subgroup, by convenience | Matches population proportions on chosen traits | Not random within subgroups |
| Snowball | Participants recruit further participants | Reaches hidden or rare populations | Networks over-represented |
Sample size and sampling error
Sampling error is the difference between a statistic and the parameter that arises only because a sample, not the whole population, was measured. It shrinks as the sample grows, roughly in proportion to \( \frac{1}{\sqrt{n}} \). Non-sampling errors (a biased frame, non-response, faulty instruments) do not shrink with a bigger sample.
Types of bias
| Bias | Source | Example |
|---|---|---|
| Selection bias | The sample is not representative because of how it was chosen | Studying only patients who reached a tertiary hospital |
| Non-response bias | Those who reply differ from those who do not | A stress survey answered mainly by the least busy |
| Survivorship bias | Only the units that “survived” a filter are seen | Judging a treatment only from patients who lived long enough to be seen in clinic |
| Recall bias | Subjects remember past exposures differently depending on the outcome | Case–control studies of diet |
| Measurement / instrument bias | The instrument reads consistently wrong | An uncalibrated balance |
| Confirmation bias | Looking for, or weighting, evidence that fits the hypothesis | Repeating only the experiments that “failed” |
| Publication bias | Positive results are more likely to be published | Published effect sizes exaggerated |
Measurement
Scales of measurement
The four scales usually attributed to S. S. Stevens:
| Scale | Properties | Examples | Meaningful statistics |
|---|---|---|---|
| Nominal | Categories, no order | Blood group, species, genotype | Counts, mode, chi-square |
| Ordinal | Ordered categories, unequal or unknown gaps | Disease grade I–IV, Likert scale, rank | Median, percentiles, rank tests |
| Interval | Equal gaps, arbitrary zero | Temperature in °C, calendar year | Mean, SD; differences but not ratios |
| Ratio | Equal gaps, true zero | Mass, length, concentration, temperature in K, counts | All, including ratios (“twice as much”) |
20 °C is not “twice as hot” as 10 °C, because 0 °C is not an absence of heat. 20 K is twice 10 K.
Accuracy, precision, errors
| Term | Meaning |
|---|---|
| Accuracy | Closeness of a measurement to the true value |
| Precision | Closeness of repeated measurements to each other |
| Systematic error | Pushes every reading the same way; reduces accuracy; not reduced by repeating; found by calibration against a standard |
| Random error | Scatters readings both ways; reduces precision; its effect on the mean is reduced by repeating |
| Resolution | The smallest change the instrument can show |
Picture shots at a target:
| Tightly grouped (precise) | Widely scattered (imprecise) | |
|---|---|---|
| Centred on the bull’s-eye (accurate) | Accurate and precise | Accurate on average, not precise |
| Off-centre (inaccurate) | Precise, not accurate (systematic error) | Neither |
Reliability and validity
| Reliability | Validity | |
|---|---|---|
| Question | Does the method give consistent results? | Does it measure what it claims to measure? |
| Analogy | Precision | Accuracy |
| Kinds | Test–retest, inter-rater, internal consistency | Internal (is the causal conclusion sound within the study?), external (does it generalise?), construct, content |
A measure can be reliable but not valid (a scale that always reads 2 kg high); it cannot be valid without being reasonably reliable.
Reading the Numbers
Centre and spread
| Statistic | Definition | Note |
|---|---|---|
| Mean | \( \bar{x} = \frac{\sum x_i}{n} \) | Pulled by outliers |
| Median | Middle value of the ordered data | Robust to outliers; use for skewed data |
| Mode | Most frequent value | The only one for nominal data |
| Range | Max − min | Depends only on two values |
| Variance | \( s^2 = \frac{\sum (x_i - \bar{x})^2}{n-1} \) for a sample | Divide by \( n \) for a whole population |
| Standard deviation (SD) | \( s = \sqrt{s^2} \) | Same units as the data; describes spread of the data |
| Coefficient of variation | \( \text{CV} = \frac{s}{\bar{x}}\times 100\% \) | Compares spread across different scales |
| Standard error of the mean (SEM) | \( \text{SE} = \frac{s}{\sqrt{n}} \) | Describes uncertainty in the mean, not spread of the data |
In a right-skewed distribution (a long tail of large values, like income or many cell sizes), mean > median > mode.
For roughly normal data, about 68%, 95% and 99.7% of values lie within 1, 2 and 3 SD of the mean. An approximate 95% confidence interval for the mean is \( \bar{x} \pm 1.96\,\text{SE} \), often rounded to \( \pm 2\,\text{SE} \).
Correlation
The Pearson correlation coefficient \( r \) lies between −1 and +1 and measures the strength of a linear association. \( r = 0 \) means no linear association; a strong curved (for example U-shaped) relation can still have \( r \approx 0 \). The square \( r^2 \) (coefficient of determination) is the fraction of the variation in one variable accounted for by a linear relation with the other: \( r = 0.7 \) gives \( r^2 = 0.49 \), about half.
Simpson’s paradox
A trend that holds in every subgroup can reverse when the subgroups are pooled, if the groups are of very different sizes and a confounder is unevenly spread.
Hypothesis tests and the p-value
A statistical test asks how surprising the data would be if \( H_0 \) were true.
- The p-value is the probability, assuming \( H_0 \) is true, of getting a result at least as extreme as the one observed.
- The significance level \( \alpha \) (commonly 0.05) is the threshold chosen in advance. If \( p < \alpha \), \( H_0 \) is rejected.
| \( H_0 \) actually true | \( H_0 \) actually false | |
|---|---|---|
| Reject \( H_0 \) | Type I error (false positive); probability \( \alpha \) | Correct; probability = power \( = 1 - \beta \) |
| Fail to reject \( H_0 \) | Correct | Type II error (false negative); probability \( \beta \) |
Power rises with a larger sample, a larger true effect, less variability and a larger \( \alpha \).
Multiple testing
If you run many tests, some will come out “significant” by chance alone. With \( m \) independent tests, all of whose nulls are true, at level \( \alpha \):
$$\text{Expected false positives} = m\alpha \qquad P(\text{at least one}) = 1 - (1-\alpha)^m$$With 20 tests at 0.05: on average 1 false positive, and \( 1 - 0.95^{20} \approx 0.64 \), a 64% chance of at least one. The Bonferroni correction uses \( \alpha/m \) for each test (here \( 0.05/20 = 0.0025 \)). Running many analyses and reporting only the significant ones is called p-hacking or data dredging.
Diagnostic and screening tests
| Condition present | Condition absent | |
|---|---|---|
| Test positive | True positive (TP) | False positive (FP) |
| Test negative | False negative (FN) | True negative (TN) |
| Measure | Formula | Answers |
|---|---|---|
| Sensitivity | \( \frac{TP}{TP + FN} \) | Of those with the condition, what fraction test positive? |
| Specificity | \( \frac{TN}{TN + FP} \) | Of those without it, what fraction test negative? |
| Positive predictive value (PPV) | \( \frac{TP}{TP + FP} \) | If I test positive, how likely am I to have it? |
| Negative predictive value (NPV) | \( \frac{TN}{TN + FN} \) | If I test negative, how likely am I to be free of it? |
Sensitivity and specificity are properties of the test. PPV and NPV also depend on the prevalence: the rarer the condition, the lower the PPV, even for a good test.
Memory Tricks
Don’t Confuse
| This | vs | That |
|---|---|---|
| Biological replicate: independent sample | Technical replicate: same sample measured again | |
| Negative control: checks for false positives | Positive control: checks the assay can detect an effect | |
| Accuracy: near the true value | Precision: near each other | |
| Systematic error: one direction, not removed by repeating | Random error: both directions, averages out | |
| Stratified: random units from every stratum | Quota: fill numbers per group by convenience | |
| SD: spread of observations | SE: uncertainty of the mean | |
| Sensitivity: P(test + given condition) | PPV: P(condition given test +) | |
| Statistical significance | Practical importance |
Common Mistakes
Practice
All questions below are original practice questions written for this page. They are not from any CSIR NET paper.
Level 1: Foundation
- A PCR reaction with all reagents but no template DNA is what kind of control?
- On which scale of measurement is a disease graded as mild, moderate or severe?
- A sample has SD 10 and n = 25. What is the standard error of the mean?
Answers
- Negative control (no-template control). A band here means contamination.
- Ordinal. The categories are ordered, but the gaps between them are not known to be equal.
- 2. \( \text{SE} = \frac{10}{\sqrt{25}} = \frac{10}{5} = 2 \).
Level 2: Part A Style
Practice Q1. A researcher dissolves a plant extract in DMSO and finds that it kills 60% of cancer cells in culture, compared with 5% death in cells grown in medium alone. The most important missing control is:
(a) cells treated with a known anticancer drug (b) cells treated with the same volume of DMSO alone (c) a second batch of the plant extract (d) normal (non-cancer) cells in medium alone
Answer: (b)
DMSO can itself be toxic to cells. Without a DMSO-only control, the 55-point difference could be due to the solvent. (a) is a positive control; useful, but this result is already positive, so it does not explain it. (d) answers a different question (selectivity).
Practice Q2. A survey team lists all 240 villages in a district, randomly selects 12 villages, and interviews every household in those 12 villages. This is:
(a) stratified sampling (b) cluster sampling (c) systematic sampling (d) quota sampling
Answer: (b) cluster sampling
Whole groups (villages) are chosen at random and every unit in them is studied: “every from some”. In stratified sampling, some households would be chosen from every village.
Practice Q3. A balance gives readings of 10.31, 10.30, 10.32 and 10.31 g for a certified 10.00 g mass. The balance is:
(a) accurate and precise (b) precise but not accurate (c) accurate but not precise (d) neither
Answer: (b)
The readings agree within 0.02 g (precise) but all are about 0.31 g high (a systematic error, so not accurate). Calibration, not more readings, is the fix.
Practice Q4. A research group runs 40 independent tests, in each of which the null hypothesis is actually true, at \( \alpha = 0.05 \). The expected number of “significant” results is:
(a) 0 (b) 1 (c) 2 (d) 4
Answer: (c) 2
Expected false positives \( = m\alpha = 40 \times 0.05 = 2 \). The probability of at least one is \( 1 - 0.95^{40} \approx 0.87 \). A Bonferroni threshold would be \( 0.05/40 = 0.00125 \) per test.
Level 3: Multi-Step (Part C depth)
Practice Q5. A study compares four fertiliser doses on wheat. The field slopes from north to south, and soil moisture changes with the slope. The researcher divides the field into five east–west strips, and within each strip assigns the four doses at random to four plots.
- This is a randomised block design with 5 blocks.
- The strips are the experimental units.
- The design removes the effect of the slope from the comparison of doses.
- There are 20 plots, so each dose is replicated 5 times.
Which statements are correct?
(a) 1 and 4 only (b) 1, 3 and 4 only (c) 2 and 3 only (d) 1, 2, 3 and 4
Answer: (b) 1, 3 and 4 only
- 1 is correct. Each strip (block) contains every treatment once, assigned at random.
- 2 is wrong. The plot is the unit to which a dose is applied; the strip is a block.
- 3 is correct. Moisture is roughly constant within an east–west strip, so comparing doses within strips removes the slope gradient from the comparison (and the analysis separates block variation from error).
- 4 is correct. \( 5 \times 4 = 20 \) plots; each dose appears once per block, 5 times in all.
Practice Q6. A screening test for a condition has sensitivity 90% and specificity 95%. The condition affects 1% of the population screened. If a randomly chosen person tests positive, the probability that they actually have the condition is closest to:
(a) 90% (b) 50% (c) 15% (d) 5%
Answer: (c) about 15%
Take 10,000 people. With the condition: 100. Without: 9,900.
- True positives: \( 0.90 \times 100 = 90 \).
- False positives: \( 0.05 \times 9{,}900 = 495 \).
- PPV \( = \frac{90}{90 + 495} = \frac{90}{585} \approx 0.154 \), about 15%.
Most positives are false because the condition is rare. Option (a) is the sensitivity, the classic trap. A confirmatory test on the positives is the usual remedy.
Practice Q7. Two labs measure the same enzyme’s activity (true value unknown) in five independent preparations each, in µmol/min/mg:
- Lab P: 4, 8, 6, 5, 7
- Lab Q: 6.1, 5.9, 6.0, 6.2, 5.8
(i) Find the mean and the sample SD for each lab. (ii) Which lab is more precise? (iii) Can you say which lab is more accurate?
Answer
(i) Lab P: mean \( = \frac{30}{5} = 6 \). Deviations −2, 2, 0, −1, 1; squares sum to \( 4 + 4 + 0 + 1 + 1 = 10 \). Sample variance \( = \frac{10}{4} = 2.5 \), SD \( = \sqrt{2.5} \approx 1.58 \).
Lab Q: mean \( = \frac{30.0}{5} = 6.0 \). Deviations 0.1, −0.1, 0, 0.2, −0.2; squares sum to \( 0.01 + 0.01 + 0 + 0.04 + 0.04 = 0.10 \). Variance \( = \frac{0.10}{4} = 0.025 \), SD \( \approx 0.158 \).
(ii) Lab Q is far more precise (SD ten times smaller; CV about 2.6% vs 26%).
(iii) No. Accuracy needs a known true value or a reference standard. Both means are 6.0, but they could share the same systematic error.
Quick Revision Table
| Idea | Key point |
|---|---|
| Negative / positive control | Rules out false positives / false negatives |
| Vehicle control | Same solvent, same volume, no active agent |
| Fisher’s principles | Replication, randomisation, local control (blocking) |
| Experimental unit | Smallest unit independently given a treatment; sub-samples are not replicates |
| Factorial | Number of combinations = product of the levels; reveals interactions |
| Stratified vs cluster | Some from every group vs every unit from some groups |
| Systematic | Every k-th unit, \( k = N/n \) |
| Scales | Nominal, ordinal, interval (arbitrary zero), ratio (true zero) |
| Accuracy vs precision | Near true value vs near each other; systematic vs random error |
| SE | \( s/\sqrt{n} \); quadruple n to halve it |
| p-value | P(data this extreme or more | \( H_0 \) true) |
| Type I / II | False positive (\( \alpha \)) / false negative (\( \beta \)); power \( = 1 - \beta \) |
| Multiple tests | Expected false positives \( m\alpha \); Bonferroni \( \alpha/m \) |
| PPV | Falls as prevalence falls |
Key Takeaways
What’s Next
- Research Ethics, Publication and Scientific Communication: what happens after the data, from authorship to peer review and citation metrics.
- Research Process, Hypotheses and Scientific Reasoning if variables and study types are not yet automatic.
- Research Aptitude: Quick Revision.
- The deeper statistics (estimation, tests, regression) are part of the Mathematical Sciences syllabus.