Does the pooled ranking hold within each batch?
Compare the conditions within each batch before interpreting the pooled result.
The question
A discovery team compares two conditions under one response criterion. The pooled summary leaves out batch labels. Both batches are valid and each observation represents a distinct sample.
| Condition A | Condition B | |
|---|---|---|
| Observations meeting the criterion | 49 | 33 |
| Total observations | 70 | 60 |
The response criterion stays the same across the two batches.
“Which condition appears ahead, and what would you check before recommending a next study?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Read the pooled result
What would you do here? Open the move you’d make.
Report the pooled rates and ask for the batch counts.
The strong move. The pooled rates can give different weight to batches for each condition.
Use 70% against 55% to conclude A performs better.
Tempting, but it misses something. The pooled summary does not show how the batch mix differs by condition.
See how a strong candidate says it
“The pooled rates are 70% for A and 55% for B. I’d check how the observations were distributed before interpreting that gap.”
Why it works: The pooled rates can give different weight to batches for each condition.
Step 2: Ask for the batch split
What would you do here? Open the move you’d make.
Request the numerator and denominator for each condition in each batch.
The strong move. Both counts are needed to compare the same response criterion within a batch.
Ask for the two total batch sizes without condition counts.
Tempting, but it misses something. Batch size alone gives no response rate for either condition.
See how a strong candidate says it
“I’d ask the analyst for the number that met the criterion and the total, by condition within each batch.”
Why it works: Both counts are needed to compare the same response criterion within a batch.
Step 3: Compare both batches
The analyst restores the labels. Batch 1 has A at 45 of 50 and B at 19 of 20. Batch 2 has A at 4 of 20 and B at 14 of 40.
What would you do here? Open the move you’d make.
Show both within-batch comparisons and explain the unequal mix.
The strong move. The conditions have different weights across the two batches. The pooled ranking reverses the within-batch comparisons.
Keep A first because 49 exceeds 33.
Tempting, but it misses something. The raw counts have different denominators and different batch mixes.
See how a strong candidate says it
“In batch 1, A is at 90% and B is at 95%. In batch 2, A is at 20% and B is at 35%. B is higher in both. A has more observations in the higher-response batch.”
Why it works: The conditions have different weights across the two batches. The pooled ranking reverses the within-batch comparisons.
Step 4: Give the next check
What would you do here? Open the move you’d make.
Report the batch-specific rates and ask for a comparison that handles batch.
The strong move. The team sees why the pooled ranking is misleading and what evidence would sharpen the comparison.
Claim the within-batch rates establish that B causes a stronger response.
Tempting, but it misses something. These rates describe the observed samples. A causal claim needs evidence from the study design and analysis.
See how a strong candidate says it
“I’d show both comparisons and the unequal mix. I’d ask the study owner how to compare the conditions with balanced batches or an analysis that accounts for the batch structure.”
Why it works: The team sees why the pooled ranking is misleading and what evidence would sharpen the comparison.
The answer, in short
- Decision
- Report the results within each batch and qualify the pooled ranking.
- Because
- The conditions contribute different numbers of observations to the higher-response batch.
- What would change it
- A comparison that handles the batch structure would give stronger evidence for the next study.
Common traps
- Combining groups with different batch mixes.
- Inferring a cause from descriptive rates.
Check your answer
- I checked both pooled rates.
- I asked for the batch split.
- I found the reversal.
- I gave a next comparison.
Last reviewed Nov 11, 2026