Practise building and evaluating experiments through random assignment, comparison, replication, blocking, matching, control, and defensible causal conclusions.
Researchers are investigating the effectiveness of using a fungus to control the spread of an insect that destroys trees. The researchers will create four different concentrations of fungus mixtures: 0 milliliters per liter (ml/L), 1.25 ml / L, 2.5 ml / L, and 3.75 ml / L. An equal number of the insects will be placed into 20 individual containers. The group of insects in each container will be sprayed with one of the four mixtures, and the researchers will record the number of insects that are still alive in each container one week after spraying.
Scoring
Parts (a), (b), and (c) are each scored as essentially correct (E), partially correct (P), or incorrect (I).
Part (a) is scored as follows:
Essentially correct (E) if the response satisfies the following three components:
Identifies the 4 concentrations (or mixtures or sprays) as the treatments
Identifies the 20 containers as the experimental units
Identifies the number of insects that are still alive in each container as the response variable
Partially correct (P) if response satisfies only two of the three components.
Incorrect (I) if the response does not meet the criteria for E or P.
Notes:
- Listing the four treatments satisfies component 1 (including ml/L is not required). However, if the list does not include all four treatments, component 1 is not satisfied.
- To satisfy component 1, the response must refer to plural concentrations/mixtures/sprays (e.g., the mixtures, the levels of the concentration). Referring only to the explanatory variable (concentration) does not satisfy component 1.
- The following responses satisfy component 2: "the 20 containers";
"the containers";
"the 20 groups of insects";
or "the groups of insects in each container." References to only "groups of insects" do not satisfy component 2 because it is unclear if these groups are formed by treatment or by container.
- To satisfy component 3, it must be clear that the response variable is being measured separately for each experimental unit. A response that says only "number of insects alive" does not satisfy component 3 because it could be referring to the total number of insects alive.
- To satisfy component 3, the response must be stated as a variable by using "number of" or equivalent. For example, "insects alive in each container" is not a variable and would not satisfy component 3.
- If the response states that the insects are the experimental units, then component 3 can still be satisfied by providing a binary response variable for each insect (e.g., whether the insect lived or died, survival status).
Part (b) is scored as follows:
Essentially correct (E) if the response indicates that there is a control group and justifies this claim by identifying the control group or by explaining that there is a treatment which contains no fungus.
Partially correct (P) if the response indicates that there is no control group because every container is sprayed with some mixture
OR
if the response states that there is a control group but implies that 0 ml / L is not a treatment (e.g., "the containers with 0 ml/L form a control group because they don't receive a treatment";
"yes, there is a group that got no treatment").
Incorrect (I) if the response does not meet the criteria for E or P.
Notes:
- The response does not need to explain the purpose of a control group.
- The response does not need to explicitly say "yes"-it can be implied by stating that there is a control group or saying "the control group is...."
Part (c) is scored as follows:
Essentially correct (E) if the response satisfies the following three components:
Creates appropriate labels for the units/treatments (e.g., label the containers from 1 through 20, label 20 slips of paper with five for each treatment)
Describes how to correctly implement the random assignment process
The random assignment process results in an equal number of experimental units assigned to each treatment
Partially correct (P) if response satisfies only two of the three components.
Incorrect (I) if the response does not meet the criteria for E or P.
Notes:
- If the response states that insects are the experimental units in part (a), the response in part (c) can be in terms of insects or containers. In either case, the same three components are used to determine the score.
- If the response states that the containers are the experimental units in part (a), but only describes how to assign insects to treatments in part (c), component 1 is not satisfied.
- For responses that use slips of paper:
If the number of slips of paper is not equal to the number of experimental units, then component 1 is not satisfied. The slips of paper do not need to be specifically identified as equally-sized.
If the slips of paper are not mixed/shuffled or the slips are not "selected at random," component 2 is not satisfied. Sampling without replacement is implied when using slips of paper, unless the response specifies sampling with replacement.
- For responses that use random number generators (or a 20-sided die):
If the initial assignment of numbers to units does not give each unit the same probability of being assigned to each treatment (e.g., units are represented by different numbers of integers), then component 1 is not satisfied.
If the response does not indicate that the numbers are selected without replacement or that different numbers must be used, the response does not satisfy component 2. The response does not need to specify the interval of numbers from which they are selecting (e.g., randomly generate a number from 1 to 20 ).
- For responses that use a table of random digits:
If the initial assignment of numbers to units does not give each unit the same probability of being assigned to each treatment, component 1 is not satisfied. For example, responses that use the labels 1 to 20 (not 01 to 20) do not satisfy component 1 because label 1 has a 101 probability of being selected but label 20 has a 1001 probability of being selected.
If the response does not indicate that the numbers are selected without replacement or that different numbers must be used, the response does not satisfy component 2. The response does not need to specify the interval of numbers from which they are selecting or state that the numbers corresponding to unused labels will be skipped (e.g., skip numbers 00 and 21 to 99).
- For responses that use a 4-sided die (or random integers from 1 to 4):
If the die is rolled for each experimental unit, then component 3 is not satisfied because an equal number of units per treatment is not guaranteed.
If the die is rolled for each experimental unit until treatments are "full," then component 1 is not satisfied because this setup doesn't allow for all possible random assignments to be equally likely (unless the order of the units is randomized initially).
- If a response groups the experimental units before any random assignment (e.g., forms five groups of four containers or four groups of five containers), and then randomly assigns treatments to the groups or randomly assigns treatments within each group, component 1 is not satisfied. However, if a response forms groups in the context of a randomized block design with a reasonable blocking variable, component 1 can be satisfied.
- If a response describes two different random assignment processes in detail (e.g., how to randomly assign insects to containers and how to assign containers to treatments), both descriptions are scored according to the three components and the lower score is used.
- Responses that assign experimental units only to groups and not to treatments (e.g., randomly select five containers and put them in group 1) do not satisfy component 3.
- If the response randomly assigns insects to containers, the containers must be assigned to a treatment to satisfy component 3. In this case, the assignment of treatment to container does not need to be at random to satisfy component 3.
Complete Response
Three parts essentially correct
Substantial Response
Two parts essentially correct and one part partially correct
Developing Response
Two parts essentially correct and no parts partially correct
OR
One part essentially correct and one or two parts partially correct
OR
Three parts partially correct
Minimal Response
One part essentially correct
OR
No parts essentially correct and two parts partially correct
Does the experiment have a control group? Explain your answer.
Part (b):
Yes. Because the 0 ml/L concentration contains no fungus, the containers that are sprayed with the 0 ml / L concentration form the control group.
Describe how the treatments can be randomly assigned to the experimental units so that each treatment has the same number of units.
Part (c):
Label each container with a unique integer from 1 to 20. Then use a random number generator to choose 15 integers from 1 to 20 without replacement. Use the first five of these numbers to identify the five containers that will receive the 0 ml / L treatment. Use the second five of these numbers to identify the five containers that will receive the 1.25 ml / L treatment. Use the third five of these numbers to identify the five containers that will receive the 2.5 ml / L treatment. The remaining five containers will receive the 3.75 ml / L treatment.
(Alternative solution) Using 20 equally sized slips of paper, label five slips with 0 ml/L, five slips with 1.25 ml/L, five slips with 2.5 ml/L, and five slips with 3.75 ml/L. Mix the slips of paper in a hat. For each container, select a slip of paper from the hat (without replacement) and spray that container with the treatment selected.
A car maker produces four different models of cars: A, B, C, and D. A group of researchers is investigating which model of car has the longest distance traveled per gallon of gas (mileage). Higher mileage is considered better than lower mileage. The researchers will conduct a study in which they contact several owners of each model of car and ask them to estimate their mileage.
Model D has an autopilot feature, in which the car controls its own motion with human supervision. James owns a Model D car and will investigate whether using the autopilot feature results in higher mileage than not using the autopilot. James will drive his car on 70 different days to and from work, using the same route at the same time each day. James will record the mileage each day.
James will use a completely randomized design to conduct his investigation. Describe an appropriate method James could use to randomly assign the two treatments, driving using the autopilot feature and driving without using the autopilot feature, to 35 days each.
Number the days in the experiment from 1 to 70. Using a random number generator, generate 35 unique integers from 1 to 70, inclusive. Assign the days with those 35 unique integers for James to drive the car with autopilot and assign the remaining 35 days for James to drive the car without autopilot.
(Alternative solution)
Using 70 equally sized slips of paper, label 35 "with autopilot" and 35 "without autopilot." Mix the slips of paper in a bag. Each day for the 70 days, select a slip of paper (without replacement) to determine the driving method for that day.
Scoring
Essentially correct (E) if the response satisfies the following three components:
Creates appropriate labels for the units/treatments
Describes how to correctly implement the random process so that every possible random assignment is equally likely
The response indicates a random process that results in 35 days assigned to using autopilot and 35 days assigned to not using autopilot
Partially correct (P) if the response satisfies only two of the three components required for E.
Incorrect (I) if the response does not meet the criteria for E or P.
Additional Notes:
- For responses that use slips of paper (or marbles or equivalent) to represent treatments
To satisfy component 1 some slips must be labeled or assigned to represent autopilot and some labeled or assigned to represent no autopilot (e.g., blue marbles represent autopilot and yellow marbles represent no autopilot).
To satisfy component 2 the slips of paper must be mixed/shuffled, and the response must clearly link the treatment selected to a day (e.g., each day James selects a slip to determine the driving method).
To satisfy component 3 the response must indicate that there are 35 slips (or marbles of a specific color) for each treatment and the response must indicate that slips of paper (marbles) are selected without replacement.
- For responses that use slips of paper labeled from 1 to 70 (or an equivalent interval)
To satisfy component 1 the days must be labeled 1 to 70.
To satisfy component 2 the slips of paper must be mixed/shuffled, and the response must clearly link the day number selected to autopilot or no autopilot.
To satisfy component 3 the response must indicate that slips of paper are selected without replacement and that 35 days are assigned to each treatment.
- For responses that use a random number generator with days labeled from 1 to 70 (or an equivalent interval)
To satisfy component 1 the days must be labeled 1 to 70.
To satisfy component 2 the response indicates that the random number generator selects numbers 1 to 70 inclusive, and the response clearly links the day number selected to autopilot or no autopilot.
To satisfy component 3 the response must indicate that numbers are selected without repeats and that 35 days are assigned to each treatment.
- For responses that flip a coin for each day (or roll a die and note odd/even, generate a random number 1 or 2, or equivalent)
To satisfy component 1 each outcome must be linked to a treatment (e.g., heads equals autopilot, tails equals no autopilot).
To satisfy component 2 the response must clearly link the outcome of the coin flip to a day (e.g., each day James flips a coin to determine the driving method). Note: If the response includes a stopping rule (e.g., when 35 days are assigned one treatment, the remaining days are assigned the other treatment), component 2 is not satisfied because this plan increases the probability that the last days will have the same treatment, which does not meet the equally likely random assignment requirement.
To satisfy component 3 the response must indicate that 35 days are assigned to each treatment using a stopping rule. If there is no stopping rule, component 3 is not satisfied.
- If a response uses a random number generator or slips of paper with the numbers 1 to 70 and does not initially number the days from 1 to 70, component 1 may be satisfied if the response indicates a link between the number selected and the day (e.g., if 3 is selected, James uses autopilot on the third day).
- Responses that do not use any random process should be scored I. For example, "number the days from 1 to 70 and use autopilot on odd-numbered days."
- Responses that use blocking do not satisfy component 2 because all possible random assignments are not equally likely.
- If the response describes two ways to perform the random assignment, assign the score for the weaker assignment process.
A dermatologist will conduct an experiment to investigate the effectiveness of a new drug to treat acne. The dermatologist has recruited 36 pairs of identical twins. Each person in the experiment has acne and each person in the experiment will receive either the new drug or a placebo. After each person in the experiment uses either the new drug or the placebo for 2 weeks, the dermatologist will evaluate the improvement in acne severity for each person on a scale from 0 (no improvement) to 100 (complete cure).
Each twin in the experiment has a severity of acne similar to that of the other twin. However, the severity of acne differs from one twin pair to another.
| Scoring for Question 2 | Score |
|---|---|
| Complete Response | 4 |
| Three parts essentially correct | |
| Substantial Response | 3 |
| Two parts essentially correct and one part partially correct | |
| Developing Response | 2 |
| Two parts essentially correct and no part partially correct OR | |
| One part essentially correct and one or two parts partially correct OR | |
| Minimal Response | 1 |
| One part essentially correct and no part partially correct OR | |
| No part essentially correct and two parts partially correct |
For the dermatologist's experiment, describe a statistical advantage of using a matched-pairs design where twins are paired rather than using a completely randomized design.
Improvement scores will vary due to many factors, including initial acne severity, what treatment is received, and other variables such as diet and genetics. Because the pairs of twins are similar in initial acne severity, pairing allows for the variation in improvement scores due to the treatment received to be distinguished from variation due to initial acne severity, unlike in a completely randomized design. Consequently, using the matched-pairs design will provide a more precise estimate of the mean difference in improvement in acne severity for the new drug compared to the placebo and make it easier to find convincing evidence that the new drug is better, if it really is better.
Scoring
Essentially correct (E) if the response describes a statistical advantage of a matched-pairs design AND satisfies the following three components:
The advantage pertains to an inference made after collecting the data (e.g., the ability to distinguish between the effects of the treatments or the precision of the estimate of the drug effect)
Indicates that the matched-pairs design is better by using a comparative word (e.g., easier, clearer, greater) or by making an explicit comparison to a completely randomized design
Includes context (e.g., "drug," "improvement," "acne," or "twins")
Partially correct (P) if the response describes a statistical advantage of a matched-pairs design AND satisfies one or two of the three components.
Incorrect (I) if the response does not satisfy the criteria for E or P.
Additional Notes:
- To be considered an advantage of a matched-pairs design, the advantage described must be true for a matched-pairs design and not be true for a completely randomized design. For example, saying that "random assignment allows us to conclude cause-and-effect" is true of both designs. Similarly, "this allows the dermatologist to make conclusions about people with differing acne severity" is true of both designs. Also, "reduces bias" and "reduces variability in the estimates of the individual treatment means" is true of neither design.
- Responses that describe only the set-up of a matched-pairs experiment do not satisfy the requirement to describe an advantage of a matched-pairs design. For example, the response "in a matched-pairs design, the members of each pair will be similar in terms of acne severity" does not describe an advantage. However, "in a matched-pairs design, we can compare two people with similar acne severity" does describe an advantage.
- Advantages of a matched-pairs design that satisfy component 1 include "makes it easier to determine if the drug is effective," "gives a better estimate of the effect of the new drug," "reduces variability in the estimate of the drug effect," "makes the difference between the drug and the placebo more easily distinguishable," and "gives a clearer picture of how well the drug works."
- Advantages of a matched-pairs design that don't satisfy component 1 include "accounts for a source of variability," "controls for potentially confounding variables," "allows you to distinguish variation due to severity from variation due to treatment," "each person can be compared to someone similar," "reduces variability," "more balanced treatment groups," and "more accurate results."
- It is acceptable to provide a disadvantage of a completely randomized design rather than an advantage of the matched-pairs design (e.g., "The completely randomized design will make it harder to find convincing evidence that the new drug is better").
- It is acceptable to use the term "blocking" as a synonym for "pairing."
- A response that states that a matched-pairs design requires a smaller sample size to get power or precision equal to that in a completely randomized design and describes this advantage in context should be scored E.
For the dermatologist's experiment, describe how the treatments can be randomly assigned to people using a matched-pairs design in which twins are paired.
For each pair of twins, label one person as twin A and label the other person as twin B. For each pair of twins, toss a coin. If the coin lands on heads, twin A gets the placebo and twin B gets the active drug. If the coin lands on tails, twin A gets the active drug and twin B gets the placebo.
OR
Label the members of each pair of twins as "Twin 1" and "Twin 2." Using a random number generator, generate an integer from 1 to 2. Give the drug to the twin whose number is selected and the placebo to the twin whose number is not selected. Repeat for all pairs of twins.
OR
Label 1 notecard "A" and another notecard "B." For each pair of twins, shuffle the cards and give one card to each twin. The twin who gets "A" receives the drug and the twin who gets "B" receives the placebo.
Scoring
Essentially correct (E) if the response randomly assigns the two treatments within pairs of twins AND satisfies the following three components:
Uses a random process (e.g., flipping a coin, using a random number generator, shuffling cards) that gives each twin in a pair a 50\% probability of getting the drug and a 50\% probability of getting the placebo
Describes how to use the random process to assign one specific twin in each pair to the drug and the other twin to the placebo
Indicates that the random assignment process will be completed for each pair of twins
Partially correct (P) if the response randomly assigns the two treatments within pairs of twins AND satisfies only two of the three components for E.
Incorrect (I) if the response does not satisfy the criteria for E or P.
Additional Notes:
- A response that does not randomly assign both treatments within pairs of twins should be scored incorrect (I). Examples include a response that describes a completely randomized design, describes a crossover design where each person receives both treatments, uses pairs other than twins, does not use random assignment, or indicates that both twins in a pair receive the same treatment.
- For responses that use slips of paper or selecting items from a hat, the slips must be shuffled (or blindly drawn) or the hat mixed or shaken to have a random process and satisfy component 1.
- To satisfy component 2, the response must describe what to do for each possible outcome of the random process and specify which treatment each twin receives. For example, none of the following descriptions satisfy component 2:
"Roll a die. If it is 1-3, give the first twin the drug and the second twin the placebo." (Response doesn't describe what to do if the die is 4-6.)
"Have one member of each pair flip a coin. If it is heads, that twin gets the drug. If it is tails, that twin gets the placebo." (Response doesn't indicate what treatment the other twin will receive.)
"Flip a coin. If it is heads, give one twin the drug and the other twin the placebo. If it is tails, do the reverse." (Response doesn't specify which twin is getting the drug.)
"Label one slip of paper "A" and a second slip of paper "B." Mix them in a hat and have each member of the pair choose one slip." (Response doesn't specify if A represents the new drug or the placebo.)
- Ignore any discussion about randomly selecting 36 pairs of twins to obtain subjects for the experiment. Likewise, ignore any discussion about how to perform the analysis for a paired design (e.g., "subtract the improvement scores for each pair of twins").
- It is acceptable to refer to each pair of twins as a block.