DIF and scaling

• Goal of DIF: compare items over groups
• Requirement for DIF analysis: multigroup scaling / estimating impact

DIF and scaling

• Anchor item selection: Kopf et al. (2015)
• Recent review: Teresi et al. (2022)

DIF and scaling

• Item pairs: Bechger & Maris (2015); Yuan et al. (2021);
• Regularization: Belzak & Bauer (2020); Magis et al., (2015); Schauberger & Mair (2020)

DIF and scaling

• Scaling: He et al. (2015); He & Cui (2020); Stocking & Lord (1983)
• DIF: Halpin (2022); Wang et al. (2022)

DIF, scaling, and robust regression

• CINEG scaling via linear regression in the presence of DIF. Points represent difficulty parameters from the 2PL model, estimated in two groups The red point is an item with DIF. The scaling parameters are written as \(\mu_1\) and \(\sigma_1\). DGP = data generating parameters; LAD = least absolute deviation; OLS = ordinary least squares.

Robust scaling: Overall approach

• Implicitly define scale parameter (impact) \(\mu\) via M-estimating equation for a location parameter

\[ \Psi(\mu) = \sum_i \psi\left(\frac{Y_i - \mu}{V[Y_i]}\right) = 0\]

• \(Y_j\) is a scaling function based on parameters of item \(j = 1, \dots , J\)
• \(V[Y_i]\) is the variance of \(Y_i\) obtained via the delta method
• \(\psi\) is a “redescending” loss function chosen to flag outliers (i.e., items with DIF)
  • Use Tukey’s bi-square for computations

• Details in Halpin (2022)

Weight function (Tukey’s bisquare)

\[ w_i = \left\{\begin{array}{ccc} \left(1 - \left( \frac{z_i}{k} \right)^2\right)^2 & \text{ for } & {\mid z_i \mid } \leq k \\ 0 & \text{ for } &{\mid z_i \mid } > k \\ \end{array} \right. \]

• If item \(i\) does not have DIF, we know \(z_i = \frac{Y_i - \mu}{V[Y_i]} \sim N(0, 1)\)
• Choose per item tuning parameter \(k_i\) based on desired Type I error rate for testing DIF
• Result: \(w = 0\) if item is outside of 95% confidence interval for “no DIF”

Robust scaling: In practice

• Step 1. Maximum likelihood estimation of a focal psychometric model
  • Estimate separately in both groups, or use configural model
• Step 2. Extract model parameter estimates and their standard errors
• Step 3. Robust scaling is implemented as a post-estimation step to
  • Provide an estimate of impact that is robust to DIF
  • Flag item parameters with DIF at the desired Type I Error rate
• Step 4. Follow-up chi-square (Wald) tests for item-level DIF
• …

A simulation study

• 2PL in two groups, DIF in item difficulty only

• DIF on item difficulties (intercepts) only (\(\Delta_i = .5\))

• Impact on mean only (\(\mu_1 = .5\))

• Focal factors

  • Number of items with DIF: 0 to \(8/15\), randomly selected
  • Method: LRT-DIF, Mantel-Haenszel, GPCM lasso, proposed M estimator

• Design factors

  • \(R = 500\) reps per number of biased items
  • \(N = 500\) person per group
  • \(I = 15\) items
  • \(\theta_{0j} \sim N(0, 1)\); \(\theta_{1j} \sim N(.5, 1)\)
  • \(a_{0i} \sim U(.9, 2.5)\); \(a_{1i} = a_{0i}\)
  • \(b_{0i} \sim U(-1.5, 1.5); b_{1i} = b_{0i} + \Delta_i; \Delta_i \in \{0, .5\}; d_{gi} = b_{gi} \times a_{gi}\)

Summary of example

• Similar conclusions as LR test, but not exactly the same
  • Test of item thresholds found same items as LR test
  • Chi-square test found 2 additional items with DIF (due to on slopes)
• Next steps
  • Can follow up with partial invariance model as before to examine item-level effects
  • May consider revising or omit items …
• Other concerns
  • Different DIF methods lead to different conclusions (in general)
  • Does any of this affect conclusions about impact??

Logic of test

• The logic of this test is the similar to the Hausman specification test

• Under the null hypothesis, both the robust estimate and the MLE are consistent (unbiased) estimates of the “true” impact

  • The MLE is more efficient, but this is not very important for us

• Under the alternative hypothesis, the both may be inconsistent, but the robust estimate will be less biased

  • Assuming < 50% of items exhibit DIF

• Consequently, the difference between the estimates can be used to test for the affect of DIF on impact

Relation to MI

• In MI, we test whether (a subset of) item parameters are equal over groups
• May reject MI with little affect on impact
  • DIF on a single item may be negligible when averaged over many items
  • DIF in opposite directions can cancel out over items
• In testing DTF
  • We don’t test any item parameters (although may down weight!)
  • We just test whether two estimates of impact are equal
  • Note that items there may be still items with DIF even if impact is not affected!

What we have covered today

• IRT-based scaling and its relation to DIF
  • More info on scaling in appendix
• Robust scaling
  • See Halpin 2022 for technical details
• Tests of DIF based on robust scaling
  • rdif_z_test and rdif_chisq_test
• Tests of DTF (impact) based on robust scaling
  • delta_test
• Worked example

References

Bechger, T. M., & Maris, G. (2015). A Statistical Test for Differential Item Pair Functioning. Psychometrika, 80, 317–340. https://doi.org/10.1007/s11336- 014- 9408- y

Belzak, W. C. M., & Bauer, D. J. (2020). Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning. Psychological Methods, 25(6), 673–690. https://doi.org/10.1037/met0000253 He, Y., &

He, Y., & Cui, Z. (2020). Evaluating Robust Scale Transformation Methods With Multiple Outlying Common Items Under IRT True Score Equating. Applied Psychological Measurement, 44(4), 296–310. https://doi.org/10.1177/0146621619886050

He, Y., Cui, Z., & Osterlind, S. J. (2015). New Robust Scale Transformation Methods in the Presence of Outlying Common Items. Applied Psychological Measurement, 39(8), 613–626.https://doi.org/10.1177/0146621615587003

Huber, P. J., & Ronchetti, E. (2009). Robust statistics (2nd ed). Wiley.

Magis, D., Tuerlinckx, F., & De Boeck, P. (2015). Detection of Differential Item Functioning Using the Lasso Approach. Journal of Educational and Behavioral Statistics, 40(2), 111–135. https://doi.org/10.3102/1076998614559747

References

Schauberger, G., & Mair, P. (2020). A regularization approach for the detection of differential item functioning in generalized partial credit models. Behavior Research Methods, 52(1), 279–294. https://doi.org/10.3758/s13428-019-01224-2

Stocking, M. L., & Lord, F. M. (1983). Developing a Common Metric in Item Response Theory. Applied Psychological Measurement, 7(2), 201–210. https://doi.org/10.1177/014662168300700208

Teresi, J. A., Wang, C., Kleinman, M., Jones, R. N., & Weiss, D. J. (2021). Differential Item Functioning Analyses of the Patient-Reported Outcomes Measurement Information System (PROMIS) Measures: Methods, Challenges, Advances, and Future Directions. Psychometrika, 86(3), 674–711. https://doi.org/10.1007/s11336-021-09775-0

Wang, W., Liu, Y., & Liu, H. (2022). Testing Differential Item Functioning Without Predefined Anchor Items Using Robust Regression. Journal of Educational and Behavioral Statistics, 47(6), 666–692. https://doi.org/10.3102/10769986221109208

Yuan, K.-H., Liu, H., & Han, Y. (2021). Differential Item Functioning Analysis Without A Priori Information on Anchor Items: QQ Plots and Graphical Test. Psychometrika, 86(2), 345–377. https://doi.org/10.1007/s11336-021-09746-5

More about scaling

• Specify 2PL IRT model in “slope-intercept” form

• Group 1

\[\text{logit}(p_{0i}) = a_{0i} \theta_{0} + d_{0i} \; \text{ with } \; \theta_0 \sim N(0, 1)\]

• Group 2

\[\text{logit}(p_{1i}) = a_{1i} \theta_{1} + d_{1i} \; \text{ with } \; \theta_1 = (\theta^*_1 - \mu) / \sigma \; \text{ and } \; \theta^*_1 \sim N(\mu, \sigma^2)\]

• Scaling involves solving for \(\mu\) and \(\sigma\)

\[a_{1i} \theta_{1} + d_{1i} = a_{1i}^* \theta_{1}^* + a_{1i}^*\]

• We know this relationship holds for some choice of \(a_{1i}^*\) and \(d_{1i}^*\) because IRT models are identified only up to a linear transformation of \(\theta\).

More about scaling

• Substituting \(\theta_1 = (\theta^*_1 - \mu) / \sigma\) in the scaling equations and doing the algebra gives:
  • \(\sigma = a_{1i}/a_{1i}^*\)
  • \(\mu = \sigma \frac{d_{1i} - d_{1i}^*}{a_{1i}}\)
• Taking the mean over items, gives the usual “naive” scaling (in slope-intercept form)
  • \(\sigma = \text{mean}(a_{1i}/a_{1i}^*)\)
  • \(\mu = \sigma \, \text{mean}(\frac{d_{1i} - d_{1i}^*}{a_{1i}})\)
• In the CINEG design, we let the item parameters in the reference group stand-in for the “unscaled” item parameters
  • \(d_{1i}^* = d_{0i}\)
  • \(a_{1i}^* = a_{0i}\)