Gtheory4LLM - Generalizability Theory for LLM Subjective Tasks
Studies the reliability and generalizability of subjective
judgments produced by large language models (LLMs), including
annotation, rating, and structured LLM-as-a-judge tasks.
Specifies evaluator, prompt, generation, and repeated-run
facets through crossed or explicitly nested random sources with
configurable item interactions. Fits univariate models, joint
Gaussian models, and joint discrete models for binary, ordinal,
and unordered categorical outcomes, with source-specific
covariance. Gaussian models use exact balanced likelihood, with
a bounded observed-row route for incomplete univariate panels.
Discrete models use dense or sparse first-order Laplace
approximation with Gaussian latent random effects. Supported
balanced decision studies compare evaluator, prompt, and
replication allocations using observed Gaussian or explicitly
requested latent binary and ordinal reliability, with random or
fixed facets after Brennan (2001). 'OpenMx' Gaussian fits
report asymptotic Wald standard errors for variance components
and delta-method coefficient intervals when curvature checks
permit them; the 'lme4' route returns point estimates only.
Discrete fits offer optional full-information latent
coefficient intervals, subject to numerical and boundary
checks. An object bootstrap refits resampled objects under each
fit's acceptance rule and gives percentile and bias-corrected
and accelerated intervals for components, coefficients and
decision studies. Provides fixed-layout call effects,
categorical score and category-vector reliability projections,
repeated-panel decision agreement, bounded cost-aware
comparisons, and Gaussian or latent binary/ordinal pilot
precision planning, and categorical point-estimator pilot
operating characteristics. Joint Gaussian-discrete fitting is
not implemented. Includes three publicly archived LLM
annotation datasets covering hate-speech, mental-health, and
drug-review tasks. Discrete fitting retains conservative size
limits; the preflight report describes supported designs and
computational limits. Generalizability coefficients follow the
variance-decomposition framework of Brennan (2001)
<doi:10.1007/978-1-4757-3456-0>.