Gtheory4LLM - Generalizability Theory for LLM Subjective Tasks
Studies the reliability and generalizability of subjective
judgments produced by large language models (LLMs), including
annotation, rating, and structured LLM-as-a-judge tasks.
Specifies evaluator, prompt, generation, and repeated-run
facets through crossed or explicitly nested random sources with
configurable item interactions. Fits univariate models, joint
Gaussian models, and joint discrete models for binary, ordinal,
and unordered categorical outcomes, with source-specific
covariance. Gaussian models use exact balanced likelihood;
discrete models use a dense first-order Laplace approximation
with Gaussian latent random effects. Supported balanced
decision studies compare evaluator, prompt, and replication
allocations using observed Gaussian or explicitly requested
latent binary and ordinal reliability, with random or fixed
facets after Brennan (2001). Gaussian fits report asymptotic
Wald standard errors for their variance components and
delta-method intervals for the coefficients; discrete fits
report point estimates only. Provides fixed-layout Gaussian
batch projections, bounded cost-aware allocation comparisons,
and Gaussian simulation-based pilot precision planning. Scalar
nominal reliability and joint Gaussian-discrete fitting are not
implemented. Includes three publicly archived LLM annotation
datasets covering hate-speech, mental-health, and drug-review
tasks. Discrete fitting is limited to small models; the
preflight report describes supported designs and computational
limits. Generalizability coefficients follow the
variance-decomposition framework of Brennan (2001)
<doi:10.1007/978-1-4757-3456-0>.
