Package: Gtheory4LLM
Type: Package
Title: Generalizability Theory for LLM Subjective Tasks
Version: 0.5.0.9000
Authors@R: person("Jin", "Liu", email = "Veronica.Liu0206@gmail.com",
    role = c("aut", "cre", "cph"))
Author: Jin Liu [aut, cre, cph]
Maintainer: Jin Liu <Veronica.Liu0206@gmail.com>
Description: Studies the reliability and generalizability of subjective
        judgments produced by large language models (LLMs), including
        annotation, rating, and structured LLM-as-a-judge tasks.
        Specifies evaluator, prompt, generation, and repeated-run
        facets through crossed or explicitly nested random sources with
        configurable item interactions. Fits univariate models, joint
        Gaussian models, and joint discrete models for binary, ordinal,
        and unordered categorical outcomes, with source-specific
        covariance. Gaussian models use exact balanced likelihood;
        discrete models use a dense first-order Laplace approximation
        with Gaussian latent random effects. Supported balanced
        decision studies compare evaluator, prompt, and replication
        allocations using observed Gaussian or explicitly requested
        latent binary and ordinal reliability, with random or fixed
        facets after Brennan (2001). Gaussian fits report asymptotic
        Wald standard errors for their variance components and
        delta-method intervals for the coefficients; discrete fits
        report point estimates only. Provides fixed-layout Gaussian
        batch projections, bounded cost-aware allocation comparisons,
        and Gaussian simulation-based pilot precision planning. Scalar
        nominal reliability and joint Gaussian-discrete fitting are not
        implemented. Includes three publicly archived LLM annotation
        datasets covering hate-speech, mental-health, and drug-review
        tasks. Discrete fitting is limited to small models; the
        preflight report describes supported designs and computational
        limits. Generalizability coefficients follow the
        variance-decomposition framework of Brennan (2001)
        <doi:10.1007/978-1-4757-3456-0>.
License: GPL-3
Encoding: UTF-8
Depends: R (>= 4.5.0)
Imports: Matrix (>= 1.6.0), OpenMx (>= 2.22.11), graphics, methods,
        stats, utils
Suggests: lme4, ordinal, knitr, rmarkdown
VignetteBuilder: knitr
URL: https://github.com/Veronica0206/Gtheory4LLM
BugReports: https://github.com/Veronica0206/Gtheory4LLM/issues
Collate: 'design.R' 'batch.R' 'family.R' 'gaussian_retry.R'
        'gaussian_engine.R' 'gaussian.R' 'gaussian_lme4.R'
        'discrete_response.R' 'discrete_dense.R' 'discrete_sparse.R'
        'discrete_mode.R' 'discrete_sparse_mode.R' 'discrete.R'
        'discrete_check.R' 'preflight.R' 'diagnostics_stages.R' 'fit.R'
        'methods.R' 'reliability.R' 'batch_reliability.R' 'planning.R'
        'simulation.R' 'visualization.R' 'report.R' 'examples.R'
Config/pak/sysreqs: cmake make
Repository: https://veronica0206.r-universe.dev
Date/Publication: 2026-10-18 00:24:19 UTC
RemoteUrl: https://github.com/veronica0206/gtheory4llm
RemoteRef: HEAD
RemoteSha: f4acb425191bf912dc139d859f46bb0cc4e03c51
NeedsCompilation: no
Packaged: 2026-10-18 06:09:17 UTC; root