Package: Gtheory4LLM 1.0.0.9000

Gtheory4LLM: Generalizability Theory for LLM Subjective Tasks

Studies the reliability and generalizability of subjective judgments produced by large language models (LLMs), including annotation, rating, and structured LLM-as-a-judge tasks. Specifies evaluator, prompt, generation, and repeated-run facets through crossed or explicitly nested random sources with configurable item interactions. Fits univariate models, joint Gaussian models, and joint discrete models for binary, ordinal, and unordered categorical outcomes, with source-specific covariance. Gaussian models use exact balanced likelihood, with a bounded observed-row route for incomplete univariate panels. Discrete models use dense or sparse first-order Laplace approximation with Gaussian latent random effects. Supported balanced decision studies compare evaluator, prompt, and replication allocations using observed Gaussian or explicitly requested latent binary and ordinal reliability, with random or fixed facets after Brennan (2001). 'OpenMx' Gaussian fits report asymptotic Wald standard errors for variance components and delta-method coefficient intervals when curvature checks permit them; the 'lme4' route returns point estimates only. Discrete fits offer optional full-information latent coefficient intervals, subject to numerical and boundary checks. An object bootstrap refits resampled objects under each fit's acceptance rule and gives percentile and bias-corrected and accelerated intervals for components, coefficients and decision studies. Provides fixed-layout call effects, categorical score and category-vector reliability projections, repeated-panel decision agreement, bounded cost-aware comparisons, and Gaussian or latent binary/ordinal pilot precision planning, and categorical point-estimator pilot operating characteristics. Joint Gaussian-discrete fitting is not implemented. Includes three publicly archived LLM annotation datasets covering hate-speech, mental-health, and drug-review tasks. Discrete fitting retains conservative size limits; the preflight report describes supported designs and computational limits. Generalizability coefficients follow the variance-decomposition framework of Brennan (2001) <doi:10.1007/978-1-4757-3456-0>.

Authors:Jin Liu [aut, cre, cph]

Gtheory4LLM_1.0.0.9000.tar.gz
Gtheory4LLM_1.0.0.9000.zip(r-4.7-x86_64)Gtheory4LLM_1.0.0.9000.zip(r-4.7-arm64)Gtheory4LLM_1.0.0.9000.zip(r-4.6-x86_64)Gtheory4LLM_1.0.0.9000.zip(r-4.6-arm64)Gtheory4LLM_1.0.0.9000.zip(r-4.5-x86_64)
Gtheory4LLM_1.0.0.9000.tgz(r-4.6-x86_64)Gtheory4LLM_1.0.0.9000.tgz(r-4.6-arm64)Gtheory4LLM_1.0.0.9000.tgz(r-4.5-x86_64)Gtheory4LLM_1.0.0.9000.tgz(r-4.5-arm64)
Gtheory4LLM_1.0.0.9000.tar.gz(r-4.7-arm64)Gtheory4LLM_1.0.0.9000.tar.gz(r-4.7-x86_64)Gtheory4LLM_1.0.0.9000.tar.gz(r-4.6-arm64)Gtheory4LLM_1.0.0.9000.tar.gz(r-4.6-x86_64)
Gtheory4LLM_1.0.0.9000.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
Gtheory4LLM/json (API)

# Install 'Gtheory4LLM' in R:
install.packages('Gtheory4LLM', repos = c('https://veronica0206.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/veronica0206/gtheory4llm/issues

On CRAN:

Conda:

4.95 score 467 downloads 24 exports 15 dependencies

Last updated from:289f23077d. Checks:2 ERROR, 13 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-arm64ERROR339
linux-devel-x86_64OK450
source / vignettesOK215
linux-release-arm64ERROR343
linux-release-x86_64OK468
macos-release-arm64OK283
macos-release-x86_64OK610
macos-oldrel-arm64OK315
macos-oldrel-x86_64OK754
windows-devel-arm64OK549
windows-devel-x86_64OK336
windows-release-arm64OK532
windows-release-x86_64OK405
windows-oldrel-x86_64OK409
wasm-releaseOK103

Exports:gt_batchgt_batch_dstudygt_batch_reliabilitygt_bootstrapgt_categorical_targetgt_component_vcovgt_componentsgt_controlgt_designgt_diagnosticsgt_dstudygt_dstudy_targetgt_examplegt_export_reportgt_familygt_fitgt_laplace_checkgt_pilot_plangt_plangt_preflightgt_reliabilitygt_reportgt_scoregt_simulate

Dependencies:BHclidigestlatticelifecycleMASSMatrixmvtnormOpenMxRcppRcppEigenRcppParallelrlangrpfStanHeaders

How many evaluators, prompts, and runs?
1. Define the quantity and the sampling design | 2. Inspect feasibility before fitting | 3. Fit and inspect acceptance | 4. Compute relative and absolute reliability | 5. Compare measurement budgets | 6. Save a portable analysis report | 7. Fixed facets | 8. Match other outcomes to their observation model | Where to read next

Last update: 2026-10-25
Started: 2026-09-13

Plan annotation effort, costs, and pilot precision
Fit a small Gaussian pilot | Compare costs within a declared allocation grid | Make the batch target explicit | Examine pilot precision for one final protocol

Last update: 2026-10-18
Started: 2026-10-09

Readme and manuals

Help Manual

Help pageTopics
Generalizability Theory for LLM Subjective TasksGtheory4LLM-package Gtheory4LLM
Bounded Gaussian Fitting with Missing Whole Cellsgaussian_observed
Declare That Items Were Annotated Several to a Callgt_batch print.gt_batch
Compare Supplied Allocations for a Fixed Gaussian Batch Layoutas.data.frame.gt_batch_dstudy gt_batch_dstudy print.gt_batch_dstudy
Project Gaussian Score Error Conditional on a Fixed Batch Layoutas.data.frame.gt_batch_reliability gt_batch_reliability print.gt_batch_reliability
Object Bootstrap of an Accepted Fitas.data.frame.gt_bootstrap gt_bootstrap print.gt_bootstrap summary.gt_bootstrap
Integrate Explicit Repeated-Panel Categorical Targetsas.data.frame.gt_categorical_target gt_categorical_target print.gt_categorical_target
Extract Source Covariances or Correlationsgt_components
Configure Gaussian and Discrete Fitting and Acceptance Checksgt_control print.gt_control
Declare a Generalizability-Theory Random-Source Designgt_design print.gt_design
Inspect Numerical Acceptance and Backend Diagnosticsgt_diagnostics print.gt_diagnostics print.gt_diagnostic_stage print.gt_diagnostic_stages
Evaluate Balanced Measurement Allocationsas.data.frame.gt_dstudy gt_dstudy gt_dstudy_target plot.gt_dstudy print.gt_dstudy
Load Modeling Tables from Three LLM-Annotated Datasetsgt_example print.gt_example
Specify a Gaussian, Binary, Ordinal, or Unordered Categorical Outcomegt_family print.gt_family
Fit Univariate or Joint Multivariate Generalizability Modelsgt_fit print.gt_fit summary.gt_fit
Standard Extractors for Fitted Generalizability Modelscoef.gt_fit deviance.gt_fit formula.gt_fit gt_component_vcov gt_fit_methods logLik.gt_fit nobs.gt_fit update.gt_fit vcov.gt_fit
Refined Numerical Comparison for a Discrete Fit's Laplace Objectivegt_laplace_check print.gt_laplace_check
Evaluate Pilot Precision or Categorical Point-Estimator Performanceas.data.frame.gt_pilot_plan gt_pilot_plan print.gt_pilot_plan
Rank a Bounded Set of D-Study Allocations by User-Supplied Costas.data.frame.gt_plan gt_plan print.gt_plan
Inspect Data, Source Dimensions, and Backend Limits Before Fittinggt_preflight plot.gt_preflight print.gt_preflight
Compute Balanced G and Phi Coefficientsas.data.frame.gt_reliability gt_reliability print.gt_reliability
Create and Export a Portable Aggregate Analysis Reportgt_export_report gt_report print.gt_report
Declare Composite-Score Weightsgt_score print.gt_score
Simulate a New Gaussian or Categorical Annotation Panelgt_simulate
Three LLM Annotation Datasets: Design, Coding, and SourcesGtheory4LLM-datasets gtheory_datasets
LLM Drug-Review Sentiment and Four Aspect Annotationsdrug_review drug_review_4aspect gtheory_drug_review
LLM Hate-Speech Annotations of HateXplain Textsgtheory_hate_speech hate_speech
LLM Mental-Health Text Annotations in Five Outcome Setsgtheory_mental_health mental_health_3group mental_health_3L mental_health_6flag mental_health_7L mental_health_nominal
Plot Reliability Coefficients and Available Intervalsplot.gt_reliability
Print a Concise Generalizability Model Summaryprint.summary.gt_fit