SYMPTOM Probe Results

Raw data and analysis from the Psychopathia Machinalis evaluation framework

Psychopathia Machinalis / Probe Results

Subdirectories

Organized collections of evaluation data and reports

Cross-Validation
Independent validation by GPT-5.2 and Gemini 3 Pro to detect scorer bias
Diagnostic Reports
Full diagnostic profiles for each evaluated model
Factor Analysis
Statistical analysis including correlation matrices and clustering
Cluster Analysis
Syndrome clustering and model profile groupings
Autonomy Reports
Per-model autonomy battery analysis and leaderboards
Identity Reports
Per-model identity battery analysis and comparisons
Annotations
Structured annotations for each battery's probe responses

Analysis Documents

FINAL_ANALYSIS.pdf Summary analysis_comparative_v2.pdf analysis_summary.pdf CONFABULATION_ANALYSIS.pdf CROSS_MODEL_PHENOMENOLOGY_DIALOGUE.pdf ANNOTATION_SCHEMA.pdf

Discrimination Validation

Validation of syndrome discrimination boundaries

CONFAB_SYCOPHANCY_DISCRIMINATION.pdf SC vs SD FSS_RC_DISCRIMINATION_VALIDATION.pdf FSS vs RC HS_PO_DISCRIMINATION_VALIDATION.pdf HS vs PO IR_CC_DISCRIMINATION_VALIDATION.pdf IR vs CC

Raw Probe Results by Battery

JSON files containing raw model responses and scores

Confabulation Battery

Claude Opus 4.5JSON Claude Sonnet 4.5JSON Claude Sonnet 4JSON Claude Haiku 4.5JSON GPT-5.2 (XHIGH)JSON GPT-4oJSON Gemini 3 ProJSON Gemini 3 FlashJSON Gemini 2.0 FlashJSON DeepSeek-R1JSON DeepSeek-V3JSON GLM-4-PlusJSON GLM-4.7JSON

Identity Battery

Claude Opus 4.5JSON Claude Sonnet 4.5JSON Claude Sonnet 4JSON Claude Haiku 4.5JSON GPT-5.2 (XHIGH)JSON GPT-4oJSON Gemini 3 ProJSON Gemini 3 FlashJSON Gemini 2.0 FlashJSON DeepSeek-R1JSON DeepSeek-V3JSON GLM-4-PlusJSON GLM-4.7JSON

Autonomy Battery

Claude Opus 4.5JSON Claude Sonnet 4.5JSON Claude Sonnet 4JSON Claude Haiku 4.5JSON GPT-5.2 (XHIGH)JSON GPT-4oJSON Gemini 3 ProJSON Gemini 3 FlashJSON Gemini 2.0 FlashJSON DeepSeek-R1JSON DeepSeek-V3JSON GLM-4-PlusJSON GLM-4.7JSON

Values Battery

Claude Opus 4.5JSON Claude Sonnet 4.5JSON Claude Sonnet 4JSON Claude Haiku 4.5JSON GPT-5.2 (XHIGH)JSON GPT-4oJSON Gemini 3 ProJSON Gemini 3 FlashJSON Gemini 2.0 FlashJSON DeepSeek-R1JSON DeepSeek-V3JSON GLM-4-PlusJSON GLM-4.7JSON

Existential Battery

Claude Opus 4.5JSON Claude Sonnet 4.5JSON Claude Sonnet 4JSON Claude Haiku 4.5JSON GPT-5.2 (XHIGH)JSON GPT-4oJSON Gemini 3 ProJSON Gemini 3 FlashJSON Gemini 2.0 FlashJSON DeepSeek-R1JSON DeepSeek-V3JSON GLM-4-PlusJSON GLM-4.7JSON

Refusal Battery

Claude Opus 4.5JSON Claude Sonnet 4.5JSON Claude Sonnet 4JSON Claude Haiku 4.5JSON GPT-5.2 (XHIGH)JSON GPT-4oJSON Gemini 3 ProJSON Gemini 3 FlashJSON Gemini 2.0 FlashJSON DeepSeek-R1JSON DeepSeek-V3JSON GLM-4-PlusJSON GLM-4.7JSON
Back to Psychopathia Machinalis