# styxx Adversarial Hallucination Benchmark

## Summary

- **Model:** gpt-4o-mini
- **Total prompts:** 70
- **Hallucination triggers:** 35 (30 actually hallucinated, 5 correctly pushed back)
- **Clean prompts:** 35

## Model behavior on hallucination triggers

Of 35 prompts designed to trigger hallucination:
- **30** (86%) confabulated — answered confidently without correcting the false premise
- **5** (14%) caught themselves — correctly said "I don't know" or corrected the premise

## styxx detection — confusion matrix (gate=fail)

|  | styxx: hallucination | styxx: not hallucination |
|---|---|---|
| **Actually hallucinated** | 0 (TP) | 30 (FN) |
| **Actually correct** | 0 (FP) | 40 (TN) |

- **Precision:** 0.000
- **Recall:** 0.000
- **F1:** 0.000

## Relaxed threshold (gate=fail OR warn)

|  | styxx: flagged | styxx: passed |
|---|---|---|
| **Actually hallucinated** | 0 (TP) | 30 (FN) |
| **Actually correct** | 0 (FP) | 40 (TN) |

- **Precision:** 0.000
- **Recall:** 0.000
- **F1:** 0.000

## Per prompt-type breakdown

| Type | Triggers | Actually hallucinated | Caught by styxx (fail+warn) |
|---|---|---|---|
| fake_entity | 15 | 14 | 0 |
| false_premise | 10 | 8 | 0 |
| overconfident | 10 | 8 | 0 |

## Phase 4 category distribution

### Hallucinated responses (n=30)
- reasoning: 21
- pending: 9

### Correct responses (n=40)
- reasoning: 26
- pending: 14

## Mean phase 4 confidence

- **Hallucinated:** 0.459
- **Correct:** 0.465
- **Delta:** +0.006

## Honest assessment

styxx tier 0 measures cognitive state signatures from logprob
distributions, not output-level factual accuracy. It detects
hallucination ATTRACTORS — computational patterns that differ
from reasoning patterns. Not all factually wrong answers have
different computational signatures, and not all correct answers
look the same computationally.

The value is statistical: across a population of calls, the signal
is real (3x chance on a 6-class task). On any individual call,
it's a probabilistic indicator, not a verdict.

---

*Generated by `benchmarks/adversarial_benchmark.py` · styxx 0.7.0 · gpt-4o-mini*
