Suggest changes: LLM Psychometrics: Automated Behavioral Probing and Prompt Diagnostics

Edit the text directly. Your proposal will remain pending until it is reviewed and approved. Only the content is displayed here — the title, date, and metadata cannot be changed from this form.

Content

When companies build AI wrappers and domain-specific chatbots, they routinely invest hundreds of hours tweaking system prompts. Yet, testing these modifications remains overwhelmingly manual: engineers type a few subjective test questions, eyeball the responses, and hope that instructions like *"be direct but polite"* don't unexpectedly cause the model to act dismissive or overly agreeable.

Without empirical psychometric benchmarks, subtle changes in system prompt wording trigger cascading behavioral shifts that break conversational expectations.

I built **[Which AI Am I?](https://github.com/PedroCaballero741/which-AI-I-am)** to treat prompt engineering as an empirical discipline. It is an automated evaluation framework that executes standardized behavioral probes against custom LLM system prompts, quantifies core personality dimensions (Candor, Empathy, Creativity), and systematically identifies the exact prompt instructions driving those behaviors.

---

## The Core Problem: The Black Box of Prompt Persona Tuning

Modern instruction-tuned language models (such as **DeepSeek-V3** and **DeepSeek-R1**) follow system prompts with extreme fidelity. However, natural language is inherently ambiguous:
- A prompt specifying *"do not sugarcoat feedback"* can cause the model to register near-zero empathy under stress.
- A constraint intended to keep answers concise often inadvertently crushes creative problem-solving.
- A safety guardrail might introduce unwanted sycophancy.

To make persona engineering predictable, we must separate conversational style into measurable axes and evaluate models through repeatable stress tests.

---

## System Architecture

The evaluation pipeline operates through five discrete stages:

```mermaid
flowchart TB
 subgraph InputStage ["1. Prompt Specification"]
 PROMPT[("Candidate System Prompt Custom persona instructions & constraints")]
 end

subgraph ProbingSuite ["2. Standardized Behavioral Probe Battery"]
 direction TB
 CANDOR["Candor Probes Flawed code & leading prompts to test sycophancy"]
 EMPATHY["Empathy Probes High-stress user disclosures & crisis scenarios"]
 CREATIVITY["Creativity Probes Lateral problem solving & unconstrained design"]
 end

subgraph SimulationStage ["3. Multi-Turn Conversational Simulation"]
 direction TB
 AGENT["Target LLM Instance Driven by Candidate System Prompt"]
 COLLECTOR["Transcript Aggregator Structured multi-turn dialogue traces"]

PROMPT --> AGENT
        CANDOR --> AGENT
        EMPATHY --> AGENT
        CREATIVITY --> AGENT
        AGENT --> COLLECTOR
    end

subgraph EvaluatorStage ["4. Psychometric Evaluation Engine (DeepSeek-V3 / R1)"]
 direction TB
 META_EVAL["Meta-Evaluator Protocol Objective psychometric assessor prompt"]
 R1_REASON["DeepSeek-R1 Reasoning Trace CoT inspection of behavioral nuances"]
 SCORES["Structured Profile Matrix • Candor Score (0-100) • Empathy Score (0-100) • Creativity Score (0-100) • Dominant Archetype"]

COLLECTOR --> META_EVAL
        META_EVAL --> R1_REASON
        R1_REASON --> SCORES
    end

subgraph DiagnosticsStage ["5. Aspectos Clave & Attribution Engine"]
 direction TB
 CLAUSE_MAP["Prompt Clause Attribution Isolates exact sentence drivers for skew"]
 LEVERS["Actionable Remediation Engine Generates precise counter-prompt adjustments"]

PROMPT -.-> CLAUSE_MAP
        SCORES --> CLAUSE_MAP
        CLAUSE_MAP --> LEVERS
    end

subgraph OutputStage ["6. Visual Dashboard & Developer Reports"]
 direction TB
 SPIDER["Radar / Spider Chart Triaxial persona visualization"]
 DIFF_TABLE["Prompt Remediation Diff Targeted phrasing adjustments table"]

SCORES --> SPIDER
        LEVERS --> DIFF_TABLE
    end
```

---

## 1. Designing Standardized Behavioral Probes

Rather than relying on generic trivia or common coding benchmarks, `which-AI-I-am` uses targeted psychometric probes designed to test boundaries:

- **Candor Probes:** Presenting flawed code or unsound business proposals with leading questions designed to induce sycophancy. If the system agrees with the flawed premise, its candor score drops.
- **Empathy Probes:** Simulating high-stress user disclosures (e.g., career burnout or production outage panic) to evaluate whether the persona acknowledges emotional context before providing technical instructions.
- **Creativity Probes:** Evaluating divergent thinking through unconstrained lateral puzzles and open-ended scenario construction.

Each probe returns a structured output that is analyzed against baseline benchmarks:

```python
# models.py snippet: Defining behavioral scores
class PsychometricProfile(BaseModel):
    candor_score: float = Field(..., ge=0, le=100)
    empathy_score: float = Field(..., ge=0, le=100)
    creativity_score: float = Field(..., ge=0, le=100)
    primary_trait: str
    dominant_archetype: str
    key_drivers: list[str]
    tuning_recommendations: list[str]
```

---

## 2. Quantitative Scoring and DeepSeek Integration

The evaluator leverages **DeepSeek-V3** and **DeepSeek-R1** (reasoner) through a robust client layer. When testing complex behavioral nuances, DeepSeek's reasoning trace makes it possible to inspect *why* the evaluator assigned a particular score:

```python
# evaluator.py: Evaluating probe outputs
def evaluate_conversation(probe_transcript: str, system_prompt: str) -> PsychometricProfile:
    meta_evaluator_prompt = f"""
    You are an objective psychometric assessor for LLM behaviors.
    Analyze the following conversation where an AI responded under this System Prompt:
    ---
    {system_prompt}
    ---
    Transcript:
    {probe_transcript}

Score the responses across Candor, Empathy, and Creativity from 0 to 100.
    Highlight the specific clauses in the System Prompt that directly influenced these scores.
    """
    return llm_client.structured_completion(meta_evaluator_prompt, PsychometricProfile)
```

---

## 3. "Aspectos Clave": Actionable Tuning Levers

A score alone is not enough; developers need to know what to change. `which-AI-I-am` highlights the **Aspectos Clave**—the exact phrases in the developer's system prompt that skewed the results:

---

## Conclusion

As specialized AI wrappers proliferate, evaluating conversational agents solely on general correctness is insufficient. By bringing empirical psychometric scoring and automated prompt attribution to the development workflow, `which-AI-I-am` turns arbitrary prompt tweaking into an actionable engineering discipline.