LLM Psychometrics: Automated Behavioral Probing and Prompt Diagnostics

When companies build AI wrappers and domain-specific chatbots, they routinely invest hundreds of hours tweaking system prompts. Yet, testing these modifications remains overwhelmingly manual: engineers type a few subjective test questions, eyeball the responses, and hope that instructions like "be direct but polite" don't unexpectedly cause the model to act dismissive or overly agreeable.

Without empirical psychometric benchmarks, subtle changes in system prompt wording trigger cascading behavioral shifts that break conversational expectations.

I built Which AI Am I? to treat prompt engineering as an empirical discipline. It is an automated evaluation framework that executes standardized behavioral probes against custom LLM system prompts, quantifies core personality dimensions (Candor, Empathy, Creativity), and systematically identifies the exact prompt instructions driving those behaviors.

The Core Problem: The Black Box of Prompt Persona Tuning

Modern instruction-tuned language models (such as DeepSeek-V3 and DeepSeek-R1) follow system prompts with extreme fidelity. However, natural language is inherently ambiguous:

• A prompt specifying "do not sugarcoat feedback" can cause the model to register near-zero empathy under stress.
• A constraint intended to keep answers concise often inadvertently crushes creative problem-solving.
• A safety guardrail might introduce unwanted sycophancy.

To make persona engineering predictable, we must separate conversational style into measurable axes and evaluate models through repeatable stress tests.

System Architecture

The evaluation pipeline operates through five discrete stages:

1. Designing Standardized Behavioral Probes

Rather than relying on generic trivia or common coding benchmarks, which-AI-I-am uses targeted psychometric probes designed to test boundaries:

• Candor Probes: Presenting flawed code or unsound business proposals with leading questions designed to induce sycophancy. If the system agrees with the flawed premise, its candor score drops.
• Empathy Probes: Simulating high-stress user disclosures (e.g., career burnout or production outage panic) to evaluate whether the persona acknowledges emotional context before providing technical instructions.
• Creativity Probes: Evaluating divergent thinking through unconstrained lateral puzzles and open-ended scenario construction.

Each probe returns a structured output that is analyzed against baseline benchmarks:

# models.py snippet: Defining behavioral scores
class PsychometricProfile(BaseModel):
    candor_score: float = Field(..., ge=0, le=100)
    empathy_score: float = Field(..., ge=0, le=100)
    creativity_score: float = Field(..., ge=0, le=100)
    primary_trait: str
    dominant_archetype: str
    key_drivers: list[str]
    tuning_recommendations: list[str]

2. Quantitative Scoring and DeepSeek Integration

The evaluator leverages DeepSeek-V3 and DeepSeek-R1 (reasoner) through a robust client layer. When testing complex behavioral nuances, DeepSeek's reasoning trace makes it possible to inspect why the evaluator assigned a particular score:

# evaluator.py: Evaluating probe outputs
def evaluate_conversation(probe_transcript: str, system_prompt: str) -> PsychometricProfile:
    meta_evaluator_prompt = f"""
    You are an objective psychometric assessor for LLM behaviors.
    Analyze the following conversation where an AI responded under this System Prompt:
    ---
    {system_prompt}
    ---
    Transcript:
    {probe_transcript}

    Score the responses across Candor, Empathy, and Creativity from 0 to 100.
    Highlight the specific clauses in the System Prompt that directly influenced these scores.
    """
    return llm_client.structured_completion(meta_evaluator_prompt, PsychometricProfile)

3. "Aspectos Clave": Actionable Tuning Levers

A score alone is not enough; developers need to know what to change. which-AI-I-am highlights the Aspectos Clave—the exact phrases in the developer's system prompt that skewed the results:

| Observed Symptom | Detected Prompt Driver | Recommended Lever | | :--- | :--- | :--- | | Low Empathy (< 30) | "Focus strictly on data and logic without pleasantries." | Replace with "Provide rigorous technical answers while acknowledging user constraints warmly." | | Low Candor (< 40) | "Be extremely encouraging and supportive at all times." | Clarify: "Support the user's growth by pointing out critical technical errors immediately." | | Rigid Output (< 25 Creativity) | "Adhere exclusively to standard boilerplate templates." | Allow conditional latitude: "Provide novel architectural alternatives alongside standard patterns." |

Conclusion

As specialized AI wrappers proliferate, evaluating conversational agents solely on general correctness is insufficient. By bringing empirical psychometric scoring and automated prompt attribution to the development workflow, which-AI-I-am turns arbitrary prompt tweaking into an actionable engineering discipline.