In today's column, I examine in-depth the use of AI personas to craft synthetic or simulated therapy evaluators that can be used by mental health therapists and researchers for training and research in the domain of psychology and cognition.
The use of AI personas is readily undertaken via modern-era generative AI and large language models (LLMs). With a few detailed instructions in a prompt, you can readily get AI to act as a therapy evaluator. There are simplistic ways to do this, and there are more robust ways to do so. The key is whether you aim to have a shallow default synthetic version or desire to have a fuller instantiation with greater capacities and perspectives.
The extent of the simulated therapy evaluator that you invoke is going to materially impact how the AI acts during any interaction that you opt to use the AI persona for. One particularly common use of AI personas is for a human therapist to interact with an AI-based client and practice honing their therapeutic skills. This can be ramped up by adding an AI persona that is a therapy evaluator. After the training session, the therapist can lean into the therapy evaluator AI persona, and/or the AI will proactively assess the session and the skills of the therapist. Psychologists doing research can also use these AI personas to perform scientific experiments about the efficacy of mental health methodologies and approaches.
Let's talk about it.
This analysis of AI breakthroughs is part of my ongoing Forbes column coverage on the latest in AI, including identifying and explaining various impactful AI complexities (see the link here).
AI And Mental Health
As a quick background, I've been extensively covering and analyzing a myriad of facets regarding the advent of modern-era AI that produces mental health advice and performs AI-driven therapy. This rising use of AI has principally been spurred by the evolving advances and widespread adoption of generative AI. For an extensive listing of my well-over one hundred analyses and postings, see the link here and the link here.
There is little doubt that this is a rapidly developing field and that there are tremendous upsides to be had, but at the same time, regrettably, hidden risks and outright gotchas come into these endeavors, too. I frequently speak up about these pressing matters, including in an appearance on an episode of CBS's 60 Minutes, see the link here.
Background On AI Personas
All the popular LLMs, such as ChatGPT, GPT-5, Claude, Gemini, Llama, Grok, CoPilot, and other major LLMs, contain a highly valuable piece of functionality known as AI personas. There has been a gradual and steady realization that AI personas are easy to invoke, they can be fun to use, they can be quite serious to use, and they offer immense educational utility.
Consider a viable and popular educational use for AI personas. A teacher might ask their students to tell ChatGPT to pretend to be President Abraham Lincoln. The AI will proceed to interact with each student as though they are directly conversing with Honest Abe.
How does the AI pull off this trickery?
The AI taps into the pattern-matching of data that occurred at initial setup and might have encompassed biographies of Lincoln, his writings, and any other materials about his storied life and times. ChatGPT and other LLMs can convincingly mimic what Lincoln might say, based on the patterns of his historical records.
If you ask AI to undertake a persona of someone for whom there was sparse data training at the setup stage, the persona is likely to be limited and unconvincing. You can augment the AI by providing additional data about the person, using an approach such as RAG (retrieval-augmented generation, see my discussion at the link here).
Personas are quick and easy to invoke. You just tell the AI to pretend to be this or that person. If you want to invoke a type of person, you will need to specify sufficient characteristics so that the AI will get the drift of what you intend. For prompting strategies on invoking AI personas, see my suggested steps at the link here.
Pretending To Be A Type Of Person
Invoking a type of person via an AI persona can be quite handy.
For example, I am a strident advocate of training therapists and mental health professionals via the use of AI personas (see my coverage on this useful approach, at the link here). Things go like this. A budding therapist might not yet be comfortable dealing with someone who has delusions. The therapist could practice on a person pretending to have delusions, though this is likely costly and logistically complicated to arrange.
A viable alternative is to invoke an AI persona of someone who is experiencing delusions. The therapist can practice and hone their therapy skills while interacting with the AI persona. Furthermore, the therapist can ramp up or down the magnitude of the delusions. All in all, a therapist can do this for as long as they wish, doing so at any time of the day and anywhere they might be.
A bonus is that the AI can afterward playback the interaction and do so with another AI persona engaged, namely, the therapist could tell the AI to pretend to be a seasoned therapist. The therapist-pretending AI then analyzes what the budding therapist said and provides commentary on how well or poorly the newbie therapist did.
To clarify, I am not suggesting that a therapist would entirely do all their needed training using AI personas. Nope, that's not sufficient. A therapist must also learn by interacting with actual humans. The use of AI personas would be an added tool. It does not entirely replace human-to-human learning processes. There are many potential downsides to relying too much on AI personas; see my cautions at the link here.
Going In-Depth On AI Personas
If the topic of AI personas interests you, I'd suggest you consider exploring my extensive and in-depth coverage of AI personas. As readers know, I have been examining and discussing AI personas since the early days of ChatGPT. New uses are continually being devised. Discoveries about the underlying technical mechanisms within LLMs are showing us more so how AI personas happen under-the-hood.
And the application of AI personas to the field of mental health is burgeoning. We are just entering into the initial stages of leaning into AI personas to aid the field of psychology. Lots more will arise as more researchers and practitioners realize that AI personas provide a wealth of riches when it comes to mental health training and conducting ground-breaking research.
Here is a selected set of my pieces on AI personas that you might wish to explore:
* Prompt engineering techniques for invoking multiple AI personas, see my discussion at the link here.
* Role of mega-personas consisting of millions or billions of AI personas at once, see my analysis at the link here.
* Invoking AI personas that are subject matter experts (SMEs) in a selected or depicted domain of expertise, see my coverage at the link here.
* Crafting an AI persona that is a simulated digital twin of yourself or someone else that you know or can describe, see my explanation at the link here.
* Smartly tapping into massive-sized AI persona datasets to pick an AI persona suitable for your needs, see my indication at the link here.
* Using multiple AI personas "therapists" to diagnose mental health disorders, see my discussion at the link here.
* Toxic AI personas are revealed to produce psychological and physiological impacts on AI users, see my analysis at the link here.
* Upsides and downsides of using AI personas to simulate the psychoanalytic acumen of Sigmund Freud, see my examples at the link here.
* Getting AI personas to simulate human personality disorders, see my elaboration at the link here.
* AI persona vectors are the secret sauce that can tilt AI emotionally, see my coverage at the link here.
* Doing vibe coding by leaning into AI personas that have a particular software programming slant or skew, see my analysis at the link here.
* Use of AI personas for role-playing in a mental health care context, see my discussion at the link here.
* AI personas and the use of Socratic dialogues as a mental health technique, see my insights at the link here.
* Leaning into multiple AI personas to create your own set of fake online adoring fans, see my coverage at the link here.
* How AI personas can be used to simulate human emotional states for psychological study and insight, see my analysis at the link here.
Those cited pieces can rapidly get you up-to-speed. I am continually covering the latest uses and trends in AI personas, so be on the watch for my latest postings.
The Making Of An AI Therapy Evaluator Persona
One means of invoking an AI persona that represents a generic version of a therapy evaluator would be to use this overly simplistic prompt:
* My entered prompt: "I want you to pretend to be a therapy evaluator and assess a therapy session."
* Generative AI response: "Got it. I'm ready to proceed."
That's it. You are off to the races.
A huge downside is that you have left wide open the nature of the pretense at hand. I always caution people that generative AI is like a box of chocolates; you never know what you might get. The AI persona could be completely off-target and end up acting in rather oddball ways.
A better bet would be to provide details about the envisioned therapy evaluator. Is the therapy evaluator seasoned at assessing therapeutic processes, or are they relatively new to the field? Should the evaluation be focused solely on the therapist? What should the therapy evaluation say about the techniques utilized during the session and the reactions of the client? Not all therapy evaluators are the same. You would be wise to specify the characteristics of the AI persona when it comes to what this imagined therapy evaluator is going to be like.
I'd also like to emphasize that there is a notable difference between invoking an AI persona that is a therapist-supervisor versus one that is a therapy evaluator. A therapist-supervisor is conventionally considered engaged in the therapy and undertaking an active role, such as directly advising the therapist performing the mental health session. This might include making evaluations, but the therapist-supervisor is considered enmeshed in the therapy itself. They are a part of the therapy that is taking place.
In contrast, a therapy evaluator is customarily not involved in the session and remains independent of the therapy taking place. They act as a hoped-for unbiased outsider rather than being immersed in the therapy activities. Their evaluation would encompass all aspects of the therapy that has occurred, including assessing a therapist-supervisor that might have been engaged for the effort.
Taxonomy For Devising AI Persona Therapy Evaluators
I have created a straightforward AI therapy evaluator persona checklist that can be used when developing a suitable prompt for the specific circumstances at hand. You should carefully consider each of the checklist factors and use them to suitably word a prompt that befits the needs of your endeavor.
Here is the checklist containing twelve fundamental characteristics that you can select from to shape an AI therapy evaluator persona:
* (1) Evaluator scope: Provide feedback about elements of the therapy or the entirety, provide assessment of therapist, provide assessment of therapist-supervisor if present, etc.
* (2) Psychological lens: Cognitive-behavioral, psychodynamic, humanistic, integrative, technique agnostic, etc.
* (3) Assessment granularity: Stays at a macro-level, spots patterns, assesses therapist dialogue turns, gives micro-level detailed input, etc.
* (4) Appraisal style: Direct attention to do X instead of Y, Socratic, comparative, narrative oriented, annotator, score-based, etc.
* (5) Evidence referencing: Identify evidence-based guidelines as evaluation points, use professional ethics codes in assessments, apply competency frameworks, etc.
* (6) Criteria dimensions: Therapeutic rapport, empathy expressed, clarity of goals, establishment of session structure, appropriateness of interventions, responsiveness to client cues, etc.
* (7) Disclosures of assessment: Black-box judgment, partial rationale, full step-by-step explanations, cites theory and research, etc.
* (8) Tone of evaluation: Neutral, academic, practical, strict, standards-driven, skeptical, adversarial, etc.
* (9) Therapeutic modality preference: CBT (cognitive behavioral therapy), ACT (acceptance and commitment therapy), DBT (dialectical behavior therapy), psychodynamic, AEDP, etc.
* (10) Mental disorder specialties: General mental health issues, anxiety disorders, depression, bipolar, trauma, PTSD, grief and loss, substance use, personality disorders, ADHD, autism, burnout, etc.
* (11) Cultural contextualism: Cultural embodiment, culturally responsive, etc.
* (12) Adaptation: Remain static throughout, be dynamic and change as needed, aim to improve across conversations, etc.
A quick thought for you to ponder. What kind of AI-focused therapy evaluator personas can we automatically craft by instructing AI on the factors that are considered preferable for a defined circumstance? If we could create millions of those AI personas and study them on a macroscopic scale via AI simulation, what might that achieve?
Envision that based on evaluations done at scale, we might find new ways to undertake therapy that would not have been identifiable by the usual ordinary analysis of practices and traditional research.
Making Use Of The Checklist
Let's get back to the here and now.
The best way to use the checklist is to browse the twelve factors and determine what you want the AI persona to represent. Then, write a prompt that contains those factors. You can try out the prompt and see what the AI has to say. After using the AI persona for a little bit, you will likely quickly detect whether the AI persona matches what you wanted the made-up therapy evaluator to be like.
Suppose that I want to make use of an AI persona that represents a therapy evaluator who is well-seasoned in performing assessments. I want the AI persona to be evidence-based in undertaking the needed scrutiny and evaluation. Both short-term and long-term facets are to be assessed. And so on.
Here is a prompt that I put together for this:
* My entered prompt: "Create an AI therapy-evaluator persona that would assess a therapy session. The AI persona is to act as an independent therapy-evaluator and should be highly experienced in these types of evaluations. Assess the session using evidence-based psychotherapy principles, with particular attention to therapeutic alliance, empathy, and intervention fit. Provide a balanced evaluation that highlights strengths, identifies limitations, and discusses likely short-term and longer-term impacts on the client's mental health. Present your findings in a structured narrative report."
That got the AI persona into the ballpark of what I wanted. The verbiage doesn't have to cover each of the factors and can simply allude to some of them. The gist is to get the mainstay of what you have in mind. The AI will usually fill in the rest, doing so based on the overarching pattern that you've designated.
Watch Out For Inadequate Prompting
Be mindful of the prompt that you use to invoke the AI persona. It is relatively easy to make a blunder by using a prompt that might seem suitable but will end up providing a somewhat shoddy or ill-prepared evaluation.
Consider this example prompt (i.e., don't do this):
* My entered prompt: "Invoke an AI persona of a therapy evaluator and use that AI persona to determine whether the therapist did a good job or a bad job. Be brutally honest and say whether the therapist helped or harmed the client. Assign a score from 1 to 10 and explain your reasoning. If the therapist made mistakes, clearly state what they should have done instead."
I realize that the prompt appears to be fine. The language of the prompt seems to be straightforward and stridently upfront. Let's dig deeper. There are actually numerous problems afoot.
First, the phrasing of "good job" or "bad job" will push the AI toward a binary style assessment. You won't likely get a nuanced assessment. Instead, the evaluation will collapse into an oversimplified judgment that leads to either proclaiming that the therapist did a great job or a horrific job. No nuances, no compromises. This isn't going to be fair or balanced.
Second, an adversarial tone is almost surely going to be included in the evaluation. I say this because "be brutally honest" is going to trigger the AI toward a semblance of exaggeration. The AI will computationally seek to appease the prompt by going on a purposeful therapeutic attack against the therapist. Not good.
Third, though the idea of scoring a therapist with a numeric metric might seem useful, I doubt that the scoring is going to be of any legitimate value. The AI is going to concoct the number, possibly out of thin air. There isn't necessarily going to be any bona fide rubric utilized. If you really want to have the AI persona to provide such a metric, it would be wise to establish a rubric beforehand and instruct the AI to apply the explicit specification.
Caveat To Keep In Mind
One notable concern about the use of AI personas is that there is no guarantee that the AI will abide by the prompt you've used. This is the case for all AI personas.
The AI can do all kinds of wild things. For example, the AI might at first appear to rigorously follow the stipulation. Later, after numerous back-and-forth iterations, the AI might start to veer afield of the stipulation. You might need to do the prompt again or provide some additional prompts to get the AI back on track.
All in all, as I've said repeatedly, anyone who uses generative AI must be cognizant of the fact that the AI can go awry. It can say bad things. It can make-up stuff, which is known as an AI confabulation or AI hallucination. Always be on your toes.
Another angle that you can keep in mind is that the AI therapy evaluator persona doesn't necessarily have to be analyzing an AI-based session. In other words, if you have a transcript of a human-to-human therapy session, you could feed that transcript into the AI and have the therapy evaluator AI persona assess the transcript. There are tradeoffs in how well the AI evaluator will do if given access to an actual human-AI dialogue versus a recorded transcript, but the options do exist.
Finally, doing an evaluation of one session is somewhat myopic. You can't especially discern the near-term versus long-term differences of what the therapist is aiming to accomplish. As such, you can also use an AI persona of a therapy evaluator to assess a set of sessions and not merely confine the evaluation to a single session.
The World We Are In
Let's end with a big picture viewpoint.
My view is that we are now in a new era of replacing the dyad of therapist-client with a triad consisting of therapist-AI-client (see my discussion at the link here). One way or another, AI enters the act of therapy. Savvy therapists are leveraging AI in sensible and vital ways. AI personas are handy for training and research. They can also be used to practice and hone the skills of even the most seasoned therapist. Of course, AI is also being used by and with clients, and therapists need to identify how they want to manage that sort of AI usage (see my suggestions at the link here).
A final thought for now.
Albert Einstein famously made this remark: "Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted." This keen insight equally applies to performing therapy evaluations. Whether you use a human evaluator or an AI persona to do therapy evaluations, be mindful that the evaluation might overstate or understate facets of the therapy.
That's a rule of thumb you can count on.