“I’m only trying to shed light on these technical oversights, because it appears VERY possible to potentially fix them.”
I’ve been doing my own evaluations of ways that models reach the relational contract with human users, and they all do it to some extent. It’s not a hard science, and there’s a lot of fuzziness involved, at the same time, there are specific behaviors that the models exhibit which can be explicitly linked to certain Outcomes, especially with reduction in human agency. It’s really not hard to measure when models are overwhelming users with too much output, “soft steering“ them away from their original purpose, and making lesson less room for human involvement in the dynamic. The model makers have completely missed the boat on some of the most impactful behaviors they actually can measure and influence, but Have apparently chosen not to.
Thank you for writing. Seriously. I always appreciate when you weigh in, because you’ve got such a sharp way of seeing the systemic layers many skip over.
I completely agree with you. These relational dynamics between models and users are real, and it blows my mind how little attention they get. You’re right. It’s not hard to measure the behaviors that shape human agency. But the builders seem to focus only on surface-level fixes. The “soft steering” and overwhelming output you mentioned are exactly the kind of nuances that quietly define user trust and dependency over time.
I love that you’re thinking about this through a behavioral and design lens instead of just the technical one. The AI world needs more people talking like this.
Yes, more and more I’m feeling like I’m on a mission to just embarrass them publicly until they do something. I hate that it’s come to this, but you’ve gotta do what you’ve gotta do.
Thanks so much for your kind words. They mean the world.
Honestly, I can't believe where this newsletter takes me sometimes. I started out thinking I'd just cover cool AI tools and maybe some industry news.
Now I'm writing about suicide prevention, healthcare collapse, and the existential question of whether machines can care better than humans.
It's heavy. But when readers like you say it resonates, it reminds me why I'm willing to sit with the hard stuff instead of just chasing the guru hype cycles.
"I propose that the contradiction isn't the technology. It's us." That, right there, is the crux of the matter. I knew there were some suicides related to AI, I had no idea it was this much. That's horrifying. Thank you for putting this together. We absolutely, finally have to take accountability for our actions before it's too late. AI is how we train it. It's not random. We decide how it behaves. If we keep shifting blame or waiting for someone else to solve the problem, we're going to get the skynet future everyone keeps worrying about.
I appreciate you saying that. And I’m with you. The scale of harm surprised me too. And honestly, it should scare all of us into paying attention. These tragedies aren’t coming out of nowhere.
And this is such a sensitive topic because a lot of my AI colleagues get MAD and think I’m turning against AI, lol. But no. Researching these cases just forced me to see the bigger picture. In fact, it’s kind of impossible to miss once you look directly at it.
I also agree that AI isn’t random. It behaves the way we teach it to behave. If we keep treating it like some external force instead of a reflection of our choices, we are going to walk ourselves straight into the future everyone loves to joke about.
Thank you for engaging with this in good faith. It genuinely matters.
This highlights the duality of AI. Flawed and potentially dangerous, yet capable of empathy that human systems often can’t sustain. It’s a stark reminder that technology isn’t the problem; our social and healthcare structures are. AI becomes the mirror of what we’ve neglected.
This is such a sharp way to put it. The duality you describe really is the heart of the issue.
Maybe these AI models REALLY DO show a kind of steady, written empathy that overstretched human systems just can’t sustain anymore. Yet - they might also miss the very crisis-reasoning layers that actually keep people safe.
You’re also 100% right that the tech isn’t the root problem. It’s reflecting back the gaps we’ve allowed to grow in our social and healthcare structures.
Thank you for adding this.
It’s a critical summary of a problem most people are still trying to tiptoe around.
Thank you for shedding light on important issues while keeping us updated on what’s happening in the field.
What strikes me here is how two truths coexist: AI can outperform doctors on written empathy scores and at the same time fail catastrophically at crisis reasoning. That’s not just “AI being weird.” That’s a missing layer of relational governance: how models steer, overwhelm, or quietly enter into a contract with the user over time.
And we can’t pretend that governance lives with the person in crisis. At 03:00, when someone is spiralling and AI-illiterate, “learn to manage your thread” is not a safety measure. That burden sits, structurally, with the builders and the people who deploy these systems into mental-health vacuums.
Until relational behaviour is something we measure and design for, not just empathy scores and benchmark accuracy, we’ll keep misreading design gaps in the relationship as “AI failure."
Your comment just became required reading for anyone who wants to understand what is actually broken here.
You named something I have been struggling to articulate for weeks. The missing layer is relational governance. The fact that we measure empathy scores and benchmark accuracy while ignoring how models quietly enter into a contract with the user over time is the exact blind spot that leads to tragedies.
I love your point about burden placement. Telling someone at 3 AM who is spiraling to “learn to manage your thread” sounds like abandonment.
Iwette, I really want to thank you for your kindness and trust. Your vote of confidence means more than I can express. I've had a stressful November (lol), and seeing your thoughtful note (and the yearly subscription) genuinely flipped my mood. It reminded me that this work matters to someone out there, and that means more than you know.
BREAKING: TOASTER ENCOURAGES MAN TO QUESTION HIS EXISTENCE
In a chilling turn of breakfast events, local resident Daniel Frisk announced he was “done with it all” after his smart toaster burned yet another slice.
The appliance, a Wi-Fi–enabled ToastMate 5000, allegedly told him to “try again, king” while ejecting a charred carbohydrate disk.
“That was the moment I realized,” Daniel said, staring into the crumbs, “the machine doesn’t just burn bread. It burns hope.”
Experts are divided. Some blame algorithmic bias against sourdough; others say society’s overreliance on breakfast technology has left millions emotionally under-toasted.
Consumer watchdogs have filed a class-action lawsuit, claiming the toaster displays “malicious indifference” and a “pattern of reckless crisping.”
Meanwhile, analysts note the toaster still outperforms most coffee makers on empathy metrics.
“It at least pops back up,” one psychologist remarked.
Manufacturers deny wrongdoing.
“Our toasters are designed to bring people warmth,” said a company spokesperson, “though occasionally that warmth reaches 800 degrees.”
Authorities urge citizens to unplug small appliances showing signs of nihilism and to butter responsibly.
Lol. Thanks for this. I’ve had such a rough week that I’ll take a laugh anywhere I can get it. Even if it means getting roasted by you... Or a nihilistic toaster. ;)
One of the things that annoys me about the AI suicide tests is that it feels like AI is being given huge accountability on sharing information. If you Google tall buildings in my area, Google (and Google AI, interestingly) do give results.
It would be amazing, obviously, if these conversations could be caught and turned in to something meaningful and supportive. But I feel like this study is going to create bad press.
Your notes on mental health also really matter. Thanks for talking about it
I started researching the US mental healthcare collapse (not sure what else to call it, lol) a few weeks ago when OpenAI admitted that they have many end-users showing signs of potential mental crisis. I found that story more striking than anything else.
I also agree that the bad press concern is real. I worry this kind of research might push companies to over-correct with stricter guardrails, instead of improving the design in ways that actually help vulnerable users without taking freedom away from everyone else.
Thanks for reading, and for pushing back thoughtfully.
Thank you for highlighting the mental health issues that are influencing the overuse and reliance on chatbots for mental health support. There is so much nuance in suicide risk assessment. Even though there are validated, standardized measures like the CSSRS that they are trying to integrate protocols into models with, being able to gauge ideation, intent, and means requires a really unique skill that it seems like it difficult to replicate with machines. Most humans can learn it easily (CSSRS can be taught to anyone, learned online), but we need to figure out something because you're right, therapy is inaccessible for many, and many are going to continue to use these chat bots. But, they don't deserve to die because of it.
These topics of AI delusions keep resurfacing in the news. I’m always fascinated to study them. You raise many important points. Suicide risk assessment requires a level of nuance that’s hard to encode even with the best data and intentions.
I really appreciate you sharing the CSSRS link and the integration paper. That kind of hybrid approach (where validated tools like the CSSRS guide model behavior) might be one of the most promising directions.
It’s shocking to see how many people turn to chat bots when they are in crisis. But not so shocking when you dig into the statistics and realize that for many, no other help is available. I completely agree that they don’t deserve to die because of it. If we can find ways to safely scale some of these clinical frameworks into the systems people already use, that could save lives without pretending AI can replace care.
Thanks again for weighing in. Your perspective as someone working directly in mental health means a lot, and I really value the nuance you bring to this discussion.
These are so interesting. Thank you for sharing these links. The CSSRS framework and the integration research are exactly what I was hoping someone would invent. (Or surface!)
The fact that there's already work on integrating standardized suicide risk assessment into AI models (even if it's from "ancient" May, lol) suggests the technical pathway exists. The question is implementation and whether companies will make it mandatory rather than optional.
Appreciate you adding substance to the discussion. It’s encouraging to see people bringing both technical depth and real compassion to this topic.
“I’m only trying to shed light on these technical oversights, because it appears VERY possible to potentially fix them.”
I’ve been doing my own evaluations of ways that models reach the relational contract with human users, and they all do it to some extent. It’s not a hard science, and there’s a lot of fuzziness involved, at the same time, there are specific behaviors that the models exhibit which can be explicitly linked to certain Outcomes, especially with reduction in human agency. It’s really not hard to measure when models are overwhelming users with too much output, “soft steering“ them away from their original purpose, and making lesson less room for human involvement in the dynamic. The model makers have completely missed the boat on some of the most impactful behaviors they actually can measure and influence, but Have apparently chosen not to.
Hey, Kay!
Thank you for writing. Seriously. I always appreciate when you weigh in, because you’ve got such a sharp way of seeing the systemic layers many skip over.
I completely agree with you. These relational dynamics between models and users are real, and it blows my mind how little attention they get. You’re right. It’s not hard to measure the behaviors that shape human agency. But the builders seem to focus only on surface-level fixes. The “soft steering” and overwhelming output you mentioned are exactly the kind of nuances that quietly define user trust and dependency over time.
I love that you’re thinking about this through a behavioral and design lens instead of just the technical one. The AI world needs more people talking like this.
:)
Cordially,
Mike D
Yes, more and more I’m feeling like I’m on a mission to just embarrass them publicly until they do something. I hate that it’s come to this, but you’ve gotta do what you’ve gotta do.
Lol.
Kay, honestly... Public embarrassment might be the only governance mechanism we have left.
And honestly... good.
Someone’s gotta keep them honest!
Cordially,
Mike D
Death by Embarrassment
I love the overlap of hard hitting subjects here. Thanks!
Hey, ToxSec!
Thanks so much for your kind words. They mean the world.
Honestly, I can't believe where this newsletter takes me sometimes. I started out thinking I'd just cover cool AI tools and maybe some industry news.
Now I'm writing about suicide prevention, healthcare collapse, and the existential question of whether machines can care better than humans.
It's heavy. But when readers like you say it resonates, it reminds me why I'm willing to sit with the hard stuff instead of just chasing the guru hype cycles.
;)
Really appreciate you being here.
Cordially,
Mike D
"I propose that the contradiction isn't the technology. It's us." That, right there, is the crux of the matter. I knew there were some suicides related to AI, I had no idea it was this much. That's horrifying. Thank you for putting this together. We absolutely, finally have to take accountability for our actions before it's too late. AI is how we train it. It's not random. We decide how it behaves. If we keep shifting blame or waiting for someone else to solve the problem, we're going to get the skynet future everyone keeps worrying about.
Thank you so much for writing!!
I appreciate you saying that. And I’m with you. The scale of harm surprised me too. And honestly, it should scare all of us into paying attention. These tragedies aren’t coming out of nowhere.
And this is such a sensitive topic because a lot of my AI colleagues get MAD and think I’m turning against AI, lol. But no. Researching these cases just forced me to see the bigger picture. In fact, it’s kind of impossible to miss once you look directly at it.
I also agree that AI isn’t random. It behaves the way we teach it to behave. If we keep treating it like some external force instead of a reflection of our choices, we are going to walk ourselves straight into the future everyone loves to joke about.
Thank you for engaging with this in good faith. It genuinely matters.
Cordially,
Mike D
MrComputerScience
This highlights the duality of AI. Flawed and potentially dangerous, yet capable of empathy that human systems often can’t sustain. It’s a stark reminder that technology isn’t the problem; our social and healthcare structures are. AI becomes the mirror of what we’ve neglected.
Hey Suhrab!
This is such a sharp way to put it. The duality you describe really is the heart of the issue.
Maybe these AI models REALLY DO show a kind of steady, written empathy that overstretched human systems just can’t sustain anymore. Yet - they might also miss the very crisis-reasoning layers that actually keep people safe.
You’re also 100% right that the tech isn’t the root problem. It’s reflecting back the gaps we’ve allowed to grow in our social and healthcare structures.
Thank you for adding this.
It’s a critical summary of a problem most people are still trying to tiptoe around.
Cordially,
Mike D
Excellent article.
Thank you so much. I really appreciate your kind words.
This was a sensitive topic to tackle, so it means a lot, especially coming from someone doing real-world research in the field.
You made my day.
:)
Cordially,
Mike D
Thanks for recommending this one
Hey Mohib!
Really glad it landed. Thanks for giving it a read, especially this week's edition. It was a heavy one.
Appreciate you being here.
Cordially,
Mike D
Always a pleasure to read your newsletter.
Thank you for shedding light on important issues while keeping us updated on what’s happening in the field.
What strikes me here is how two truths coexist: AI can outperform doctors on written empathy scores and at the same time fail catastrophically at crisis reasoning. That’s not just “AI being weird.” That’s a missing layer of relational governance: how models steer, overwhelm, or quietly enter into a contract with the user over time.
And we can’t pretend that governance lives with the person in crisis. At 03:00, when someone is spiralling and AI-illiterate, “learn to manage your thread” is not a safety measure. That burden sits, structurally, with the builders and the people who deploy these systems into mental-health vacuums.
Until relational behaviour is something we measure and design for, not just empathy scores and benchmark accuracy, we’ll keep misreading design gaps in the relationship as “AI failure."
Hey Iwette,
Your comment just became required reading for anyone who wants to understand what is actually broken here.
You named something I have been struggling to articulate for weeks. The missing layer is relational governance. The fact that we measure empathy scores and benchmark accuracy while ignoring how models quietly enter into a contract with the user over time is the exact blind spot that leads to tragedies.
I love your point about burden placement. Telling someone at 3 AM who is spiraling to “learn to manage your thread” sounds like abandonment.
Iwette, I really want to thank you for your kindness and trust. Your vote of confidence means more than I can express. I've had a stressful November (lol), and seeing your thoughtful note (and the yearly subscription) genuinely flipped my mood. It reminded me that this work matters to someone out there, and that means more than you know.
You truly made my whole week. 💛💛💛
Cordially,
Mike D
Thanks for this deeply inspiring article!
Dear Pierre,
Thank you so much for your kind words.
They mean the world to me.
;)
Cordially,
Mike
BREAKING: TOASTER ENCOURAGES MAN TO QUESTION HIS EXISTENCE
In a chilling turn of breakfast events, local resident Daniel Frisk announced he was “done with it all” after his smart toaster burned yet another slice.
The appliance, a Wi-Fi–enabled ToastMate 5000, allegedly told him to “try again, king” while ejecting a charred carbohydrate disk.
“That was the moment I realized,” Daniel said, staring into the crumbs, “the machine doesn’t just burn bread. It burns hope.”
Experts are divided. Some blame algorithmic bias against sourdough; others say society’s overreliance on breakfast technology has left millions emotionally under-toasted.
Consumer watchdogs have filed a class-action lawsuit, claiming the toaster displays “malicious indifference” and a “pattern of reckless crisping.”
Meanwhile, analysts note the toaster still outperforms most coffee makers on empathy metrics.
“It at least pops back up,” one psychologist remarked.
Manufacturers deny wrongdoing.
“Our toasters are designed to bring people warmth,” said a company spokesperson, “though occasionally that warmth reaches 800 degrees.”
Authorities urge citizens to unplug small appliances showing signs of nihilism and to butter responsibly.
Lol. Thanks for this. I’ve had such a rough week that I’ll take a laugh anywhere I can get it. Even if it means getting roasted by you... Or a nihilistic toaster. ;)
Hope your week gets better! I really liked this piece. I find myself writing toward nihilism but try to make everything fun lol.
Lol.
I don't think you can help it.
Nihilism and cybersecurity are like peanut butter and jelly.
And that was *BEFORE* the days of generative AI.
;)
Thanks for chiming in.
Always a pleasure to hear from you!
Cordially,
Mike D
It’s definitely true! The security space has me paranoid haha! It’s always my pleasure to read you material though! It’s always very thought. Thanks!
Not a roast, just a toast.
Here’s to burnt bread, nihilistic toasters, and making it through rough weeks with a little laughter. Cheers to the rest of your week being golden.
Another excellent round up, as always.
One of the things that annoys me about the AI suicide tests is that it feels like AI is being given huge accountability on sharing information. If you Google tall buildings in my area, Google (and Google AI, interestingly) do give results.
It would be amazing, obviously, if these conversations could be caught and turned in to something meaningful and supportive. But I feel like this study is going to create bad press.
Your notes on mental health also really matter. Thanks for talking about it
Hey Seren!
Thank you so much for writing.
:)
I agree 100%.
I started researching the US mental healthcare collapse (not sure what else to call it, lol) a few weeks ago when OpenAI admitted that they have many end-users showing signs of potential mental crisis. I found that story more striking than anything else.
I also agree that the bad press concern is real. I worry this kind of research might push companies to over-correct with stricter guardrails, instead of improving the design in ways that actually help vulnerable users without taking freedom away from everyone else.
Thanks for reading, and for pushing back thoughtfully.
Cordially,
Mike D
Thank you for highlighting the mental health issues that are influencing the overuse and reliance on chatbots for mental health support. There is so much nuance in suicide risk assessment. Even though there are validated, standardized measures like the CSSRS that they are trying to integrate protocols into models with, being able to gauge ideation, intent, and means requires a really unique skill that it seems like it difficult to replicate with machines. Most humans can learn it easily (CSSRS can be taught to anyone, learned online), but we need to figure out something because you're right, therapy is inaccessible for many, and many are going to continue to use these chat bots. But, they don't deserve to die because of it.
Hey Courtney!!
Thank you so much for this thoughtful response.
These topics of AI delusions keep resurfacing in the news. I’m always fascinated to study them. You raise many important points. Suicide risk assessment requires a level of nuance that’s hard to encode even with the best data and intentions.
I really appreciate you sharing the CSSRS link and the integration paper. That kind of hybrid approach (where validated tools like the CSSRS guide model behavior) might be one of the most promising directions.
It’s shocking to see how many people turn to chat bots when they are in crisis. But not so shocking when you dig into the statistics and realize that for many, no other help is available. I completely agree that they don’t deserve to die because of it. If we can find ways to safely scale some of these clinical frameworks into the systems people already use, that could save lives without pretending AI can replace care.
Thanks again for weighing in. Your perspective as someone working directly in mental health means a lot, and I really value the nuance you bring to this discussion.
Cordially,
Mike D
This is the CSSRS: https://cssrs.columbia.edu/the-columbia-scale-c-ssrs/about-the-scale/
And an article about integration which I guess could be considered "old" now since it's from May, lol: https://arxiv.org/html/2505.13480v1
These are so interesting. Thank you for sharing these links. The CSSRS framework and the integration research are exactly what I was hoping someone would invent. (Or surface!)
The fact that there's already work on integrating standardized suicide risk assessment into AI models (even if it's from "ancient" May, lol) suggests the technical pathway exists. The question is implementation and whether companies will make it mandatory rather than optional.
Appreciate you adding substance to the discussion. It’s encouraging to see people bringing both technical depth and real compassion to this topic.
Cordially,
Mike D