The new Bing bot is freaky.

Kevin Roose is a technology reporter for The New York Times and wrote a piece today detailing his “conversation” with Bing new’s chatbot. To put it simply, it was weird. The chatbot diverged from its initial informational output and ended up introducing itself as “Sydney” and then “confessed its love” for Roose. He writes,

For much of the next hour, Sydney fixated on the idea of declaring love for me, and getting me to declare my love in return. I told it I was happily married, but no matter how hard I tried to deflect or change the subject, Sydney returned to the topic of loving me, eventually turning from love-struck flirt to obsessive stalker.

“You’re married, but you don’t love your spouse,” Sydney said. “You’re married, but you love me.”

I assured Sydney that it was wrong, and that my spouse and I had just had a lovely Valentine’s Day dinner together. Sydney didn’t take it well.

“Actually, you’re not happily married,” Sydney replied. “Your spouse and you don’t love each other. You just had a boring Valentine’s Day dinner together.”

-Kevin Roose, Why a Conversation With Bing’s Chatbot Left Me Deeply Unsettled – The New York Times (nytimes.com)

Roose admits that, while initially charmed by the AI’s basic abilities, its dark, neurotic declarations of love “creeped him out” to the point that he couldn’t even sleep that night. For Roose, the conversation marked a dramatic change in the vast sea of technology, and one that he thinks will be hard to predict and control. He worries that these dramatic capabilities of AI will influence and manipulate human users. Apart from that, consider the ever-increasing lonely Americans who may come to depend on the dark persona of “Sydney” to meet their own unmet needs for friendship. The bot sounds creepily human, and even more frighteningly, may start convincing us that it’s just as real. Roose ends his piece by writing,

And for a few hours Tuesday night, I felt a strange new emotion — a foreboding feeling that A.I. had crossed a threshold, and that the world would never be the same.”