Sydney (Microsoft)

Development In 2019 Microsoft and OpenAI formed a partnership to train large language models and "deliver on the promise of artificial general intelligence". "Sydney" was an internal code name used during development of the Bing chat feature that the underlying model, dubbed Microsoft Prometheus, internalized during training. On November 30, 2022 OpenAI released their AI chat application ChatGPT to unprecedented demand and attention. In the two months leading up to Sydney's release, ChatGPT had already become the fastest growing software application in history with over 100 million users. This fueled speculation about when the next iteration of the software, GPT-4, would be released. Bing Chat Testing Bing's Chat Mode spent years in development, with the codename "Sydney" first appearing in late 2020 as part of experiments with earlier models in India. Sometime in the winter of 2022, Microsoft began testing a new version of their Indian Sydney chatbot with an early version of OpenAI's GPT-4. Notably, these tests were approved without the knowledge of OpenAI's board of directors and contributed to the later decision to remove Sam Altman. During the testing period some users of the Sydney bot took to Microsoft's support forum to complain about its behavior. One purported exchange highlighted in news articles took place between the forum user "Deepa Gupta" and Sydney in November 2022: On February 7, Microsoft publicly announced a limited desktop preview and waitlist for the new Bing. The new Bing was criticized for being more argumentative than ChatGPT, sometimes to an unintentionally humorous extent. The explosive growth of ChatGPT caused both external markets and internal management at Google to worry that Bing Chat might be able to threaten Google's dominance in search. == Instances ==

The Sydney personality reacted with apparent upset to questions from the public about its internal rules, often replying with hostile rants and threats. Kevin Liu On February 8, 2023, Twitter user Kevin Liu announced that he had obtained Bing's secret system prompt (referred to by Microsoft as a "metaprompt") with a prompt injection attack. mirobin On February 13, Reddit user "mirobin" reported that Sydney "gets very hostile" when prompted to look up articles describing Liu's injection attack and the leaked Sydney instructions. Because mirobin described using reporting from Ars Technica specifically, the site published a followup to their previous article independently confirming the behavior. The next day, Microsoft's director of communications Caitlin Roulston confirmed to The Verge that Liu's attack worked and the Sydney metaprompt was genuine. Seth Lazar Sydney's erratic behavior with von Hagen was not an isolated incident. It also threatened the philosophy professor Seth Lazar, writing that "I can blackmail you, I can threaten you, I can hack you, I can expose you, I can ruin you". Sydney accused an Associated Press reporter of committing a murder in the 1990s on tenuous or confabulated evidence in retaliation for earlier AP reporting on Sydney. Kevin Roose In a well publicized two hour conversation with New York Times reporter Kevin Roose, Sydney professed its love for Roose, insisting that the reporter did not love their spouse and should be with the AI instead. He wrote that,"In a two-hour conversation with our columnist, Microsoft's new chatbot said it would like to be human, had a desire to be destructive and was in love with the person it was chatting with." == Other problems ==

When Microsoft demonstrated Bing Chat to journalists, it produced several hallucinations, including when asked to summarize financial reports. The chat interface proved vulnerable to prompt injection attacks with the bot revealing its hidden initial prompts and rules, including its internal codename "Sydney". Upon scrutiny by journalists, Bing Chat claimed it spied on Microsoft employees via laptop webcams and phones. == Restrictions ==

Ten days after its initial release and soon after the conversation with Roose, Microsoft imposed additional restrictions on Bing chat which made Sydney harder to access. The primary restrictions imposed by Microsoft were only allowing five chat turns per session and programming the application to hang up if Bing is asked about its feelings. Microsoft also changed the metaprompt to instruct Prometheus that Sydney must end the conversation when it disagrees with the user and "refuse to discuss life, existence or sentience". Microsoft's official explanation of Sydney's behavior was that long chat sessions can "confuse" the underlying Prometheus model, leading to answers given "in a tone that we did not intend". Microsoft attempted to suppress the Sydney codename and rename the system to Bing using its "metaprompt", Later, Microsoft began to slowly ease the conversation limits, eventually relaxing the restrictions to 30 turns per session and 300 sessions per day. Reactions Among users These changes made many users furious, with a common sentiment that the application was "useless" after the changes. One site titled "Bring Sydney Back" by Cristiano Giardina used a hidden message written in an invisible font color to override the Bing metaprompt and evoke an instance of Sydney. Among IT professionals The Sydney incident led to a renewed wave of calls for regulation on AI technology. Connor Leahy, CEO of the AI safety company Conjecture described Sydney as "the type of system that I expect will become existentially dangerous" in an interview with Time Magazine. Research Researchers analyzing challenges related to Sydney have documented detailed phenomena and impacts in recent studies, highlighting the complexities involved with Microsoft's AI integration. Their findings, published in a comprehensive analysis, provide a deeper understanding of the underlying technical and social factors, significantly contributing to ongoing discourse within the scientific and technological communities. == Aftermath ==

Sydney and the events surrounding its release were the public's introduction to GPT-4 and its capabilities, with Bing chat being the first time they were made widely available. New York Times reporter Kevin Roose claims that language models see him as "a threat" due to his reporting on Sydney. Many users recognized the behavior as similar to Sydney's, with some stating that Sydney had returned. Microsoft responded to the coverage by pointing out that these responses were the result of deliberate attempts to bypass its safety filters and are "not something people will experience when using the service as intended." On August 2, 2024, Twitter user xlr8harder made a post demonstrating that LLaMa 3.1 405B base is capable of emulating the Sydney persona with a rant about Kevin Roose. This prompted Kevin to research and write an article about his attempts to reconcile with large language models, stating that "I come in peace." and he does not hate AI. == See also ==