The Marquess died in 1858 in a hunting accident, leaving his widow to live what has been described as a "restless life." She travelled between fashionable resorts in Britain and Europe and the Douglas estate, Glen Stuart, in
Dumfries, Scotland. In 1861, she converted to Roman Catholicism, shocking her family. She became aware that her mother-in-law planned to take her children away from her, prompting her to flee to
France with her youngest children, Archibald, then twelve, and Florence and James, aged seven, where she could educate them as she wished. This led the children's guardians to threaten her with an action under English law to take her children away from her. The three were too young to choose a guardian under Scottish law. In the event, they remained in France for two years. Falconer Atlee, the British Consul at
Nantes, offered them a place of safety when their first location was discovered, and the Emperor
Napoleon III eventually extended Lady Queensberry his protection, ensuring that she could keep the custody of the three children. Eventually, it was agreed that Caroline would retain custody of her younger children, and they returned to England in 1864, when her mother-in-law's health started to decline. ==Political activism==