The duration of her relationship with the king has been much discussed. She was definitely the mistress of James IV during 1496–97, and possibly as early as 1495. Records show her living at
Stirling Castle from 3 June 1496, and from 30 October to March 1497 at
Linlithgow Palace. Her presence, and perhaps a previous similar arrangement for another mistress in the royal houses, was noted by the Spanish ambassador
Pedro de Ayala. Ayala later wrote of James IV:"When I arrived, he was keeping a lady with great state in a castle. He visited her from time to time. Afterwards he sent her to the house of her father, who is a knight, and married her [to a third party]. He did the same with another lady, by whom he had had a son." the Merlioun brothers were building the "King's House" at
Stirling Castle In the summer of 1496 Margaret Drummond was living at Stirling Castle, in the care of Elizabeth Forrester, Lady Lundie, the wife of Sir John Lundy of Lundie. His nephew
Robert Lundie was the owner of
Balgonie Castle and the keeper of
Balgonie Castle. A new bed was bought for Margaret's chamber at Stirling on 3 June 1496. Lady Lundie managed Margaret's expenses. James IV played "
tables", a form of
backgammon with Lady Lundie. Spices and confections sent to Stirling at this time were probably for her. At the time the castle was a building site where John and
Walter Merlioun were working on a new lodging now called the "King's Old Building." Margaret moved to Linlithgow and the care of Sir David Kinghorn. New clothes were sent to her from Edinburgh. Margaret returned to Stirling, and Lady Lundie was given £10
Scots for keeping her for 11 days, and then she went home at the end of March 1497. in 1501 commemorating Margaret Drummond and her sisters. ==Family==