On 17 October 1859 at
St Paul's Church, Knightsbridge, Irby was married to Hon. Augusta Caroline Saumarez (1841–1929), second daughter of
John Saumarez, 3rd Baron de Saumarez and, his first wife, Caroline Esther Rhodes (daughter of William Rhodes of Kirskill Hall,
Yorkshire, and
Bramhope Hall, Yorkshire). Together, they were the parents of: • Hon. Cecil Saumarez Irby (1862–1935), who married Florence Augusta Dormer, daughter of Clement Upton-Cottrell-Dormer of
Rousham House and Florence Anne Upton, in 1885. who married Gordon Frederic Deedes, eldest son of Rev. Canon Gordon Frederick Deedes, in 1890. • Hon. Gilbert Neville Smyth (1864–1940), who married Esmé Edwards, daughter of George Oldham Edwards and Emily Frances Way, in 1891. He was succeeded in the barony by his eldest son,
George. Upon his death without issue in 1941, the barony passed to
Greville Northey Irby and
Cecil Eustace Irby, both sons of his second son, Cecil. ==References==