Algernon Greville was the second son of Fulke Greville, 5th Baron Brooke, son of Robert Greville, 2nd Baron Brooke, and his wife Sarah Dashwood. He married Mary, daughter and coheir of Lord Arthur Somerset, the youngest son of Henry Somerset, 1st Duke of Beaufort. Their daughter, Mary, who married Shuckburgh Boughton in 1736, had Sir Charles William Rouse Boughton, 1st and 9th Bt.