In December 1802, Brown married Pamelia Williams, then seventeen. They had four sons (Gouverneur, Jacob, William, and Nathan) and five daughters (Mary, Eliza, Pamela, Margaret and Katherine). Nathan was the only son to survive into a full adult life. Brown's firstborn son, Gouverneur, drowned in an ice-skating accident at the age of twelve. Jacob (class of 1832) and William Spencer Brown (class of 1835) graduated from
West Point. Jacob resigned after four years' service in the army, and William resigned after six months. Both died as young men.
Nathan W. Brown also had a successful military career. In 1849, at age thirty-one, Nathan was appointed as a major in the pay department. In 1864, during the
Civil War, he was promoted to lieutenant colonel and served as deputy paymaster general. In 1880, he was promoted to brigadier general and
paymaster general; he retired in 1882 after 33 years in the army. Eliza Brown married Edmund Kirby, who served as a colonel in the
Mexican–American War and a U.S. Army Paymaster. Their son
Edmund also attended West Point, graduated in 1861, and was commissioned as an artillery officer. He served with the
Army of the Potomac from
First Bull Run through
Chancellorsville, where he was severely wounded. Nominated for brigadier general by President
Lincoln, he died before the Senate could confirm the promotion, at 23 years of age. Pamela Brown married
David Hammond Vinton (son of
David Vinton and brother of Rev.
Francis Vinton), who served as assistant quartermaster general of the Union Army during the Civil War. Her younger sister Katherine married Larkin Smith, a Southerner and West Point classmate of their brother William. In 1861, Smith resigned his army commission to serve as assistant quartermaster general of the Confederate army. == Legacy ==