Colonel James Butler was the son of Richard Butler of Galmoye and the grandson of Edward Butler, 2nd Viscount Galmoye. He was the heir of Piers Butler, 3rd Viscount Galmoye who died without male issue. However, Colonel Butler's succession to the viscountcy was not recognised. This was because an act of the Irish Parliament had declared the titles of the 3rd Viscount forfeit in 1697. He was a Colonel of the Irish Brigade in the service of France.