Olivier was the son of Raymond de Termes, count of the Termes region of the Carcassonne district, then lying on the border between
Languedoc and
Catalonia. The region was overwhelmingly
Cathar in religion, and Oliver's father was besieged by
Simon de Montfort in his castle at Chateau de Termes in 1210. Having survived the fall of the castle, Oliver made his way south with other refugees from the crusade to the court of Aragon, where he befriended the future
King James I of Aragon. He was also companions in youth with
Raymond II Trencavel of
Albi, and
Raymond VII of Toulouse. ==Military career==