Prior to white settlement, Aboriginal people of the Konongwootong Gundidj clan lived in the local area. The first white explorers to pass through the area were the expedition led by
Major Thomas Mitchell in 1836 who spoke enthusiastically of the landscape's
green hills, soft soils and flowery plains, describing it as ideal for farming and settlement, naming it
Australia Felix. The first white settlers in the area were the Henty brothers, who had landed in
Portland, Victoria in 1834 and who claimed 28,000 hectares between what are now the towns of Casterton and
Coleraine. '
Warrock' Station, a sheep farming settlement, was established in 1841, 26 km north of what would be Casterton. The township of Casterton began on the crossing site of the Glenelg River, the location having been surveyed in 1840, and the first pub, the Glenelg Inn, was established in 1846, with a post office opening the following year. In 1891, a large number of Casterton women signed the
Victorian Women's Suffrage Petition to be tabled in the Victorian Parliament to grant women the right to vote (which was not allowed until 1908). By the 1890s, increasing soil erosion saw wheat-farming around Casterton begin to decline, and it was largely replaced by meat, wool and dairy farming. Casterton's population expanded in the early 20th century, especially in the 1920s with the arrival of large numbers of soldier-settler farmers and during the post-war era in the 1950s. The Rail line to Casterton was closed on 12 September 1977. The town's population began to decline in the 1990s, which was consistent with the statewide trend of decreasing populations in many rural areas and the ageing of the local population. As of the
2021 census, the town had a population of 1,673 with the average age being 58. ==Traditional ownership==