Around 1140, the brothers Schwikher and Otto von Welsperg had the castle built on a rocky outcrop above Welsberg. Its most distinctive feature is the unusually tall keep, believed to be the oldest part of the complex. Over the centuries, the castle was extended and altered several times. As the Lords of Welsperg grew in influence, they had Thurn Castle built opposite Welsperg Castle in the 13th century.
A major turning point came with the fire of 1765, which destroyed large parts of the castle. During the reconstruction that followed, it was never fully restored to its original form. Nevertheless, it remained closely connected to the von Welsperg family for centuries, and its distinctive keep still stands above the village today.