The descendants of
William Pictavus held Burg for less than a century and by 1523, it had passed by marriage to a family with the surname of
Waleis (Wallis). It is most likely these who built an early manor house South of the church, with an adjacent fishpond. It was during their lordship that 'Wallis' was added to the village name.
By 1412, the manor had passed, (again by marriage), into the hands of
Sir William Gasgoine, High Sherriff of Yorkshire, whose family estate was at Garthorpe near Leeds. Sherds of green-glazed pottery found on the site of the early manor house date from the 14thC., suggesting that it may have been abandoned soon after this time.
Thomas Gasgoine, a younger brother of a later Sir William, appears to have lived at Burghwallis. He died in 1554; the brass on his grave, showing him dressed in a suit of armour, can be seen in the church.
During the reign of Elizabeth I, the manor passed into the ownership of the
Anne family and remained with them for 400 years. In the early years, their main residence was at Frickley and the Hall at Burgwallis was used as a Dower house or as the residence of the eldest son. The Anne's sold the Burghwallis estate in 1941.