The land on which the Tower stood belonged to the de Knowsley family, a branch of the de Lathom family in the 12th and 13th centuries. It was inherited by Isabel Lathom, sister of Sir Thomas Lathom and wife of Sir John Stanley, c. 1390. The earliest record of the Tower as a fortified house is in the licence granted to Sir John Stanley in 1406 to embattle his house of stone and lime. The Tower was used by the Stanleys as a residence and for entertaining. It remained in their possession until 1651 when the 7th Earl of Derby was required to forfeit all his English estates. It was regained in 1665. The Tower during the Civil War was used by both sides, sometimes as a prison, for Lord Molyneux, a Royalist, was confined there in 1651. As part of the forfeited estates of the seventh earl, beheaded in 1651 for his loyalty to the Stuarts, the Tower was sold; but the purchaser, Alexander Green, was perhaps acting for the Stanleys. At any rate, the place was recovered by the eighth earl, though Alexander Green was the tenant in 1663. According to the Hearth Tax return; he had been Mayor in 1660. About that time, under leases from the Corporation, people began to build houses on the river side of the Tower, to the annoyance of the Earl of Derby, who complained about it in 1669, but without redress. The Greens and their representatives, the Athertons, retained some interest in the building and transferred it in 1682 to Thomas Clayton, a successful Liverpool merchant and he appears as holding it in the assessment roll of 1708. It is stated that a number of the Jacobites captured at Preston in 1715 were imprisoned in the Tower. There were certainly seventy tried at Liverpool the next year; some were executed, and others transported to slavery in the West Indies.