From the Wikipedia page 
Middleham Castle in Wensleydale, in the county of North Yorkshire, was built by Robert Fitzrandolph, 3rd Lord of Middleham and Spennithorne, commencing in 1190. It was built near the site of an earlier motte and bailey] castle. In 1270 it came into the hands of the Neville family, the most notable member of which was Richard Neville, 16th Earl of Warwick, known to history as the "Kingmaker", a leading figure in the Wars of the Roses. Following the death of Richard, Duke of York at Wakefield in December 1460, his younger sons, George, Duke of Clarence and Richard, Duke of Gloucester, came into Warwick's care, and both lived at Middleham with Warwick's own family. Their brother King Edward IV was imprisoned at Middleham for a short time, having been captured by Warwick in 1469. Following Warwick's death at Barnet in 1471 and Edward's restoration to the throne, his brother Richard married Anne Neville, Warwick's younger daughter, and made Middleham his main home. Their son Edward was also born at Middleham and later also died there.
Richard ascended to the throne as King Richard III, but spent little or no time at Middleham in his two-year reign. After Richard's death at Bosworth in 1485 the castle remained in royal hands until the reign of James I of England, when it was sold. It fell into disuse and disrepair during the 17th Century. It was garrisoned during the English Civil War, but saw no action. The ruins are now in the care of English Heritage.
See also Middleham