Raby Castle is situated at Staindrop in County Durham. The origins of Raby date back to its occupancy by the Nevills. Construction of the Great Hall, for instance, is thought to have been started in the 1320s, in the time of the 1st Lord (Baron) Neville (of Raby), who died in the spring of 1331. It continued under his successors, the 2nd and 3rd Lords, who reigned 1331–67 and 1367–88. It was during this period that the Great Hall was extended north and south and towards its end that such luxuries as kitchens were added, these dating from the late 1370s. The man responsible for the craftmanship was John Lewyn, a master mason who also worked on Durham Cathedral. Raby was not however the family's chief stronghold due to its low-lying position and difficulty in defending.

The traces of its earliest occupants can still be found at Raby and include the Neville Gateway, Joan's Tower and the Bulmer Tower on the east flank. Other notable features dating from the Middle Ages include Clifford's Tower, at 80 feet the tallest of them all, and the Kitchen Tower, with its deeply cut frontal trench running vertically up and down.

With the eclipse of the Nevills following the disastrous rising of the Northern Earls in 1569 Raby was taken into the possession of the Crown. In 1613 JAMES I granted it to his favourite Robert Carr, Earl of Somerset, but with Carr's disgrace it reverted again to the Crown. Sir Henry Vane the Elder, whose cousins preferred the spelling Fane and had conferred on them the formerly Nevill-held Earldom of Westmorland, took a lease on Raby in 1616 and bought it outright ten years later. Whatever work was done in the following century has left little trace and it is thought Vane's chief purpose in acquiring the estate was the land. But he did keep the castle in reasonable repair, cannibalising Barnard Castle, which he pulled down, and using its stones on Raby. He also added an external arcade on the east side of the courtyard, its balustraded roof serving as a balcony for diners sallying forth from the Great Hall above. He also regularised the appearance of the central block by inserting a two-storey, five-bay facade with triangular pediments and adorned by another balustrade.

Raby was much knocked about during the Civil War and again at the beginning of George I's reign by the 1st Baron Barnard, apparently out of sheer malice towards his son Gilbert, later 2nd Baron. Two hundred workmen were brought on site to strip the lead off the roof, the glass from the window frames and the doors from their hinges. They even ripped up the floors. Trees were felled in their hundreds and the parkland was let to a farmer, who ploughed it over. But it was only in the mid-19th century that Vane's alterations, which sound on the whole rather agreeable, were swept away, this time by the well-intentioned but in fact far more destructive 'improving' hand of William Burn. With Gilbert's accession a golden age at last dawned. He had won an injunction against his father to prevent any further damage and brought in James Paine, whose splendid, gently curvaceous Hunter's Gallery at Raby has been called the Morning Star of the Gothick Revival.

Work by Paine that survives includes the Stucco Room and much of the interior of the west wing between Joan's and Clifford's Towers. Gilbert's son the 3rd Baron, promoted Earl of Darlington the year after his father's death, employed a less distinguished north-country architect, John Carr of York, at first using him for minor alterations but gradually working up to large-scale projects, notably the extraordinary transmuting of the hall into a massive vaulted chamber which would allow a coach and horses to enter. This was all the more necessary because the courtyard was proving too small for 18th-century vehicles. Carr also worked on the Round Tower, creating an additional drawing room. His plans for an entirely new facade, however, were never executed.

The early 19th century saw the doing over of Carr's drawing room in the Chinese taste for a projected visit by the Prince of Wales (later George IV). But the principal alterations were much later, around the mid-century, when Burn began his association with Raby. He removed medieval vaulting from the Bulmer and Clifford Towers, inserted over-large window glazing, made copious use of new stone which contrasted starkly with the mellowed fabric of the older work, and in particular demolished a wooden roof over the Baron's Hall.