Castle Stalker - in the Gaelic, Stalcaire, meaning Hunter or Falconer - is believed originally to have been the site of a Fortalice (a small fortified building) belonging to the MacDougalls when they were Lords of Lorn, and built around 1320. The MacDougalls lost their title after their defeat by King Bruce at Brander Pass in 1308 but regained it for a period after 1328. In about 1388 the Lordship of Lorn passed to the Stewarts, the lands including Castle Stalker.

It is believed that Castle Stalker, much in its present form, was built by the then Lord of Lorn, Sir John Stewart, who had an illegitimate son in 1446, and it is reasonable to suppose that he built and occupied the Castle about that time.