The  opposite castle rock, whose settlement can be traced back to prehistoric times, the Shepherd's Wall was never permanently inhabited. However, people have already come here in prehistory, as evidenced by the finding of two bronze axes from the Late Bronze Age belonging to the Lusatian culture. For centuries, the foothills of the Shepherd's Wall were settled in the present-day Labský Embankment. The village of Weiher, written for the first time in the early 16th century, was situated here. The name Weiher (translated by the pond) was derived from ponds and wetlands, which in these places formed a no longer existing delta of the mouth of the Jílovský stream to the Elbe. Weiher's inhabitants were mainly boatmen and fishermen, but also craftsmen such as basket makers and carpenters. The village gradually expanded along today's Teplická Street towards the Jílovský Valley. Its inhabitants transformed the forests along the Elbe bank to Chrochvice on fields and meadows. Also the southern slopes of Pastýřská stěna and the neighboring Červený Vrch have turned into meadows and pastures. The Shepherd's Wall was probably given its name by now. In 1540 we have a certain Klement Schäfer in Weiher, whose land was "under the rock". The surname Schäfer (shepherd) later passed to the adjacent wall, which was called Schäferwand, ie the Schäfer wall, in the current inaccurate translation of the Shepherd's Wall. This name was used only for the rock wall towering above the Elbe, while the rest of the massif stretching towards the Scrapers was called Rotberg (Červený Vrch). 19th century romanticism gave rise to a legend about shepherd and treasure.

If we get to the middle of the neighboring village of Červeny Vrch, and we have enough imagination, we will see a "head of a sleeping shepherd" in the rock wall below today's restaurant, especially visible in a fine snow powder. The whole rock massif of Pastýřské stěny (or Červený Vrch at that time) was part of the Děčín estate and for centuries it was used as an agricultural forest for logging. On the southern slope of the hill was a manor vineyard, whose name still reminds the name of the street Na Vinici.