Just head right across the road and directly into the forest there along the mountain wall, you'll get to them within 30 seconds of running.
|Elber|————————|Road|————————————————————-|Cave with bandits|