And I got to the ruined elven baths, which I've read is part of the main questline. But I'm not there yet. But what annoys me is the way they handle this. I can go down to the wall that blocks access to the baths, destroy the wall with Aard but I can't get down because of either an invisible wall or because I don't have the "jump down" prompt.

Couldn't they just not allowed me to break the wall? It really breaks immersion when you see a beautiful place, which the elven baths seems to be, and not be able to explore it because of an invisible wall. Dammit. And I was having so much fun exploring.