As for the conflict between story and open world, it's mostly about how the story is handled. The usual mistake is to introduce too much momentum. If you want the player to explore an open world with tons of side quests, you can't have a main story with much urgency. If there is an element of immediate danger, it makes no sense for the player to get distracted by side quests and peripheral content.
That's why Morrowind works much better than Oblivion. In Morrowind you are literalliy tasked with learning more about the world and its history, while in Oblivion you have a Daedra invasion that never really happens.
Sure, you can have a quest to rescue somebody from a burning building. But if you walk away from the fire, the building should have burnt down on your next visit. If the person in peril was essential to the main story, somebody else could have made the rescue. (Wouldn't it be nice to have "competing" heroes saving the day each time the player fails an essential task? Could make an interesting side theme, really.)
