Witcher 3 TW3 pros...please help - difficulty setting question

I started the base game on Hard difficulty (the setting one level below the most difficult). This is the best game I've played in a while, so trying to savor every morsel. I am probably 1/3rd thru the Novigrad part of the game, so still a lot of awesome gaming ahead of me. I'm getting worried. My concern is my level is getting a bit ahead of some of the quest material, and while the combat was challenging up to this point, it is starting to get easy, and I haven't even maxed out with special gear and all the ability points. So my questions for TW3 veterans: Is it worth a restart so I can change to maximum difficulty? Has anyone experimented with that to determine how much more challenging the combat is on the highest level? Or should I change the way I am doing quests (for example do the main plot line quests first when I'm lower level and do side quests after). I love this game but the last few main plot battles have been really easy, which takes some of the fun out of it for me personally. I'm not looking for Dark Souls...but at least threat of death if I don't utilize the right skills, pots, signs, etc. Any feedback welcome. Kind of need to decide if I should restart now before I advance much further. Thanks!

You're right that the game is not Dark Souls, in TW3 there is no mandatory grind.
Don't restart as quests lower than your level won't gain you as much XP as you'd normally get.
Don't skip side content, TW3 is not GTA5 nor Assassin's Creed 4, every side quest is a piece of a bigger puzzle instead of being just a repetitive and dull filler.
And stop caring for current mobs being easy. There are harder mobs awaiting you ahead.

I advise against trying to do all the side content in one playthrough. It's possible to get over-leveled, and the game is so large you might get burned-out before finishing.

Main quest
Side quest
Hunt a monster
Play Gwent
Travel there
Go back over here
Now travel over here
Repeat

Don't patch the game, kill all the cows and then spend it all on Passiflora and Gwent.
That's what I did.