I played a fair bit of Nehrim and then lost interest - I'd like to try again, but i remember Oblivion with this mod running rather slowly on my i7 + GTX480 system. Did performance improve with later releases?
All I can say is that I played it with a Dual Core AMD and a GF 8800 and it ran fine back then (not on highest settings). Guess something is wrong with your system or your memories play a trick on you.
The only performance issue I had with Nehrim was slowdown in this certain forest, which is well documented. You have to turn your distant views off or to the lowest setting. Other than that the game played great on max settings. The FPS would dip a bit in places like the middle of Erothin, but it still stayed very playable.