I think you are right. I'm a bit confused because their skin is dark in Skyrim isn't it? That was a curse for the Resdayn (Vvardenfell) Aldmer by Azura after Red Mountain. If they are from Summerset Isles then they should not have dark skin….
Unless I'm not remembering the patrols in Skyrim correctly, they seemed pretty dark in complexion…

362 hours! Now we are talking