and I haven't suffered any ill effects yet.
This is something you can only know if your ear doctor tests it with a frequency synthesiser.
The important thing to remember about ears is:
Many structures inside the ear are too small to repair them. If they are damaged you are fucked.
If something happens, especially an acute hearing loss, visit a doctor immediately. Within the first 24 hours they can try to contain the damage by improving blood circulation and heat, relying on your body´s self-repair capabilities. After that you have ca. 6 weeks to try vacuum chamber therapy - with decent chances.
I hear a constant noise which sounds like a the one a shell makes when you hold it to your ear. 24 hours a day. And I can consider myself lucky because it´s rather unintrusive and has a low volume, so that just about every external noise overshadows it. Other people hear a loud BEEP - BEEP - BEEP the whole day ...
Since then I never go to loud places without earplugs. Never.
Of course all this isn´t necessarily connected to headphones. But the difference I see between headphones and speakers is:
The danger for your ear comes from two things: volume and abrupt changes between audio wave mountains and valleys. In a room full of speakers you can choose a place at which you "feel" safe, which means where you think the overlapping mountains & valleys caused by the different speakers more or less neutralise itself. And you can use earplugs to get the noise down and filter certain frequencies out.
Headphones expose you to the full dosis everytime. You can´t avoid it and you can´t run away. This is probably no problem if you turn the volume down, but it´s easy to forget it, especially if the software combines quiet voice overs with aggressive effects.