There is a way to stop the repeated CHA debuff, otherwise it becomes a real problem.
I suppose it's well-known: once you have seen "Yaniel" follow the corridor to the right (East), open the door and get in the room just in front (so, always East).
There are 3 tablets with skill checks to pass, in Religion, World and Arcana. Religion is the highest DC at 30, IIRC. Each time you fail, you get to defeat spawning enemies, who can damage your abilities. It's all fun.
(for the story) The first time I went TB mode so I could contain them more easily. Equipped band of WIS, drank owl potion, and I needed to roll a 16. I rolled an 11, fight, a 4, fight, a 2, fight, a 3, fight, a 1…, fight, an 11, fight. Then I gave up, realizing my religious character was down 3 or 4 ability points.
I reloaded and used RTwP. Band/potion again, then I rolled something too low and the usual 2-3 enemies spawned, but the hand symbol was still there and I could continue clicking it without any additional spawns on failure, while my fellows were taking care of the ghosts.
Sure enough, after half a dozen attempts, it succeeded. Same technique with the 2 others, which are easier anyway, I got them on the first attempt.
=>
Don't try TB mode, the hand symbol disappears each time. Use RTwP and keep someone near the tablet to keep trying while the combat is ongoing.
=> Don't neglect religion in Wrath of the Righteous, it's a recurring theme.