Games hasn't evolved much since Pac-Man in terms of A.I. There's no developer who's even the least interested in it. It has even gotten less interesting to most of them than e.g in the 90's.. i'd say shooters overall has worse A.I than before, its probably because it can be quite CPU heavy and its "better" for the CPU to handle other stuff than advanced A.I.
GalCiv 2 had decent A.I, Galciv3 is much worse (dev has no problems to admit it).. This is very common for games, that it only gets worse.
It's really up to the players to demand it and it's up to reviewers to give a bad score if A.I is bad, but no one cares, everyone expects it to be bad or just "like normal". Kind of funny really, graphics has not evolved like that at all. everyone absolutely demanded good graphics, shitty graphics could even give a very poor overall score, but for A.I no, just expect it to be shit and be happy, 10/10.. lol