So then, the mere observation of fact (immigrants are not white, majority society is white) automatically makes you a racist. Cool.

A white person would be a minority in Harlem. A white person moving into Harlem would reduce the black majority. Noting that statistical fact makes me a racist, right? Gimme more enlightenment!