Yes, scary. That's what I meant by unreliable. Those tools seem unable to say 'I don't know', which I understand since it's only a process returning the closest similar pattern. Although the author of the video showed a case where the tool couldn't find data for 2 of the 5 cases it had to summarize and simply said so, leaving blanks instead of making something up. But in my experience, I've had references that were completely invented, several times. For example, books with fancy ISBN numbers or with titles that simply didn't exist. When I asked the AI about it, it simply said 'sorry, you're right, it doesn't exist'...I'm more worried about the fact that that these LLM will be used in a whole heap of fields irresponsibly. And even though I hope people keep in mind that you always have to double check what they generate; basically don't trust anything it produces without double checking, at a certain point that will be lost I'm afraid. It has too good of an ability to seem anthropomorphic, especially when people don't scrutinize it properly.
If the AI could at least give a certainty ratio with its answers. It wouldn't be entirely accurate since it ultimately depends on the credibility of the original facts it absorbed, but it would be a good indication about its own reasoning.
Thankfully, we're already getting more critical because of the Internet. I remember people being much more gullible before (and yet, journalists weren't more cautious about checking what they wrote than today).