The acid test would be reaching a conclusion that is correct even if it is extremely unpopular, which means being right even when the training data is almost entirely wrong.

For example, Galileo concluded, after observing the moons of Jupiter from a telescope he engineered, that it was far more probable that Earth revolved around the Sun than the other way around. This view was so unpopular that he was forced to recant and placed under house arrest!

If you had trained an LLM on material back then, it would've given you the popular, but wrong, explanation. Due to social and legal pressure, it likely wouldn't even acknowledge the possibility that the Earth revolved around the Sun.

For AI to help us understand the true nature of the universe, it must be able to discard the popular, but wrong, in favor of the unpopular, but right.