The difference is that Elon is not a politician. His opinions on EVs, rocketry, and free speech (Tesla, SpaceX, X) are pretty robust. His opinions outside those fields may not be, but they are not relevant because he does not operate in those areas. Just like a brilliant neurosurgeon may have some weird views outside of neurosurgery.
Politicians should be under much closer scrutiny because they devise policy in areas that encompass vast swaths of society, far outside each's individual expertise.