Outlier analysis can’t rely on data to compare to other things because they are in completely different regimes and it’s not clear what differences matter. The extreme take on the incomparability of outliers is that data don’t matter and you should just inductively say which things matter or not and why.
However, it may be possible to use data to support or deny a narrative. People often extrapolate anecdotes about outliers in an intellectually dishonest way. So perhaps more accurate than “you can’t use data when looking at outliers” is “you can’t use data to compare outliers to others in their category but you can and should use outliers to compare them to themselves.” Using data like this is using it in the historical sense, not the scientific sense. Historical narrative is not scientific inquiry or engineering design but it is valuable. Further, this use of data in a narrative sense suggests that Outlier analysis is a historical discipline.