Why tree-based models beat deep learning-based on tabular data
The autoher think that Random Forest are very good for situations with missing data. And the paper he evaluated implement removing missing data for each columns. The author says that he doesn’t like to do some preprocess for data analysis.
Why do tree-based methods beat deep learning?
- NNs are biased to overly smoothed solutions. Neural Nets based on gradient, the decision boundary of the Neural Nets should be smooth over, but the Random Forest could have a irregular pattern for more precise decision.
- Uniformative features affect more MLP-like NNs. The decision trees are designed to have information gain and entropy when decide the paths to follow.
- NNs are invariant to rotation, actual data is not. NNs are maintaining their original performance, while all other learners actually lose quite a bit of performance.
