The reason why a Trump victory was such a surprise was because of the overconfidence that spectators had in electoral forecasting models and polling data. For example, because of things like 538’s electoral forecasting “the market” was predicting almost a guaranteed Clinton victory. However, Taleb frequently put out there that there were serious flaws with these forecasting methodologies. Despite the fact that Nate Silver’s 538 had slightly more bearish probabilities than things like the New York Times there is no doubt that 538’s Bayesian approach was extremely fragile to new new data i.e. poor out of sample performance. Just to pick on 538 here, as Nate’s work gained the most publicity, the below diagram shows his estimated odds.
There has been a recent surge in the application of machine learning and natural language processing in the field of economics and finance. In particular, in being able to quantify sentiment and what that measure means for asset pricing. In terms of a macroeconomic indicator; one would think that policy uncertainty is pretty explanatory for real investment expenditure. This has been a particularly important question given the short term effect from the UK Brexit referendum; see for example a speech by the BoE member Ben Broadbent (click link).
So I came across this 2013 course in High-Dimensional Data from the National Bureau of Economic Research, and it is pretty cool.
- Prediction with high dimensional data
- Applications using text as data
- Estimating treatment effects with high-dimensional data
- Nuts and bolts: Computing with large data