The popularity of algo trading has exploded with the democratization of trading platforms, python packages, and machine learning. A quick search for algo trading brings up hundreds of posts and scholarly articles. But something isn’t quite right.
As you look at all the ways to “predict stock prices with neural networks” you’ll notice that the prediction is just a naive forecast of the current value. The “buy sell hold” decisions from deep reinforcement learning never seem to work out of sample. The models just don’t learn anything meaningful.
This doesn’t come as a surprise to anyone in the quant trading industry. It’s an open secret that AI hasn’t been successful at extracting trading signals or price prediction. If it were that easy, anyone who can install scikit learn and pandas would be a millionaire.
For centuries, there have been slick salesmen promising get rich quick schemes and passive income, but no other entrepreneurship niche is a more wretched hive of scum and villainy than the retail finance industry. From seniors hoping for a more secure retirement, to young people hoping to avoid the drudgery of ascending the corporate ladder, there are always new entrants hoping to learn a skill that will change their lives.
These aspiring traders buy books on technical analysis and pattern recognition, purchase courses, and spend hours watching trading strategies. Then they take these systems to the markets where they quickly donate all their money to a brokerage firm and smart money on Wall Street.
The content producer gets ad revenue from videos, courses, books, and commissions for sending traffic to brokers. Brokers get paid their commissions. The retail traders are wiped out.
The question is why? Why is it more profitable for so many to peddle nonsense than to trade?
The bottom line is the lack of a meaningful, persistent statistical edge.
Whether it’s the RSI, MACD, candlesticks, Fibonacci, Heiken Ashi, Renko, Elliot Wave, Gartley, the beloved SuperTrend, or countless others, all indicators work some of the time and fail miserably at others. Within a year or two, anyone who has studied well should be able to piece together strategies that make money over a year timeframe. But the strategies fail to be profitable over a long time horizon. That, or the edge is so rare that, even though it works well, it doesn’t make enough money to build real wealth.
After endless backtesting, they still never seem to have a combination of win rate and profit ratio with positive expectancy over a long enough time to make real money.
At this point, many decide to learn a coding language to remove the emotional, “human factor”. This just systematizes their doomed strategies so they can lose money faster.
Often in desperation, they turn to something they do not fully understand — Machine Learning. Surely the most advanced methods of statistical learning will provide an edge!
Sadly, this is not to be. Market data is highly noisy, low dimensional, not identically distributed, and its price derived samples aren’t independent. Too often the samples are not even mutually exclusive. Using technical indicators as features for these models just leads to the most brutal overfitting.
Most give up. Those that remain wonder, “Am I wasting my time?” “Is this even possible?” “Are economics professors right about the market being efficient thus rendering my Herculean labors in vain?”
Yes and no.
Profitable retail trading is real, but it diverges down two paths. One is manual trading combined with fundamental catalysts. The fundamentals give a reason to expect volatility in a particular direction. Technical analysis is used as confirmation and a more precise entry signal. Combined with good risk management and lots of practice, one can build the true secret sauce of professional traders of this kind — discretion.
After watching hundreds of hours of setups and a thorough understanding of fundamentals, one begins to understand what markets will react to, what’s already priced in, and what will be a surprise. One can look for false narratives and exploit imbalances in positioning. In other words, one can use fundamentals to find the situations where technical analysis works. Fundamentals act as a filter for technical analysis.
The other path is quantitative. Let go of the dream of pointing an ml algorithm at price data and learning good regressions or profitable classifications.
OHLC data is fundamentally unsuitable for statistical learning without heroic transforms and feature engineering.
That is where the real work is. Focus on extracting the signal from the noise with transforms, signal processing, and statistical tests. Validate that edges are real and persistent. Combine these elements into a strategy and win. If this second path is of interest to you, join me on my journey of real innovation in this space.