Table Of Content

Recap
Backtests Are Not For Research
A Word About Fitting
Other Uses Of Backtests During Design
Alternative Approaches
Edge First
Correlation With Future Returns
Using Synthetic Price Data
Out Of Sample Testing
Walk Foward Testing
Refined Workflow - Idea First
Risk Premia & Risk Factors
Momentum Risk & Value Risk
Trend Following Rationale
Mean Reversion Rationale
Trend Following Example Signals
Mean Reversion Example Signals
Next Weeks Issue

Recap

We had a look at the basic structure any serious trading strategy should have and crafted a first draft for our own to get a better feel for the C.O.R.E. of our trading business.

This week we're going to talk about the O - Opening Rule in more detail.

But before we can do that we need to talk about the Table of Contents of Marcos López de Prado's book Advances In Financial Machine Learning, specifically this section:

Headings like The Dangers of Backtesting and Backtesting Is Not a Research Tool should get your head spinning right away.

What does this exactly mean?

Backtests Are Not For Research

The typical retail workflow of designing a trading strategy usually looks like this: select trading rules,
run a backtest,
look at its performance,
change parameters of trading rules,
run the backtest again,
compare performances,
select better perfoming set of parameters,
rinse and repeat until you got the absolute best set of parameters.

This is dangerous!

What you have to keep in mind is that financial returns are often pretty random. True, there are concepts like different market regimes (trending vs. mean-reverting), fundamental reasons for long-term price movements, and events driving short-term volatility. But overall, financial instruments usually exhibit some kind of random walking behaviour with periods of drift.

Remember our discussion about the standard deviation (volatility)? The price graph of any financial instrument you can pull up in a charting tool like Tradingview is just one random example path out of a bunch of possible outcomes within a distribution. Each time the daily candle closes, next days close lands somewhere in the distribution range based on the instruments current volatility.

To visualize this (oversimplified) you can slap the Bollinger Band Indicator on your BTC price chart. It plots the volatility range (std_dev * 2) around the 20 day moving average of price. Each days candle close could've happened anywhere between the upper red and lower black band.

When you're solely relying on historical price data and its backtest results to select the best performing parameters for your trading rules, you're not really researching trading signals with strong predictiveness or fundamental logic about how to extract value from the markets. You're overfitting your trading rules to a specific historical pattern which may not repeat in the future.

That is even after accounting for common pitfalls like we've discussed in the 4F's of backtesting framework!

This makes data-driven backtesting a bad tool for signal research because it does not provide real insights into why a particular strategy made money. The only thing it tells us is that a specific set of parameters and trading rules were lucky enough to catch a big draw in the past. Or as De Prado puts it

"Instead what they have found are the lottery tickets that won the last game. The winner has cashed out, and those numbers are useless for the next round."

A Word About Fitting

A data first approach often leads to overfitting. As soon as you make a decision based on the performance of your backtest, you're fitting your strategy to the data sample used in that backtest. In Sample Fits are rarely a good idea, especially when doing multiple iterations!

This doesn't mean that fitting in itself is bad though. It's a natural and necessary step in the design process and if done right can lead to improvements in the strategies overall performance. But you should keep the number of times you've fitted based on historical price data to the absolute minimum. There are better places for fitting.

Okay, so what are backtests good for then?

Other Uses Of Backtests During Design

Another approach to use backtests is not to improve strategies but to discard bad ones. They work great for tasks like checking parameter sensitivity, confirming cost assumptions or validating your strategies behaviour in different scenarios.

If you have a trend following strategy, does it behave like a trend following strategy should? Does it make money in periods where trend following was generally profitable? Does it lose money under choppy conditions?

And if so do the skew and equity profiles check out? Since trend following strategies are usually positively skewed, they would likely consist of occasional big jumps in equity adjacent by slow bleeding during chop. Mean reversion strategies on the other hand are often negatively skewed so their equity curves should look more like lots of slow grindups into occasional big drops. (More on this later)

In addition it's worth noting if it gives bullish signals in bullish environments. The same goes for bearish forecasts.

And last but not least: if your model is supposed to catch longer-term trends, is your trading signal also changing course slowly?

If your strategies behaviour fails to align with any of these assumptions during a backtest, the chances of the strategy being a false positive increases!

None of these topics are concerned with the actual performance of your strategy. Consulting real market data is only done to verify that your trading strategy is behaving like it should in this approach. It never uses backtest until the trading signals have been fully developed. If the backtest then fails, start the design process from the beginning!

Alternative Approaches

There's a bunch of different techniques you can use to improve this data first approach. Let's start with probably the best option:

Edge first

Instead of focusing on what trading rules and parameters have performed best in the past, you should come up with an idea on how to extract value from the markets instead. This idea should be based on your fundamental understanding of how markets work and describe why it makes sense for you to get paid. This is often referred to as having an edge or simply alpha.

To be able to come up with alpha ideas we need to know about possible sources of returns. But lets quickly talk about some other techniques before we go into more detail on edges.

Correlation With Future Returns

The goal of this analysis is to better understand your ideas impact on your strategy. Understanding the rationale behind your edge is only one part of the problem. You definitely can and should go deeper!

The purpose of a trading rule is to generate buy and sell signals. They are basically forecasts about what will happen next so you can position yourself accordingly to extract value. It's relatively easy to check if your trading rule does in fact have predictive power, how strong that predictive power is and its effect on your overall equity curve.

Using Synthetic Price Data

Using synthetic price data instead of real prices during the first stages of the design process is a common technique in the ML for finance world. It completely eliminates the chance of overfitting to a particular set of real market data so it's definitely worth taking a look at.

Synthetic price series can be constructed many different ways with different levels of complexity. Depending on what characteristics you want the price series to have you'd usually choose one ore a combination of the following algorithms: Geometric Brownian Motion (GBM) to create trend-like or random-walking behaviour, Ornstein-Uhlenbeck for mean reverting environments or even something more sophisticated ones like Markov Regime-Switching Models or full on ML models like Generative Adversial Networks (GAN). A great example for GANs can be found in Stefan Jansen book (Machine Learning For Algorithmic Trading).

No matter what you choose, the goal is to create a realistic price series to train your model on before exposing your ideas to real market data.

Out Of Sample Testing

When Out-Of-Sample (OOS) Testing you split your data in 2 chunks: The In Sample (IS) chunk on which you fit (or train) your trading rules and the Out Of Sample (OOS) chunk on which you run the IS fitted rules again and then compare their performances. If the OOS perfomance is vastly different to the IS perfomance, your strategy might suffer from overfitting.

True OOS testing can only be achieved by using synthetic price data, papertrading or livetrading and then observing your trading system day by day live to compare it with your backtest results.

Walk Foward Testing

The most common cross validation method for backtesting is probably Walk-Forward testing. It is a special kind of OOS testing, where you constantly resplit the OOS and IS chunks in a sliding window fashion throughout the price series. The parameters themselves are constantly adapting to the changing data which helps eliminate some data-snooping bias.

Graphic taken from the book Building Winning Algorithmic Trading Systems by Kevin J. Davey

Refined Workflow - Idea First

Taking the points disussed into considerations we can alter our design approach:
come up with an idea of how to extract value from the markets,
test and fir on synthetic price data
Test it on real data as a reality check

So far, so good.. But how do we come up with ideas?

To come up with ideas for potential edges, we need to be clear about what we're going after. What market behaviour are we going to "exploit" to generate returns? Why would we be able to use this behaviour to our advantage and why would it continue to work in the future? We need to know how we will earn money!

Risk Premia & Risk Factors

One way to earn money in the markets is by farming the so called risk premium. The risk premium is the reward you're earning as compensation for taking on more risk than with a risk-free investment like buying the market index or T-bills. The difference between your returns and the returns of the same amount into a risk-free investment is your extra reward, also known as excess returns.

There are a hand full of established and publicly known risk factors like momentum, value, volatility to only name a few. Each of which with a lot of reasoning and more or less decay attached to them.

The following list is not meant to be conclusive. It only scratches the surface but is a good example on how to approach the idea generation process.

Momentum Risk & Value Risk

A simple example on how to extract value from momentum is Trend Following, in which you bet on the assumption that assets that go up or down in price will continue to do so until there's a shift in momentum. When trend following, you're exposing yourself to the risk of a trend reversal or momentum crash against you.

Another extremely popular category of strategies is Mean Reversion. Its goal is to identify when prices or other metrics have deviated too much from some average and are likely to return to it soon.

While they both work towards the goal of extracting returns from the market, they do so in a different manner, which is also visible in their equity graph and skew structure.

The equity curve of a trend following strategy usually shows a positively skewed formation with lots of small losses accompanied by occasional large spikes to the upside while mean reversion looks more negatively skewed with lots of little gains and a few big drops.

Trend Following Rationale

Trend following or betting on momentum relies on the fundamental concept of supply and demand imbalances due to several factors:

Humans often exhibit herding behavior, especially in uncertain or complex environments like financial markets. This means that individuals tend to follow the actions of others, assuming that the group is better informed. When a trend starts, more and more investors jump on, reinforcing the trends momentum.

Another catalyst for trends is the different reaction time across participants to new information. Financial markets operate on information that doesn't get absorbed by all participants at once. There usually is some delayed reaction to new information, creating trends as prices gradually adjust to rising buying or selling pressure.

Institutional investors like hedge funds may ignite trends when entering or exiting big positions due to a general lack of liquidity for the market. Their trade activity helps drive and sustain trends this way.

Mean Reversion Rationale

One type of popular mean reversion strategy is trading correlation or cointegrated pairs of instruments. Even though prices are mostly random-walking, it is possible to find different pairs of stocks or other instruments that reveal a cointegration of their paths when plotted against one another, effectively spawning some kind of stationary series which sometimes deviates from its initial starting point but then quickly contracts back.

Probably the simplest example for such a strategy are trading stocks of companies that are competing in the same sector. Like Coka Cola and Pepsi (don't take our word for it, test for it! We will show you the code to do it later.) Or some related ETFs based on the same underlying like the Gold ETF $GLD and the Gold Miners ETF $GLX.

When looking for mean reversion opportunities in cointegrated pairs you're trying to harvest a value premium by longing the undervalued asset and shorting the overvalued one, expecting their spread to converge again. The core risk of this approach is that assets deemed mispriced might not revert to their 'fair' price due to fundamental or other reasons, which can be hard to model for.

Pair trading increases your capital requirements because you need to assume at least 2 opposite postions so depending on your bankroll it might be harder for you to construct a meaningful portfolio of instruments to trade.

Relying on only 2 instruments can also greatly increase your overfitting risk during the design process. Overall MR pair trading is a bad fit for beginners from our point of view. The negative skew profile makes things only worse for new traders. Expecting large losses by design could punish you even harder with lacking risk management.

Trend Following Example Signals

To be able to spot shifts in momentum we need to somehow measure momentum or something else that can give us a hint about a trend starting or turning.

A popular choice amongst technical indicators is the Moving Average Crossover. It consists of 2 Moving Averages of price, one with a faster measurement (smaller lookback window) and one with a slower one (bigger lookback window).

Since the smaller lookback window is reacting faster to price changes, each time it crosses the bigger one, a shift in momentum took place. It's not the offspring of the trend though because moving averages are lagging indicators. An MA(14) needs at least 14 consecutive price points before it can form so a lot of price movement already happened before it reacts.

Another trend following indicator is the Bollinger Band. It calculates and forms a channel around the price using the standard deviation of the instrument. Breakout strategies are trying to capture when price broke out of a previous range based on price, volatility or other drivers.

Mean Reversion Example Signals

As we said, we believe these types of strategies aren't suited for beginners but we'll give an example nonetheless. Throughout this series we're going to take a look at how we can approach mean reversion to make it even worth it and what we actually need to be able to trade it.

A very common choice for spotting mean reversion opportunities is the use of Z-Scores. They quantify the distance of prices (or other factors) of two pairs to their mean in terms of standard deviation (volatility) and have customizable thresholds for being too high (mean revert back down) or too low (revert back up). If the two pairs being scored have different price units, the scores need to be normalized.

Next Weeks Issue

Next week we're going to take some of these concepts and mix them together so we can use them as improved ingredients for our C.O.R.E. recipe.. Throughout this series we're going to test a bunch of different trading ideas to see if we can find an edge and then implement them on a budget retail live trading account.