Skip to Content

Bringing it Backtesting

People don’t often think of finance professionals as creative. You know the stereotype; suit and tie or pencil skirt and blazer types. Very buttoned up and boring. I’ve been on my own personal journey to find creativity at work and within the finance industry. The freedom, and expectation, to be creative is one of the reasons I joined Composer.

However, I think there is a reason for this finance stereotype. People in finance, and particularly those in investment management, are responsible for investors’ money. That is a huge responsibility. And if these people are any good at their job, they take this very seriously. This seriousness can lead to a perception that many in finance, and their solutions, are boring.

To be crystal clear, there is nothing wrong with boring. Warren Buffet’s advice to put most of your money into VOO, Vanguard’s S&P 500 ETF, is really, really good advice (check out Composer’s Buffet symphony). It’s easy to understand and implement, low cost and tax efficient, and investors can set it and forget it. I’m gushing, but I love simple, effective solutions. 

I also believe that there is room for creativity in investing. And obviously, there is much more to the world than the S&P 500. One of my takeaways from the rise of retail investing is that people are looking to be more involved and creative with their portfolios. They want to try new things, express personal views, and test their mettle against the market. I think that is beautiful. 

Yet, retail investors face the same challenges and must play by the same rules as the professionals. What do I mean? Well, costs, taxes, diversification, efficient markets, risk management, time horizons, and risk tolerances all matter. If you’re going to jump into the fray, you need to be armed with the right tools. 

One of the most powerful tools for investors is backtesting. So, let’s dig into backtesting, its issues, and how you can harness its power.          

Investing with a time machine

Backtesting is a method of evaluating an investment strategy by looking at how it would have performed in historical market conditions. It lets you see what it would have been like to go back in time and implement that idea in the past. It’s time travel. 

The implicit assumption for backtesting, and this is a critical point, is that the future will resemble the past. Someone reading this, who is much more detail-oriented than I am, is saying, “Hold on. Isn’t there a disclosure at the bottom of all your blogs that says something like, past returns are not indicative of future performance.” Yes, that is true. Both the existence of the disclaimer and the statement. Just because something has happened in the past does not mean it will happen again, and relationships between assets and economic factors change over time. 

But, that doesn’t mean backtesting isn’t useful, though! It’s employed by professionals all the time. According to a CFA Institute survey, over 50% of analysts used backtesting in the past 12 months. Why? Because backtesting provides data, rigor, and insight that wouldn’t otherwise be available. And, if you are exceptionally good at processing that data, perhaps you can carve out a slight edge. Jim Simons, the founder of the hedge fund Rentech, said, “If we have enough data, I know we can make predictions.” [1]

To evaluate a simple strategy that combines a handful of asset classes (e.g., US stocks, emerging market stocks, bonds) it's relatively easy to develop a perspective on the risk and return profile of the strategy. You could calculate expected return and volatility by hand with projections from an economist. And while the calculations might get messier as you add more assets, it is still relatively straightforward.

But, how do you evaluate a strategy that is dynamic? Take a strategy that is weighted by volatility or one that switches between two assets. The assets in these examples will change in response to prevailing market conditions. For more complex strategies, it’s difficult to know how it might perform once it’s live. How much volatility should I expect? What's a reasonable rate of return? Who knows!

This is where we believe backtesting can help. [2] If we assume that the strategy was implemented in the past, we can play the tape forward and take note of how it performed. Now we have returns, standard deviation, Max Drawdown, and other metrics for our symphony. 

Even better, we can compare those characteristics to benchmarks [3] and other symphonies that we are considering. Backtesting gives you the power to test and iterate on investment strategies before you start trading. You can be creative!

Backtesting software allows investors to evaluate hypothetical performance before they invest.

Composer's backtester in action

So what’s the catch? 

As we discussed above, backtesting is like time travel. However, this introduces a number of biases and issues that we as investors must fight against.

If I gave you a time machine, what would you do? Go back and invest in Apple and Amazon when they first went public? Steer the Titanic around the iceberg? Stop the Philadelphia 76ers from drafting Ben Simmons? Ok, that last one is personal. The point is, you know how the story ends, and you would make decisions based on what you know. This is called look-ahead bias.

Let me give you an investing example. What if I wanted to test if overweighting growth stocks was a prudent investment strategy? I can use Vanguard’s Growth ETF (VUG) and I will compare performance to the S&P500 (SPY) and Vanguard’s Value ETF (VTV). Going back to the beginning of 2005 through today, it looks like tilting towards growth has been an excellent strategy. Growth (VUG) has returned almost 2% more per annum than SPY and more than 3% more per annum than VTV. So let’s load up on growth and ride this wave for the next 18 years!

A comparison of growth and value investing strategies relative to the S&P500.

Data from January 1, 2005 to February 22, 2022

Well, not so fast. I knew growth outperformed over this period. I experienced it as an analyst overseeing some of the largest value-focused mutual funds in the US. It was a rough stretch. The question we want to answer is whether growth will continue to outperform.

Perhaps an even better question is: do we believe that the factors that led to growth’s outperformance over the period will continue, and then based on that, do we have conviction in this strategy? Ah. Now we are at second-level thinking, and I can plug one of my favorite writings on investment strategy: Howard Marks’ memo titled It’s Not Easy.

Another, and somewhat related issue is data-snooping. This bias involves testing multiple strategies and picking the best one. Investors get hooked on tweaking and optimizing their backtest, which leads to overfitting. You can see how this bias leads investors astray. By picking the best performing strategy, investors think they are setting themselves up for success, but they must remember that BACKtests are BACKWARDS looking. Why should they believe that the strategy they have selected will continue to outperform in the future? 

Better backtesting

To effectively use backtesting, investors should consider its output as data, not returns. Perhaps this all seems a bit pedantic but bear with me. Investing is about taking on calculated risks. I loved Kris’ blog on this topic last week. And backtesting is a tool that can give you data and insight to inform the risks you take. The backtest is not something you can invest in. You can’t go back to 1997 and buy AMZN at its IPO price. It’s impossible. So the backtest isn’t a return. The challenge, and power of backtesting, is making sense of what the backtest data is telling you and applying the insight appropriately to your risk-taking.

Here are a few things you can do to improve your backtesting and become a better investor.

Simon Sinek Rule

For those of you who don’t know Simon Sinek, his popular books and Ted Talks advocate for starting with “why.” In investing, I think that means asking why I am building this symphony and why will it work? 

Let’s start with the first one. This question forces you to start portfolio construction from first principles. What are you trying to accomplish, and what assets and rules will help get you there?

For example, if you evaluate reducing portfolio volatility, start by asking yourself, “how much volatility or how big of a market drop am I comfortable with?” Then, put together a symphony that shifts from relatively riskier assets (e.g., stocks) to relatively less risky assets (e.g., US treasuries), like this Risk On Risk Off trade. The trigger for the trade should match the “why.” In this example, bond market outperformance over the past quarter triggers a shift from Risk On to Risk Off. You can also check out a levered version.

A risk-on-risk-off investment strategy using Composer's automated trading software.

Paired Switching S&P 500 and Bonds represents a Risk On Risk Off trade

Now, let’s look at the second question: why do I think this symphony will work? This question forces you to look into the future, and if your only answer is that it worked in the past, you may want to re-evaluate. This step requires critical thought. Do I expect the environment or reasons for the strategy’s past performance to continue? 

For our Risk On Risk Off symphony, the answer to that question is somewhat complicated. Do I expect the symphony to reduce volatility and drawdowns? Yes, I think that’s reasonable given it shifts to US treasuries, an asset with lower historical and expected standard deviation. Do I expect the symphony to outperform buy-and-hold or perfectly time downturns? No, I don’t think that’s reasonable. Future market movements, including the timing, length, and depth of drawdowns, have proven almost impossible to predict. 

Starting with “why” will help you fight look-ahead bias. If you start with a forward-looking view towards what you want to accomplish, you’ll be less likely to game the backtest. And if you can articulate an explanation for why you expect the symphony to perform going forward, you’ll have a clear investment thesis to test. 

Zoom In

A natural question when using backtesting is: what time period should you use? Some experts recommend using the longest time-period possible to evaluate an investment strategy. Makes sense. Viewing performance across many different market environments (e.g., high rates, low rates, recession, expansion) should paint a clearer picture than, for example, seeing how it did over the last month. This is why symphonies on the Composer Discover page default to the longest time horizon possible. It’s a great place to start. Here is the Paired Switching symphony from above over the longest time period we have data:

Risk-on-risk-off investment strategy backtested from October 22, 2002 to February 22, 2022

Data from October 22, 2002 to February 22, 2022

However, it is important to zoom in and test the symphony in different time periods. How did the strategy perform through the GFC? What about the COVID Crash? Or how about 2021, when volatility was muted and the S&P returned ~27%? By zooming in, you can start to draw insight into when the symphony performs well and when it may lag. Take it step further and pick random time periods. Or, roll through backtests starting with the earliest period and move forward by set intervals, a year or a month for example. Evaluating performance across all these periods will give you insight into how the symphony may perform when it’s live in an uncertain future state. Here is the same symphony from above looking at only its first five years of data:

Risk-on-risk-off investment strategy backtested from October 22, 2002 to October 22, 2007

Data from October 22, 2002 to October 22, 2007

From October 22, 2002 thru October 22, 2007 the symphony lagged the S&P 500 (represented by SPY), but importantly it delivered reduced volatility and Max Drawdown. 

Trading Balance Beam

Symphonies must balance responsiveness and trading costs. More frequent rebalancing, like daily and weekly, is more responsive to changing market conditions. However, every time a symphony trades, it incurs transaction costs, namely bid-ask spreads. Further, transactions can trigger capital gains taxes. And short-term capital gains, where assets are held less than a year, are taxed at a higher rate than long-term capital gains. Both of these factors contribute to the unfortunate truth that higher turnover = higher cost. 

Investors should weigh the benefits of more frequent trading against the transaction and tax costs of turnover. The CFA institute recommends starting with monthly rebalancing for backtests. Through backtesting, you can evaluate how often a symphony would have traded in the past to help inform your rebalance frequency decision. For example, if your symphony logic is such that it is hardly ever triggered and the symphony rarely trades, there isn’t necessarily an issue with daily rebalancing. However, a signal that triggers more frequently in the backtest may lead to high portfolio turnover if the rebalance frequency is set to a shorter window.  

Too Much of  a Good Thing: Overfitting 

The goal when building and backtesting a symphony, and I want to be emphatic, is to develop a portfolio that matches your goals and risk tolerance. The goal is NOT to develop a symphony that perfectly times the bottom of the Global Financial Crisis, participates in the full rebound, and delivers a crazy Sharpe ratio. The GFC will not happen again in the same way and your symphony needs to be built for the future. 

Backtests are a tool to evaluate risks and returns of symphonies. To avoid overfitting, test your symphony using a range of inputs. For example, test your Risk On Risk Off strategy using 5%, 7%, 10%, and 15% Drawdowns. Are the results consistent? Does one result look way better than the others? If so, it may be statistical noise. I can’t think of a good argument why using a drawdown of 7% would be way better than 5% or 10% [4]. Another way to avoid overfitting is apply the Simon Sinek rule and start with why.

Onwards and Upwards

Investors have more data and computing power at their disposal than anytime in history. And by making it easy to backtest dynamic symphonies, Composer is bringing that power to retail investors. 

Start with why. Test multiple periods. Avoid the temptation to “look ahead” and overfit. Balance trading costs. Do all that and you can harness the power of backtesting and build symphonies that are ready to take on the future. 

As always, happy building.

Important Disclosures

Investing in securities involves risks, including the risk of loss, including principal. Composer Technologies Inc., is an SEC Registered RIA. The SEC has not approved this message.

Certain information contained in here has been obtained from third-party sources. While taken from sources believed to be reliable, Composer has not independently verified such information and makes no representations about the accuracy of the information or its appropriateness for a given situation. In addition, this content may include third-party advertisements; Composer has not reviewed such advertisements and does not endorse any advertising content contained therein.

This content is provided for informational purposes only, as it was prepared without regard to any specific objectives, or financial circumstances, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not intended as a recommendation to purchase or sell any security and performance of certain hypothetical scenarios described herein is not necessarily indicative of actual results. Any investments referred to, or described are not representative of all investments in strategies managed by Composer, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results.

Charts and graphs provided within are for informational purposes solely and should not be relied upon when making any investment decision. Past performance is not indicative of future results. The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see Composer's Legal Page for additional important information.