The dangers of backtesting ETF volatility trading strategies

Anybody who has spent any time researching trading strategies for the XIV will have no doubt encountered an endless stream of charts that look something like this one below:

At first glance it looks like an amazing trading strategy doesn’t it?  Who doesn’t want to turn 10,000$ into several million in just 12 years?  Well, if only investing was that easy.  The truth is nothing this good exists in the real world of live trading.  It only exists in a simulated world of perfect 20/20 hindsight curve fitted backtesting.

From 2004 – 2010:

Those results start in 2004, and by the end of 2010 we’re already at a million dollars.  10,000$ to 1,000,000$ in 7 years, wow.  The problem with this is, the XIV didn’t actually launch as a tradable product until the very end of 2010.

Now the people who report these kinds of eye popping backtests will no doubt tell you that those prices are based on a re-creation of actual VIX futures data going back to 2004 and they are.  We can actually simulate the data going backwards and re-create what the XIV would technically have looked like.

The problem however is that due to the way XIV derives it’s price, the very existence of the product would have dramatically changed it’s past values.

The XIV isn’t a stock that rises and falls with supply and demand.  What drives the price of XIV is the trading activity in the VIX futures market.  In the past if you wanted to hedge a portfolio or trade volatility there were only a few choices.  You could trade options on the stock market indexes like the S&P 500 for example, or you could directly trade VIX Options or VIX Futures.  Now with the introduction of these exchange traded volatility products, many investors are bypassing those and going straight to owning the much more accessible XIV or VXX for example.  And not in small amounts either.  There are tens of millions of shares being traded daily.  When you add up the trading volume on the more popular ones like XIV, VXX, ZIV, SVXY, TVIX just to name a few, the volume is extremely heavy.

If products like the XIV actually existed in 2004, that would have effected the hedging behavior of market participants.  It would have changed the composition of options being traded on the S&P 500 which would have affected the values of the VIX Index.  The composition of VIX futures would have been different, which would have changed the prices of the volatility products had they existed.  In the end, the XIV prices would likely not have been what they are in these simulated backtests.  Nobody knows how close or far away they would have been, but we can say they would not have been the same which for me is enough to discount them.  Any uncertainty in data isn’t worth using.

To be clear, XIV prices before it’s launch in late 2010 are completely irrelevant to real world live trading.

2011 & 2012:

These results could potentially be real and could have existed in live trading.  There’s a couple of problems here as well though:

• Very few people were actually trading these products in those first few years, and even fewer were trading them with the same system rules they are today.  The majority are just backtests of “what if.”  When I first started trading these products in 2010, there was very low volume and it didn’t really pick up significantly until around 2013.  Very few people were trading them back then. When I first launched my business there were only a couple others doing something similar.

• Those two years were about as perfect as it gets for trading these products.  A strong bull market with several distinct and quick sell offs mixed in.  You couldn’t design it any better.

So while traders do use this data from the first two years after the products launch to develop future trading systems, it’s not realistic to expect these conditions to actually repeat themselves in the future.

2013 – Present:

Look at the chart again.  Do you notice how the performance is at the same level in late 2015 as it was in late 2012?  Results are completely flat for the past 3 years.  So even if those results are based on real trades from 2013 until today, does it even matter?  Whatever system they are using in that test clearly isn’t working anymore.

So what are we left with?

The incredible eye popping backtest showing 10,000$ turning into several million is essentially just two good years.  Anything before 2011 didn’t exist so we need to remove that.  Everything from 2013 onward hasn’t produced any gains at all so we can eliminate that.  We’re left with a backtest of 2011 and 2012 only, and like I said it’s not likely those trades ever took place or will repeat themselves anytime soon.

Conclusion:

Here at Volatility Trading Strategies, we only post results going back to the official launch of the XIV in late 2010. Of course in a backtest using the simulated XIV numbers dragged back to 2004 our strategy would have had exceptional performance as well, but we feel it’s unethical to post such things on the website because again, the XIV didn’t exist until late 2010.  We will even go a step further and say what others won’t.  Results before late 2012 aren't very meaningful either because those conditions likely won't repeat themselves. What is meaningful though is how various trading systems performed during the much more realistic years from 2013 to the present.  Our results since January 2013 have been just as strong as they were in testing for all those previous years.  Our trading system is just as successful today as it was when we started.

Want to join the Awesome VTS Community?

* All information, analysis, and articles on this site are provided for informational purposes only. Nothing herein should be interested as personalized investment advice as I make no recommendations to buy, sell, or hold any securities or positions. I'm making this website available "as is" with no warranty or guarantees of its accuracy, completeness, or current's. If you rely on this website or any of the information contained, you do so entirely at your own risk. I do not hold myself out as a financial advisor and nothing herein is a solicitation for any fund or securities mentioned. Although I may answer general questions about the information herein, I'm not licensed or registered under security laws to address your personal investment situation. Past performance is not indicative of future results. Any and all financial decisions are the sole responsibility of you the individual.