The Hazards Of One-Size-Fits-All Performance Attribution

DAL Investments suggests that we should dispense with narrowly focused benchmarks for evaluating actively managed mutual funds, according to The New York Times. “As much as people in the fund industry may want to measure their performance against a narrowly defined index, the reality is that most investors judge their returns against the S.& P. 500, for better or worse,” writes Times reporter Paul Sullivan. That’s hardly a rationale for using the S&P 500, or any one benchmark, for analyzing a wide spectrum of investment strategies. Using one index certainly simplifies the critical task of portfolio attribution, but at what cost?

Don’t misunderstand: the case for parsimonious models is well founded. As Einstein said, “Make things as simple as possible, but not simpler.” Forcing explanations of what’s driving returns through the prism of one benchmark, however, seems destined to violate this rule by erring on the side of excess simplicity.
Granted, the S&P 500 is probably the most popular benchmark in the money game. But unless you’re reviewing a large-cap U.S. equity portfolio, the S&P is of limited value. Comparing apples to apples is crucial for evaluating investment results and distinguishing between skill and luck. The required analysis may be inconvenient when using multiple benchmarks, but there’s not much point to relying on faulty analysis just because it’s easy.
This is old news, of course. Every professional investment analyst worthy of the title understands why comparing, say, a small-cap manager to a big-cap index will lead to spurious conclusions. Similarly, if we’re crunching the numbers on a multi-asset class strategy—mixing stocks and bonds, for instance—it’s essential that we review the portfolio in the context of a multi-asset class benchmark in order to understand what’s driving results.
One populuar methodology that can help is known as style analysis. Professor Bill Sharpe introduced the idea of returns-based style analysis in his influential 1988 paper “Determining a Fund’s Effective Asset Mix.” In a follow-up article he explains that “once a set of asset classes has been defined, it is important to determine the exposures of each component of an investor’s overall portfolio to movements in their returns.” Also, here’s an interview with Sharpe where he discusses his views on style analysis in some detail.
(For a primer on multi-factor analysis with Excel, see financial planner William Bernstein’s “Rolling Your Own: Three-Factor Analysis.” Alternatively, you can let software, such as Zephyr’s StyleAdvisor, perform the heavily quantitative lifting. For a quick style-analysis summary of multi-fund portfolios, see Morningstar’s Instant X –Ray tool.)
The rationale for looking at portfolios through a style analysis lens is largely intuitive in light of decades of research in asset pricing that identifies a wide array of factors driving risk premia. The capital asset pricing model’s one-factor description of market returns may still be a good description of asset pricing for broadly diversified portfolios across asset classes over the long term, but that leaves plenty of room for exceptions in the short run. Stephen Ross’s arbitrage pricing model from the 1970s anticipated no less.
Even within the equity universe there’s a rainbow of variables that appear to explain returns, from the widely recognized small-cap and value factors to the more exotic liquidity and momentum drivers, for instance. The message is that if you’re trying to understand why a strategy or fund delivered a particular set of results over a specific time period, factor-based style analysis can wipe away a lot of the mystery. By contrast, using the S&P 500 alone for portfolio attribution is a bit like trying to analyze the business cycle by looking only at interest rates.
No wonder that reviewing historical returns solely through an S&P 500 prism isn’t likely to reveal much if anything (unless you’re looking at a plain-vanilla large-cap equity strategy that’s focused now and forever on U.S. securities). DAL seems to think otherwise. According to the Times, DAL reviewed 306 funds that were founded before 1989 that are still with us in 2011 and

they all invest broadly with various styles and none concentrated on one sector. The data spans 21.75 years, from Dec. 31, 1989, to Sept. 30, 2011. The performance of the funds was measured against the Vanguard S.& P. 500 Index Fund, which had annual returns of 7.65 percent during that time…

Over the two decades of the data, no one investment strategy dominated, and most were successful for only four to five years, on average. Not one fund beat the benchmark every year.

In fact, most funds underperformed the S.& P. 500 about a third of the time.

It’s debatable how much insight this type of analysis offers. Perhaps one could argue that if our goal is simply to beat the S&P, DAL’s perspective is useful. But surely that’s a goal that’s far too narrow for most investors. Even if we think it’s appropriate, identifying funds that beat the S&P without providing deeper analysis of why they outperformed doesn’t offer much strategic insight.
To take an extreme example, if you’re picking stocks from a pool of small-cap companies it’s a safe bet that you’ll report different risk and return results relative to a large-cap equity universe. In other words, your tracking error will be high vs. the S&P 500. But high tracking errors call out for a superior methodology for determining if a small-cap manager is adding value (or not) by analyzing the portfolio against a small-cap benchmark. Of course, high tracking errors vs. a specific benchmark could mean that you’re using the wrong benchmark.
The analysis is more complicated for multi-asset class strategies, but Sharpe’s style analysis helps here too. By regressing a fund’s returns against several related benchmarks that are associated with the strategy, it’s easy to identify the risk factors behind the performance. Style analysis isn’t perfect, of course, and so there’s a case for using other methodologies for analyzing returns and risk. But style analysis is easy to perform and it generally offers good results so it’s an obvious place to start.
Multi-factor analysis may not reveal much beyond what’s expected, although results can be surprising in some cases. For instance, it’s not unusual to find out that a celebrated large-cap manager has been dipping into small-cap stocks to deliver impressive results. That’s hardly a crime, although it’s misleading to bill yourself as a brilliant large-cap manager while downplaying the fact that small caps are juicing returns. In my own quantitative travels, I routinely run multi-factor analysis on various funds and it’s fairly common to find that a manager who purports to be a specialist in a given asset class or style is actually double- or triple-dipping into a range of risk factors.
When it comes to evaluating multi-asset strategies, there’s more data to crunch but the principle’s the same. For example, when running style analysis on asset allocation funds I regress returns against the Global Market Index (GMI), a passively weighted index of all the major asset classes. The question for active multi-asset class strategies is the same for evaluating single-asset class portfolios: Is the manager adding value? Standard finance theory suggests that a broadly defined, unmanaged benchmark of asset classes will deliver competitive results, and my own analysis of 1,000-plus multi-asset class mutual funds vs. GMI offers empirical support.
Another approach that tries to answer this question in quantitative terms is running a multi-factor analysis that regresses a fund’s returns against several benchmarks that represent the opportunity set of asset classes.
Perhaps the main challenge in style analysis is choosing the right set benchmarks for the analysis. As investment consultant Ron Surz reminds, not all style indices are created equal.
The larger point is that style analysis is necessary unless we’re confident that a strategy is pure and transparent. That’s rarely the case, which is one of the arguments for using index funds: you’re always sure that you own a particular beta.
Life is more complicated for for those who hold actively managed funds. But this much is clear: If performance attribution is to have any value, we need robust and relevant benchmarks. One size fits all may be an easy solution, but it’s rarely the right solution.

2 thoughts on “The Hazards Of One-Size-Fits-All Performance Attribution

  1. JP

    Holdings-based attribution is in theory superior. The problem is finding timely updates. From a practical standpoint, returns-based attribution is the solution since it can be calculated easily and reflects recent if not current data. Holdings-based attribution, by contrast, is usually dependent on number from previous months or quarters.

Comments are closed.