Once upon a time there was only one factor—the market, a la the capital asset pricing model. But after a half century of crunching the numbers since CAPM was born, “now we have a zoo of new factors,” as Professor John Cochrane observed a few years ago. In theory, identifying more factors opens the door for building superior risk-adjusted portfolios. But some practitioners worry that “the proliferation of factors is deeply troubling,” as Research Affiliates explained recently. Why? Because not all factors are created equal and securitizing what looks like a productive risk premium on paper is tricky when it comes to real-world results.

Finding success in the factor zoo, in other words, is quite a bit more challenging than it appears when reading finance journals. But for those who are willing to try, there are numerous ETFs and mutual funds to choose from in the new world order. Taking the marketing material at face value tells us that clever strategists can build smart-beta portfolios that leave their standard-beta counterparts in the dust. That’s certainly possible, but the pernicious rumor promoted by some folks that happy outcomes are inevitable is misleading at best.

The main problem is that quite a lot of what some see as compelling evidence in favor of going off the deep end with smart beta funds is really just cherry-picking the strongest performers. You can certainly find ETFs and mutual funds that deliver encouraging results in the art/science of mining smart beta. But there are plenty of dogs as well. The real question is whether there’s any evidence that, all else equal, an asset allocation strategy populated with smart-beta funds reliably outperforms its conventional-beta counterparts in a convincing degree on a risk-adjusted basis?

Coming up with an answer is tougher than it sounds, in part because there aren’t a lot of smart-beta ETFs and mutual funds with sufficiently long records to run a robust test. The original factor strategies—i.e., small-cap and value—have been around for a few decades and so there’s a relatively deep and wide empirical record to study on this front. And the results are encouraging, particularly when it comes to value. But there’s a bigger mystery with the newer generation of factor funds that target an array of risk premiums, such as momentum, quality, and volatility. And more are on the way.

Some of this is little more than data mining. Looking for relatively strong relationships in the cross section of security returns is child’s play at this point, thanks to the rise of inexpensive computing power. But the transition from encouraging in-sample results to out-of-sample confirmations using real-world funds is a slippery affair. Most of the studies to date focus on a single asset class; kicking the tires when it comes to asset allocation is still in its infancy with regards to smart beta analysis.

As a preliminary test, I recently ran a test using a set of smart-beta funds that track indexes designed by one of the more respected names in the business. The analysis is compelling because the smart-beta funds I review have been around for at least five years and hug benchmarks designed by a single firm. Meanwhile, there are low-cost alternatives that track conventional cap-weighted indexes. In short, we have the ingredients for a robust test of real-world results. It’s hardly definitive, but it offers some perspective on how smart beta fares in asset allocation.

I created two sets of equity portfolios—a smart-beta strategy and its standard-beta counterpart for a US/foreign equity allocation using five allocation buckets (US broad, US small cap, US value, foreign developed, foreign Asia ex-Japan). The initial portfolio weights are identical. I ran the numbers with a year-end rebalancing strategy vs. a buy-and-hold portfolio. In both cases the results are the virtually the same, namely: there’s not a lot of difference between smart-beta and conventional-beta portfolios over the past five-year period.

In a future post, I’ll lay out the details with a review of the numbers, at which point I’ll name names. For now, let’s just say that the data suggests that building portfolios with smart-beta funds may not be a silver-bullet solution that reliably outperforms a comparable strategy using conventional index funds. Why? Several reasons.

First, smart-beta funds have higher expense ratios, although for the test I ran the funds under scrutiny charged only moderately higher fees vs. the traditional index funds. Another challenge is the simple fact that smart beta doesn’t always outperform, at least not reliably so across all asset classes at all times. This is a major challenge for analysis in this corner because there’s a growing number of vendors using a wide set of criteria for designing funds. Ideally, investors will select only those products that will deliver superior results and otherwise use standard index funds. But this is harder than it sounds, particularly for time horizons over, say, one to three years.

In my test, some of the smart-beta funds outperformed (some of the time), but others stumbled. The result: the wins cancelled out the gains and the overall results tracked the portfolios built with conventional index funds.

Selecting one or two smart-beta funds and earning superior results over standard index products is one thing, but it’s a tougher game when applied to a broad asset allocation strategy over a longer-term horizon. Some of this is due to the variation in the design quality of products, but there’s also lots of debate about what’s likely to work in the smart-beta zoo vs. what’s an anomaly that won’t survive beyond the realm of backtesting. As a recent paper (“Facts and Fantasies About Factor Investing”) by researchers at Lyxor Asset Management explains:

From a professional point of view, only a few number of risk factors and anomalies are reliable. Among these relevant factors, we find for example [small-cap, value and momentum]. But, even with a reduced set of less than 10 factors, there are again a lot of questions to answer in order to understand what the nature, the behavior and the risk of these factors are. Academics have done extensive studies on these questions and their work can help to find the answers, but some questions still remain open, in particular the level of the risk premia.

That last point, about the level of risk premia, is critical. Indeed, after adjusting for commissions, taxes and various real-world frictions, there’s a high bar for arguing that a given factor is a viable candidate for use in real-world portfolios.

The bottom line is that the evolution from conventional-beta products to smart-beta funds comes with a number of hazards. By contrast, the transition from conventional active management to plain-vanilla indexing over the past generation has been and remains a more reliable process, particularly in the context of designing and managing multi-asset class portfolios. That doesn’t mean that smart beta isn’t a productive development in assert pricing and money management. But it turns out that there’s a lot more art than science in the next generation of indexing than some folks would have you believe. As a result, beating Mr. Market’s asset allocation over the long run will likely remain as challenging as ever.