Market Efficiency  (updated 27 Feb 99)

The efficient market hypothesis (EMH) says that at any given time, asset prices fully reflect all available information. That seemingly straightforward proposition is one of the most controversial ideas in all of social science research, and its implications continue to reverberate through investment practice. As MIT finance professor Andrew Lo writes in the introduction to two volumes that collect the key articles on the EMH, 'it is disarmingly simple to state, has far-reaching consequences for academic pursuits and business practice and yet it is surprisingly resilient to empirical proof or refutation'.

The simple statement implies but does not limit information to be strictly financial in nature. It may incorporate investor perceptions whether correct or otherwise. This richer interpretation of the EMH provides for variations from its stronger forms, which suggest that further data study, unless perhaps insider-based, is unlikely to be fruitful. The second derivative of an investor perception overlay on financial information allows for intuition, judgment and the quest for new tools that markets may discover in the pursuit of profits above the average.

The chief corollary of the idea that markets are efficient, that prices fully reflect all information, is that price movements do not follow any patterns or trends. This means that past price movements cannot be used to predict future price movements. Rather, prices follow what is known as a 'random walk', an intrinsically unpredictable pattern. The random walk is often compared to the path a sailor might follow out of a bar after a long, hard night drinking.

In the world of the strong form EMH, trying to beat the market becomes a game of chance not skill. There will be superior performers generating better investment returns but only because statistically there are always some people above the average and others below. Hence, debate about the EMH becomes a question of whether active portfolio management works: is it possible to beat the market or are you better off avoiding the transactions costs and simply buying an index fund? And, as an active manager, the issue is whether it works for me, a sample size of one? (see ACTIVE PORTFOLIO MANAGEMENT)

The answer to these questions depends not only on whether you accept the EMH but, if so, in what form. There are essentially three:

· The weak form of the EMH asserts that all past market prices and data are fully reflected in asset prices. The implication of this is that technical analysis cannot be used to beat the market (see TECHNICAL ANALYSIS).

· The semi-strong form of the EMH asserts that all publicly available information is fully reflected in asset prices. The implication of this is that neither technical nor fundamental analysis can be used to beat the market.

· The strong form of the EMH asserts that all information - public and private - is fully reflected in asset prices. The implication of this is that not even insider information can be used to beat the market.

Gurus of efficient markets: Eugene Fama and Burton Malkiel

Although the concept of the random walk can be traced back to French mathematician Louis Bachelier's doctoral thesis 'The Theory of Speculation' in 1900, the EMH really starts with Nobel Laureate Paul Samuelson and his 1965 article, 'Proof that Properly Anticipated Prices Fluctuate Randomly'. But it was Chicago finance professor Eugene Fama with his 1970 paper 'Efficient Capital Markets' who coined the term EMH and made it operational with the foundational epithet that in efficient markets, 'prices fully reflect all available information'.

Fama argued that in an active market of large numbers of well-informed and intelligent investors, stocks will be appropriately priced and reflect all available information. In these circumstances, no information or analysis can be expected to result in outperformance of an appropriate benchmark. Because of the wide availability of public information, it is nearly impossible for an individual to beat the market consistently.

Another professor, Burton Malkiel of Princeton, popularized the notion of the random walk implication in his bestseller A Random Walk Down Wall Street. He suggested that throwing darts (or, more realistically, a towel) at the newspaper stock listings is as good a way as any to pick stocks and is likely to beat most professional investment managers. Malkiel does suggest in the later part of his work how those who insist on trying to beat the market might attempt to do so, but he indicates that they are unlikely to be successful.

Since the EMH was formulated, countless empirical studies have tried to determine whether specific markets are really efficient and, if so, to what degree. Andrew Lo's volumes bring together some of the most significant contributions, including a paper called simply 'Noise' by the late Fischer Black. It says:

'Noise in the sense of a large number of small events makes trading in financial markets possible. Noise causes markets to be somewhat inefficient, but often prevents us from taking advantage of inefficiencies. Most generally, noise makes it very difficult to test either practical or academic theories about the way that financial or economic markets work. We are forced to act largely in the dark.'

Counterpoint

A central challenge to the EMH is the existence of stock market anomalies: reliable, widely known and inexplicable patterns in returns. Commonly discussed anomalies include size effects, where small firms may offer higher stock returns than large ones; and calendar effects, such as the 'January effect' - which seems to indicate that higher returns can be earned in the first month compared to the rest of the year - and the 'weekend effect' or 'blue Monday on Wall Street' - which suggests that you should not buy stocks on Friday afternoon or Monday morning since they tend to be selling at slightly higher prices. There are also the supposed indicators of undervalued stocks used by value investors, such as low price-to-earnings ratios and high dividend yields (see VALUE INVESTING).

But while there is no doubt that anomalies occur in even the most liquid and densely populated markets, whether they can be exploited to earn superior returns in the future remains open to question. If anomalies do persist, transactions and hidden costs may prevent them being used to produce outperformance, as well as the rush of other investors trying to exploit the same anomalies. It may be possible that opportunities arise in quanta bursts and then disappear rather like the track in a cloud chamber. If so, by the time we wish to measure the recurrence of an event, it has occurred and passed by, unlikely to be repeated in the same form.

Further challenges to the EMH come from the study of behavioral finance, which examines the psychology underlying investors' decisions and uses it to explain such phenomena as stock price over-reaction to past price changes and stock price under-reaction to new information (see INVESTOR PSYCHOLOGY). Many studies seem to confirm the implication of over- and under-reaction that there are 'pockets of predictability' in the markets: contrarian strategies of buying 'losers' and selling 'winners' can generate superior returns; and prices do tend to regress to the mean.

A light-hearted yet cutting angle on the impact of psychology on market efficiency turns up in a 1997 article on the website of MIT economics professor Paul Krugman. Attending a big conference of money managers, Krugman detects 'The Seven Habits of Highly Defective Investors', the behavioral traits that he says make the markets anything but efficient: think short-term; be greedy; believe in the greater fool; run with the herd; overgeneralize; be trendy; and play with other people's money. 'What I saw', Krugman recounts, 'was not a predatory pack of speculative wolves: it was an extremely dangerous flock of financial sheep.'

Of course, the vast majority of successful professional investors claim they have disproved the EMH. (The unsuccessful are engaged in other pursuits giving this sample technique hindsight bias.) And any active manager, no matter what his record, will be eager to argue that the markets are not efficient in order to justify his work as an agent for hire by others. Even the financial media has a powerful interest in decrying the EMH: if all information is fully reflected in prices, what value is there in the information they supply?

But are the investors who really beat the market consistently over, say, a five year period simply the inevitable result of a standard distribution? After all, if a hundred people toss coins five times in a row, the probability is that two or three of them will have called correctly five times straight. In the same way, probability indicates that there will be someone occupying the Warren Buffett investment performance slot and it will be someone who has done the right things - but is it skill or luck? And by the time we might have statistical verification, such a long period - fifty years or so - will have passed to make the study useless.

And, of course, just as the challenge of stockpicking is to identify a superior performer before the fact rather than in hindsight, so it is with investment managers. In many cases, strong performers in one period frequently turn around and underperform the next, and, as statistics would predict, a number of studies show that there is little or no correlation between strong performers from one period to the next.

Nevertheless, as Peter Bernstein points out in a 1998 address to European financial analysts, 'even though beating the market is increasingly difficult, more and more people undertake the effort'. He suggests that 'the enormous volume of trading in today's markets is an important indication that market efficiency in the pure sense has no relevance to the real world of investing', and that 'equilibrium prices are impossible in a dynamic and restless world of noise traders in the market'.

Andrew Lo adds: 'As with any other industry, innovation and creativity are the keys to success, so why should we find it surprising that those who are capable of such feats outperform the rest of the pack? If the EMH in its classical form seems to be violated so often, maybe we economists ought to re-examine our theory instead of arguing that the world is crazy.' (see FINANCIAL ENGINEERING).

Guru response

Burton Malkiel responds with extracts from the latest edition of his classic book. In a new chapter entitled 'The Assault on the Random-Walk Theory: Is the Market Predictable After All?', he writes: 'I have reviewed all the recent research proclaiming the demise of the efficient-market theory and purporting to show that market prices are, in fact, predictable. My conclusion is that such obituaries are greatly exaggerated and the extent to which the stock market is usefully predictable has been vastly overstated.'

'First, there are considerable questions regarding the long-run dependability of these effects. Many could be the result of 'data snooping', letting the computer search through the data sets of past securities prices in the hopes of finding some relationships. With the availability of fast computers and easily accessible stock market data, it is not surprising that some statistically significant correlations have been found, especially because published work is probably biased in favor of reporting anomalous results rather than boring confirmations of randomness. Thus, many of the predictable patterns that have been discovered may simply be the result of data mining - the result of beating the data set in every conceivable way until it finally confesses. There may be little confidence that these relationships will continue in the future.'

'Second, even if there is a dependable predictable relationship, it may not be exploitable by investors. For example, the transaction costs involved in trying to capitalize on the January effect are sufficiently large that the predictable pattern is not economically meaningful. Third, the predictable pattern that has been found, such as the dividend-yield effect, may simply reflect general economic fluctuations in interest rates or, in the case of the small-firm effect, an appropriate premium for risk. Finally, if the pattern is a true anomaly, it is likely to self-destruct as profit maximizing investors seek to exploit it. Indeed, the more profitable any return predictability appears to be, the less likely it is to survive.'

'It is abundantly clear that techniques that work on paper do not necessarily work when investing real money and incurring the large transactions costs that are involved in the real world of investing. As a successful portfolio manager (ranked in the top 10% of all money managers) once sheepishly told me, 'I have never met a back test I didn't like'. But let's never forget that academic back tests are not the same thing as managing real money.'

'In summary, pricing irregularities and predictable patterns in stock returns may well exist and even persist for periods of time, and markets can be influenced by fads and fashions. Eventually, however, any excesses in market valuations will be corrected. Undoubtedly, with the passage of time and with the increasing sophistication of our databases and empirical techniques, we will document further apparent departures from efficiency and further patterns in the development of stock returns. Moreover, we may be able to understand their causes more fully. But I suspect that the end result will not be an abandonment of the belief of many in the profession that the stock market is remarkably efficient in its utilization of information.'

Where next?

There is what Andrew Lo calls 'a wonderfully counter-intuitive and seemingly contradictory flavor' to the idea of informationally efficient markets: the greater the number of participants, the better their training and knowledge and the faster the dissemination of information, the more efficient a market should be; and the more efficient the market, the more random the sequence of price changes it generates, until in the most efficient market, prices are completely random and unpredictable. That is to say that the more lemmings there are, the less likely they all are to fall over the cliff. Similarly, if everyone believes the market is efficient, then it will no longer be efficient since no one will invest actively. In effect, efficient markets depend on investors believing the market is inefficient and trying to beat it.

In reality, markets are neither perfectly efficient nor completely inefficient. All are efficient to a certain degree - and new technology probably serves to make them more efficient. But some markets are more efficient than others. And in markets with substantial pockets of predictability, active investors can strive for outperformance. Peter Bernstein concludes that there is hope for active management: 'the efficient market is a state of nature dreamed up by theoreticians. Neat, elegant, even majestic, it has nothing to do with the real world of uncertainty in which you and I must make decisions every day we are alive.'

Read on

In print

Andrew Lo, Market Efficiency: Stock Market Behavior in Theory and Practice, two volumes of the most important articles on the subject, including Eugene Fama's seminal 1970 review, Paul Samuelson's 1965 article and Fischer Black's 1986 article
Andrew Lo and Craig Mackinlay, A Non-Random Walk Down Wall Street
Burton Malkiel, A Random Walk Down Wall Street, a long-time bestseller, first published in 1973 and now in preparation for its seventh edition

Online

web.mit.edu/krugman/www - Paul Krugman's website www.ssrn.com - website of the Social Science Research Network, which features many important papers in investment, including Eugene Fama's 'Market Efficiency, Long-term Returns and Behavioral Finance'