Overall high quality comments on my discussion of tenure. A few responses.
Professors at the vast majority of colleges and universities do very little research, long-term or any other type. Serious research is concentrated at 50-100 universities. So it is hard to see the length of time it takes to complete major research as an argument for tenure at the remaining 3000 or so colleges and universities. Moreover, Bell Labs in its heyday, and other corporate research centers have encouraged long-term research without giving tenure. Good organizations, whether universities or corporations, will see the potential of original research, whereas bad ones will not, with or without tenure.
I do not know enough about what Boston University offered. If they did offer both tenure and non-tenure options, the data would tell us something useful about the value placed on tenure, although more risk-averse professors will tend to choose the tenure route.
I agree with some of the comments that several year contracts that are renewable may well be the way to go in academia. This would provide better incentives to professors during their prime years. It would also help get around the foolish Federal law that prevents universities from forcing older professors to retire, except in extreme circumstance.
Perhaps university administrators desire tenure because their evaluation by higher ups is shortsighted. But good colleges and universities would be better managed than that, given the competitiveness of the market for higher education in the United States. In fact, the survival of tenure in such a competitive higher education market often makes me wonder if the arguments against tenure are overlooking some important reasons why tenure may improve performance and efficiency.
James Miller was a student of mine, and a very good student indeed. He is also politically conservative in the sense that he believes in the advantages of free markets and a private enterprise system. Unfortunately, most faculties, including many economics departments, do not appreciate such views. That said, I must add that I do not know the situation at Smith, and why he was initially denied tenure, although it is interesting and relevant to the answer that he apparently won his appeal.