Free Web Hosting Provider - Web Hosting - E-commerce - High Speed Internet - Free Web Page
Search the Web

[Home] [Papers] [Courses] [Contact]
A FRAMEWORK FOR MIS EFFECTIVENESS RESEARCH

Jamal Munshi, Sonoma State Univesity
All rights reserved

INTRODUCTION

Information systems are costly to purchase, deploy, and maintain. Therefore, in a world where business enterprise is operated for value maximization according to the theory of rational choice, it is natural to suppose that MIS offers economic value and that this value overcomes the costs. As such it has rightly been an objective of MIS research for at least two decades to determine the economic role of MIS. Today information system effectiveness continues to occupy the highest priority of the MIS research agenda.

Yet much of the research on the effectiveness or success of information systems has relied on psychometric measures of user satisfaction also referred to as 'user information satisfaction' or UIS. These studies define MIS success narrowly in only subjective terms. In a recent paper Melone [14] argued that the UIS construct is inadequate as a surrogate of effectiveness and that UIS questionnaire construction and methodology are lacking in scientific rigor. MIS researchers have failed to use the accumulated knowledge and practices of their reference disciplines.

An alternative measure offered by some is system usage which is based on the principle that if the system is being used it must be effective and the more it is used the greater the effectiveness. Srinavasan [15] argues for usage and develops a correlation between usage and satisfaction. More recently Barki and Huff [2] have combined usage and satisfaction scores as dependent variables in an effort to measure success of DSS implementations.

A more ambitious proposal is to define the effectiveness of the information system as the degree to which the business goals, for which the IS was deployed, are actually achieved. Although these ideas are appealing from a viewpoint of rational objectivism, the difficulty of defining and measuring such variables has forced researchers to once again resort to questionnaires. The end result is that MIS effectiveness is determined not in terms of observations made by the researcher but in terms of an average of opinions of users, developers, and managers [6].

While satisfaction scores and opinions can be useful in comparing two systems within the same user/developer community, they make inter-firm comparisons difficult. These difficulties convinced many researchers to abandon the questionnaire approach and make direct observations of accounting and economic variables that would be indicative of the achievement of business goals [1] [5].

However, at the firm and industry levels, these studies have been frustrated by the concomitant effect of intervening and extraneous variables that are much more powerful than the information system itself. That is, the so called 'bottom line' of the business enterprise is so greatly affected by economic and market conditions that only a small amount of the variance may be attributed to the information system being tested. Also, some organizations are more efficient than others in utilizing information systems. Thus, the component of the total variance contributed by information systems may not be easily detected in large cross-sectional studies. The lack of statistical power possibly played a role in the inconclusive nature of the results reported by Chismar and Kriebel [4].

The problem with statistical power may be minimized if the study is confined to a specific application or implementation or if it is confined to a controlled laboratory experiment. The study of a sales information system by Lucas [12] is an example of the former while the analysis of a simulated strategic information system by King and Rodriguez [10] is exemplary of the latter.

But what these studies gain in internal validity they lose in the ability to externalize the findings to the 'real world'. Findings of the success of a single implementation are usually viewed as anecdotal and do not lead to convincing statements of a general nature; and the laboratory environment such as one where hypothetical business problems are solved by MBA students lack the realism of a business setting.

Some of these problems are overcome by the use of scientifically designed case studies as described by Lee [11]. In these studies, a carefully defined hypothesis is tested in a real-life situation in which 'fortuitous' circumstances allow rigorous control of concomitant variables. If there is no error of measurement the hypothesis can be rejected if it is violated by a single observation. A successful study of this nature has been reported by Markus [13].

However the economic measure of IS effectiveness as the "conversion of IS investment into real output" [16] is itself in question in organizational paradigms where politics and conflict are important motivating factors. Markus [13] found "re-distribution of power" to be an important impact of IS implementation while other researchers have described the roles played by leadership and managerial control issues in IS implementation decisions.

The measurement of IS effectiveness at the firm level as proposed by Crowston and Treacy [5] and Bakos [1] necessarily require that the researcher have a theory of the firm that precludes conflict such as that caused by agency issues [8]. These theories allow for managerial behavior that is not consistent with profit or value maximization. Since IS implementation decisions are made by managers, managerial behavior and motivation are likely to be important variables in the study of IS effectiveness.

It therefore appears that neither a purely subjective paradigm nor a purely functional paradigm of the firm is adequate for understanding the motivations for and the effectiveness of the deployment of information system. There is a wide spectrum within which the concept of effectiveness may be interpreted and the dimensions in which such an interpretation may be placed. Definitional and measurement issues have retarded the orderly and scientific accumulation of knowledge in this field.

The purpose of this paper is to present a multi-dimensional 'framework' of IS effectiveness that can accommodate many of these apparently disjointed and conflicting views. It is hoped that the adoption of such a framework will foster a cumulative tradition in effectiveness research in which the complementary role of individual research efforts is emphasized and the frictional losses of conflict are reduced.

Three dimensions of effectiveness are identified and explored in this paper. These are the dimension of scope, the dimension of measurement, and the dimension of social paradigm. The dimensions are first described without reference to the placement of previous research within the 'effectiveness space' they define. An analysis of past research within this framework is presented in the full paper available from the author.

THE DIMENSIONS OF EFFECTIVENESS

The dimension of scope describes how broadly the concept of effectiveness is to be applied. In the narrowest sense it is applied to a single implementation of a specific application program. This can be expanded to include multiple implementations of the same program or to an entire class of applications. We can refer to these as the application level and we may interpret the effectiveness measure in terms of the design, usability, and usefulness of the application in question.

An enhanced sense of generality can be realized by considering the impact of IS on an entire firm regardless of application. At the firm level the effectiveness measure can be related to the firm's MIS organization, policy, budget, as well as attitudes and opinions. This generality can be further increased by considering an entire class of firms in a homogeneous line of business. An industry or in a broader sense, an entire sector of the economy can be included in what can be termed the industry level. At the economy level the impact of information technology on the entire economy can be assessed as has been done by Jonscher [9] in his famous paper. At the extreme generality of scope, the society level, sociologists may consider the impact of information technology on society at large.

The dimension of measurement addresses the type of data to be collected, the method of their collection, and the manner of their interpretation. This dimension can be broadly divided into two parts - direct observations of business variables and psychometric measures of attitude and behavioral variables. Psychometric measures are made by constructing questionnaires designed to assess attitudes and opinions on various types of ordinal scales. The psychometric measures can be further sub-divided into two groups; those that measure attitudes (the 'perceptual level') and those that elicit opinions.

The perceptual level lies at the lowest end of the measurement dimension in terms of objectivity. These questionnaires measures the subjective evaluation of users, managers, and IS designers. The user information satisfaction studies can be described by the properties of this level. The influence of subjectivity is lower in the opinion level in which the questions in the instrument are meant to gather data about behavior such as IS utilization and business performance rather than attitude.

A much higher level of objectivity is attained when the researcher directly observes variables that measure utilization and performance instead of eliciting opinions about them. 'Utilization' is the extent to which the IS is used by the intended users for the intended purpose. 'Performance' is the measure of improvement of the business function supported by the IS.

The field observation level of utilization and performance measures are obtained from business records or through visual or electronic inspection and are therefore independent of personal opinions and attitudes. The variables to be observed, however, may be difficult to identify and their causal connection to the MIS may be questioned. This is because of the presence of powerful intervening variables that relate to accounting, marketing, economics, and managerial strategy and efficiency. These problems are avoided at the highest level of measurement, the controlled experiment. Although the experiment offers the highest degree of precision, both the scope and the ability to extend the results beyond the controlled environment of the experiment are limited.

The third dimension of IS effectiveness uses the organizational paradigms described by Burrell and Morgan [3] and adapted to information systems by Hirschheim [7]. This dimension adds generality to the analysis by allowing for the possibility that not all activities of business enterprises are interpretable as if they were rational organizations objectively seeking to maximize the wealth of the owners. Other paradigms of business organizations exist in which managers may take action to increase their utility rather than the owners' wealth and where various degrees of conflict exist between managers, workers, and stakeholders.

The four paradigms presented by Hirschheim may be placed along a single dimension according to the extent to which they approach the normative model of rational choice. These are, from the lowest level to the highest, Neohumanism, Social Relativism, Radical Structuralism, and Functionalism.

Functionalism is characterized by a high degree of order and objectivity and satisfies the normal assumption of rational choice. In this scenario, the MIS supports rational decision makers who arrive at optimal decisions for the firm and seek to increase valuation through objective and scientific means. The effectiveness of the MIS can therefore be correctly assessed by performance measures of the business or the so called 'bottom line' as suggested by Crowston and Treacy. Such measures of performance, however, may not be applicable in other paradigms.

Radical structuralism, for instance, would allow the managers, workers, and owners to each have their own separate agenda. Although each group attempts to maximize utility according to rational choice and objectivity, their goals are allowed to be in conflict. Agency theory and the theory of corporate control put forth by Jensen and Meckling [8] and others are consistent with radical structuralism as is the important finding by Markus [13] of the role of 'politics' in IS implementation success. In firms where radical structuralism is important, performance measures that subsume only functionalism would reveal only a part of the overall impact of information systems. For example, managers may deploy information systems to extend managerial control at the expense of the wealth of the owners and the satisfaction of the workers. If such were the case, the effectiveness of the system may not be assessed without an objective measure of the additional control made possible by the system.

Social relativism and neohumanism emphasize subjective experiences of individuals rather than the objective goals of groups . These two organizational paradigms differ with respect to the degree of order and conflict. Social relativism combines order and subjectivity. Each individual is viewed as working toward company objectives for the common good but in his own way. Users create their own reality and MIS helps them create new realities. In neohumanism, divergent individual goals create conflict. The IS supports the discourse between individuals who are viewed as idealists seeking radical change. The effectiveness of IS within the subjective paradigms may be measured best with psychometric tests of attitude, interests, and opinions such as user information satisfaction.

This description of paradigms is not meant to imply that any real organization can be characterized by a single paradigm but only to suggest that these concepts can help us identify the components of IS effectiveness in a broad sense.

Figures 1 and 2 are graphical depictions of the IS effectiveness framework described above. In each figure, rectangles are drawn to encompass regions of research activity.

FIGURE 1

A FRAMEWORK FOR THE DEFINITION OF MIS EFFECTIVENESS
SHOWING THE SCOPE AND MEASUREMENT DIMENSIONS

A FRAMEWORK FOR THE DEFINITION OF MIS EFFECTIVENESS
SHOWING THE SOCIAL PARADIGM AND MEASUREMENT DIMENSIONS

CONCLUSIONS

The framework of IS effectiveness presented may be used not only to consolidate past research but to plan future research. It is hoped that such a structured approach will prove a more constructive way in which effectiveness research can build a cumulative tradition and lay a foundation of knowledge. The ultimate aim is to construct a theory of the firm from an IS perspective that can serve as a common point of reference for research in information systems.

REFERENCES

[1] Bakos, J. Yannis, "Dependent Variables for the Study of Firm and Industry Level Impacts of Information Systems", Proceedings of the Fifth International Conference on Information Systems, 1985, pp. 10-23

[2] Barki, H. and S.L. Huff, "Implementing Decision Support Systems: Correlates of User Satisfaction and System Usage", INFOR, Vol. 28 No. 2, May 1990, pp. 89-101

[3] Burrell, G. and G. Morgan, "Sociological Paradigms and Organizational Analysis", Heinemann Press, London, 1979.

[4] Chismar, William G. and Charles H. Kriebel, "A Method for Assessing the Economic Impact of Information Systems Technology on Organizations", Proceedings of the Fifth International Conference on Information Systems, 1985, pp. 45-56

[5] Crowston, Kevin and Michael E. Treacy, "Assessing the Impact of Information Technology on Enterprise Level Performance", Proceedings of the Sixth International Conference on Information Systems, 1986, pp. 299-310

[6] Gallagher, Charles A., "Perceptions of the Value of a Management Information System", Academy of Management Journal, Vol. 17 No. 1, 1974, pp. 46-55

[7] Hirschheim, Rudy and Heinz K. Klein, "Four Paradigms of Information System Development", Communications of the ACM, Vol. 32 No. 10, October 1989, pp. 1199-1215

[8] Jensen, Michael C. and William H. Meckling, "Theory of the Firm: Managerial Behavior, Agency Costs, and Ownership Structure", Journal of Financial Economics, Vol. 3, 1976, pp. 305-360

[9] Jonscher, C., "Information Resources and Economic Productivity", Information Economics and Policy, 1983, pp. 13-35

[10] King, William R. and Jaime I. Rodriguez, "Evaluating Management Information Systems", MIS Quarterly, September 1978, pp. 43-51

[11] Lee, Allen S., "A Scientific Methodology for MIS Case Studies", MIS Quarterly, March 1989, pp. 33-50

[12] Lucas, Henry C., "Performance and the Use of an Information System", Management Science, Vol. 21 No. 4, April 1975, pp. 908-918

[13] Markus, M.L., "Power, Politics, and MIS Implementation", Communications of the ACM, June 1983, pp. 430-444

[14] Melone, Nancy Paule, "A Theoretical Assessment of the User Satisfaction Construct in Information Systems Research", Management Science, Vol. 36 No. 1, January 1990, pp. 76-91

[15] Srinavasan, A., "Alternative Measures of System Effectiveness: Associations and Implications", MIS Quarterly, September, 1985, pp. 243-253

[16] Weill, Peter, and Margrethe H. Olson, "Managing Investment in Information Technology: Mini Case Examples and Implications", MIS Quarterly, March 1989, pp. 2-11.

Original file name: swafad91

This file was converted with TextToHTML - (c) Logic n.v.