<Previous Lesson

Organization Development

Next Lesson>

Lesson#26

Evaluating and Institutionalizing Organization Development Interventions-1

Evaluating and Institutionalizing Organization Development Interventions Measurement:

Providing useful implementation and evaluation feedback involves two activities: selecting the appropriate variables and designing good measures.
Selecting Variables
:
Ideally, the variables measured in OD evaluation should derive from the theory or conceptual model underlying the intervention. The model should incorporate the key features of the intervention as well as its expected results. The general diagnostic models described earlier meet these criteria. For example, the joblevel diagnostic model proposes several major features of work: task variety, feedback, and autonomy. The theory argues that high levels of these elements can be expected to result in high levels of work quality and satisfaction. In addition, as we shall see, the strength of this relationship varies with the degree of employee growth need: the higher the need, the more that job enrichment produces positive results. The job-level diagnostic model suggests a number of measurement variables for implementation and evaluation feedback. Whether the intervention is being implemented could be assessed by determining how many job descriptions have been rewritten to include more responsibility or how many organization members have received cross-training in other job skills. Evaluation of the immediate and long- term impact of job enrichment would include measures of employee performance and satisfaction over time. Again, these measures would likely be included in the initial diagnosis, when the company’s problems or areas for improvement are discovered. Measuring both intervention and outcome variables is necessary for implementation and evaluation feedback. Unfortunately, there has been a tendency in OD to measure only outcome variables while neglecting intervention variables altogether.
It generally is assumed that the intervention has been implemented and attention, therefore, is directed to its impact on such organizational outcomes as performance, absenteeism, and satisfaction. As argued earlier, implementing OD interventions generally take considerable time and learning.

It must be empirically determined that the intervention has been implemented; it cannot simply be assumed. Implementation feedback serves this purposes guiding the implementation process and helping to interpret outcome data Outcome measures are ambiguous without knowledge of how well the intervention has been implemented. For example, a negligible change in measures of performance and satisfaction could mean that the wrong intervention has been chosen, that the correct intervention has not been implemented effectively, or that the wrong variables have been measured. Measurement of the intervention variables helps determine the correct interpretation of outcome measures. As suggested above, the choice of intervention variables to measure should derive from the conceptual framework underlying the OD intervention. OD research and theory increasingly have come to identify specific organizational changes needed to implement particular interventions. These variables should guide not only implementation of the intervention but also choices about what change variables to measure for evaluative purposes. The choice of what outcome variables to measure also should be dictated by intervention theory, which specifies the kinds of results that can be expected from particular change programs. Again, the material in this book and elsewhere identifies numerous outcome measures, such as job satisfaction, intrinsic motivation, organizational commitment, absenteeism, turnover, and productivity. Historically, OD assessment has focused on attitudinal outcomes, such as job satisfaction, while neglecting hard measures, such as performance. Increasingly, however, managers and researchers are calling for development of behavioral measures of OD outcomes. Managers are interested primarily in applying OD to change work-related behaviors that involve joining, remaining, and producing at work, and are assessing OD more frequently in terms of such bottom-line results.

Designing Good Measures
:
Each of the measurement methods described earlier has advantages and disadvantages. Many of these characteristics are linked to the extent to which a measurement is operationally defined, reliable, and valid. These assessment characteristics are discussed below. 1.

Operational definition


. A good measure is operationally defined; that is, it specifies the empirical data needed how they will be collected and, most important, how they will be converted from data to information. For example, Macy and Mirvis developed operational definitions for the behavioral outcomes (see Table 9). They consist of specific computational rules that can be used to construct measures for each of the behaviors. Most of the behaviors are reported as rates adjusted for the number of employees in the organization and for the possible incidents of behavior. These adjustments make it possible to compare the measures across different situations and time periods. These operational definitions should have wide



applicability across both industrial and service organizations, although some modifications, deletions, and additions may be necessary for a particular application. Operational definitions are extremely important in measurement because they provide precise guidelines about what characteristics of the situation are to be observed and how they are to he used. They tell OD practitioners and the client system exactly how diagnostic, intervention, and outcome variables will be measured. 2.
Reliability


. Reliability concerns the extent to which a measure represents the “true” value of a variable; that is, how accurately the operational definition translates data into information. For example, there is little doubt about the accuracy of the number of cars leaving an assembly line as a measure of plant productivity; although it is possible to miscount, there can be a high degree of confidence in the measurement. On the other hand, when people are asked to rate their level of job satisfaction on a scale of 1 to 5, there is considerable room for variation in their response. They may just have had an argument with their supervisor, suffered an accident on the job, been rewarded for high levels of productivity, or been given new responsibilities. Each of these events can sway the response to the question on any given day. The individuals’ “true” satisfaction score is difficult to discern from this one question and the measure lacks reliability. OD practitioners can improve the reliability of their measures in four ways. First, rigorously and operationally define the chosen variables. Clearly specified operational definitions contribute to reliability by explicitly describing how collected data will be converted into information about a variable. An explicit description helps to allay the client’s concerns about how the information was collected and coded. Second, use multiple methods to measure a particular variable. The use of questionnaires, interviews, observation, and unobtrusive measures can improve reliability and result in more comprehensive understanding of the organization. Because each method contains inherent biases, several different methods can be used to triangulate on dimensions of organizational problems. If the independent measures converge or show consistent results, the dimensions or problems likely have been diagnosed accurately.’ Third, use multiple items to measure the same variable on a questionnaire. For example, in Job Diagnostic Survey for measuring job characteristics, the intervention variable “autonomy” has the following operational definition: the average of respondents’ answers to the following three questions (measured on a seven— point scale): 1. The job permits me to decide on my own how to go about doing the work. 2. The job denies me any chance to use my personal initiative or judgment in carrying out the work. (Reverse scored) 3. The job gives me considerable opportunity for independence and freedom in how I do the work.

Table 9: Behavioral Outcomes for Measuring OD Interventions: Measures and Computational Formulas Behavioral Outcomes for measuring OD Interventions: Measures and Computational Formulae Behavioral Measure Computational Formula


Absenteeism rate (monthly) Σ Absence days Average workforce size x working days Turnover rate (monthly Σ Tardiness incidents Average workforce size x working days Internal stability rate (monthly) Σ Turnover incidents Average workforce size Strike rate (yearly) Σ Internal movement incidents Average workforce size Accident rate (yearly) Σ Striking Workers x Strike days Average workforce size x working days Grievance rate (yearly) Σ of Accidents, illnesses Total yearly hours worked X 200,000
Σ Grievance incidents Average workforce size Σ Aggrieved individuals Average workforce size x working days Productivity Total Output of goods or services (units or $) Direct and/or indirect labor (hours or $) Below standard Actual versus engineered standard Below budget Actual versus budgeted standard Variance Actual versus budgeted variance Per employee Output/average workforce size Quality: Total Scrap + customer returns + Rework – Recoveries ($, units or hours) Below standard Actual versus engineered standard Below budget Actual versus budgeted standard Variance Actual versus budgeted variance Per employee Output/average workforce size Downtime Labor ($) + Repair costs or dollar value of replaced equipment ($) Inventory, supply and material usage Variance (actual versus standard utilization) ($) By asking more than one question about “autonomy,” time survey increases the accuracy of its measurement of this variable. Statistical analyses (called psychometric tests) are readily available for assessing the reliability of perceptual measures, and OD practitioners should apply these methods or seek assistance from those who can apply them.’’ Similarly, there are methods for analyzing the content of interview and observational data, and OD evaluators can use these methods to categorize such information so that it can be understood and replicated. Fourth, use standardized instruments. A growing number of standardized questionnaires are available for measuring OD intervention and outcome variables. 3.
Validity


. Validity concerns the extent to which, a measure actually reflects the variable it is intended to reflect. For example, the number of cars leaving an assembly line might be a reliable measure of plant productivity but it may not be a valid measure. The umber of cars is only one aspect of productivity; they may have been produced at an unacceptably high cost. Because the number of cars does not account for cost, it is not a completely valid measure of plant productivity. OD practitioners can increase the validity of their measures in several ways. First, ask colleagues and clients if a proposed measure actually represents a particular variable. This is called face validity or content validity. If experts and clients agree that the measure reflects the variable of interest, then there is increased confidence in the measure’s validity. Second, use multiple measures of the same variable, as described in the section about reliability, to make preliminary assessments of the measure’s criterion or convergent validity. That is, if several different measures of the same variable correlate highly with each other, especially if one or more of the other measures have been validated in prior research, then there is increased confidence in the measure’s validity. A special case of criterion validity, called discriminant validity, exists when the proposed measure does not correlate with measures that it is not supposed to correlate with. For example, there is no good reason for daily measures of assembly—line productivity to correlate with daily air temperature. The lack of a correlation would be one indicator that the number of cars is measuring productivity and not some other variable. Finally, predictive validity is demonstrated when the variable of interest accurately forecasts another variable over time. For example, a measure of team cohesion can be said to be valid if it accurately predicts improvements in team performance in the future. It is difficult, however, to establish the validity of a measure until it has been used. To address this concern, OD practitioners should make heavy use of content validity processes and use measures that already have been validated. For example, presenting proposed measures to colleagues and clients for evaluation prior to measurement has several positive effects: it builds ownership and commitment to the data-collection process and improves the likelihood that the client system will find the data meaningful. Using measures that have been validated through prior research improves confidence in the results and provides a standard that can be used to validate any new measures used in collecting the data. Plant: Individual:

Research Design:


In addition to measurement, OD practitioners must make choices about how to design the evaluation to achieve valid results. The key issue is how to design the assessment to show whether the intervention did in fact produce the observed results. This is called internal validity. The secondary question of whether the intervention would work similarly in other situations is referred to as external validity. External validity is irrelevant without first establishing an intervention’s primary effectiveness, so internal validity is the essential minimum requirement for assessing OD interventions. Unless managers can have confidence that the outcomes are the result of the intervention, they have no rational basis for making decisions about accountability and resource allocation. Assessing the internal validity of an intervention is, in effect, testing a hypothesis—namely, that specific organizational changes lead to certain outcomes. Moreover, testing the validity of an intervention hypothesis means that alternative hypotheses or explanations of the results must be rejected. That is, to claim that an intervention is successful, it is necessary to demonstrate that other explanations— in the form of rival hypotheses—do not account for the observed results. For example, if a job enrichment program appears to increase employee performance, such other possible explanations as new technology, improved raw materials, or new employees must be eliminated. Accounting for rival explanations is not a precise, controlled, experimental process such as might be found in a research laboratory. OD interventions often have a number of features that make determining whether they produced observed results difficult. They are complex and often involve several interrelated changes that obscure whether individual features or combinations of features are accounting for the results. Many OD interventions are long-term projects and take considerable time to produce desired outcomes. The longer the time period of the change program, the greater are the chances that other factors, such as technology improvements, will emerge to affect the results. Finally, OD interventions almost always are applied to existing work units rather than to randomized groups of organization members. Ruling out alternative explanations associated with randomly selected intervention and comparison groups is, therefore, difficult. Given the problems inherent in assessing OD interventions, practitioners have turned to quasiexperimental research designs. These designs are not as rigorous and controlled as are randomized experimental designs, but they allow evaluators to rule out many rival explanations for OD results other than the intervention itself, Although several quasi-experimental designs are available, those with the following three features are particularly powerful for assessing changes: 1.

Longitudinal measurement


. This involves measuring results repeatedly over relatively long time periods. Ideally, the data collection should start before the change program is implemented and continue for a period considered reasonable for producing expected results. 2.

Comparison unit


. It is always desirable to compare results in the intervention situation with those in another situation where no such change has taken place. Although it is never possible to get a matching group identical to tile intervention group, most organizations include a number of similar work units that can be used for comparison purposes. 3.

Statistical analysis


. Whenever possible, statistical methods should be used to rule out the possibility that the results are caused by random error or chance. Various statistical techniques are applicable to quasiexperimental designs, and OD practitioners should apply these methods or seek help from those who can apply them.

Table 10: Quasi Experimental Research Design


Quasi- Experimental Research Design Monthly Absenteeism (%) SEP. OCT. NOV. DEC. JAN FEB MAR APR Intervention group 2.1 5.3 5.0 5.1 Start of intervention 4.6 4.0 3.9 3.5 Comparison group 2.5 2.6 2.4 2.5 2.6 2.4 2.5 2.5 Table 10 provides an example of a quasi-experimental design having these three features. The intervention is intended to reduce employee absenteeism. Measures of absenteeism are taken from company monthly records for both the intervention and comparison groups. The two groups are similar yet geographically separate subsidiaries of a multi-plant company. Table 10 shows each plant’s monthly absenteeism rate for four consecutive months both before and after the start of the intervention. The plant receiving the



intervention shows a marked decrease in absenteeism in the months following the intervention, whereas the control plant shows comparable levels of absenteeism in both time periods. Statistical analyses of these data suggest that the abrupt downward shift in absenteeism following the intervention was not attributable to chance variation. This research design and the data provide relatively strong evidence that the intervention was successful. Quasi-experimental research designs using longitudinal data, comparison groups, and statistical analysis permit reasonable assessments of intervention effectiveness. Repeated measures often can be collected from company records without directly involving members of the experimental and comparison groups. These unobtrusive measures are especially useful in OD assessment because they do not interact with the intervention and affect the results. More obtrusive measures, such as questionnaires and interviews, are reactive and can sensitize people to the intervention. When this happens, it is difficult to know whether the observed findings are the result of the intervention, the measuring methods, or some combination of both. Multiple measures of intervention and outcome variables should be applied to minimize measurement and intervention interactions. For example, obtrusive measures such as questionnaires could be used sparingly, perhaps once before and once after the intervention. Unobtrusive measures, such as the behavioral outcomes shown in Table 9, could be used repeatedly, thus providing a more extensive time series than the questionnaires. When used together the two kinds of measures should produce accurate and non-reactive evaluations of the intervention. The use of multiple measures also is important in assessing perceptual changes resulting from intervention. Considerable research has identified three types of change alpha, beta, and gamma—that occur when using self-report, perceptual measures.

Alpha Change


concerns a difference that occurs along some relatively stable dimension of reality. This change is typically a comparative measure before and after an intervention. For example, comparative measures of perceived employee discretion might show an increase after a job enrichment program. If this increase represents alpha change, it can be assumed that the job enrichment program actually increased employee perceptions of discretion. If comparative measures of trust among team members showed an increase after a team-building intervention, then we might conclude that our OD intervention had made a difference.

Beta Change:


Suppose, however, that a decrease in trust occurred – or no change at all. One study has shown that, although no decrease in trust occurred, neither did a measurable increase occur as a consequence of team-building intervention. Change may have occurred, however. The difference may be what is called a beta change. As a result of team-building intervention, team members may view trust very differently. Their basis for judging the nature of trust changed, rather than their perception of a simple increase or decrease in trust along some stable continuum. This difference is called beta change. For example, before-and-after measures of perceived employee discretion can decrease after a job enrichment program. If beta change is involved; it can explain this apparent failure of the intervention to increase discretion. The first measure of discretion may accurately reflect the individual’s belief about the ability to move around and talk to fellow workers in the immediate work area. During implementation of the job enrichment intervention, however, the employee may learn that the ability to move around is not limited to the immediate work area. At a second measurement of discretion, the employee, using this new and recalibrated understanding, may rate the current level of discretion as lower than before.

Gamma change


involves fundamentally redefining the measure as a result of an OD intervention. In essence, the framework within which a phenomenon is viewed changes. A major change in the perspective or frame of reference occurs. Staying with the example, after the intervention team members might conclude that trust was not a relevant variable in their team building experience. They might believe that the gain in their clarity and responsibility was the relevant factor and their improvement as a team had nothing to do with trust. For example, the presence of gamma change would make it difficult to compare measures of employee discretion taken before and after a job enrichment program. The measure taken after the intervention might use the same words, but they represent an entirely different concept. The term “discretion” may originally refer to the ability to move about the department and interact with other workers. After the intervention, discretion might be defined in terms of the ability to make decisions about work rules, work schedules, and productivity levels. In sum, the job enrichment intervention changed the way discretion is perceived and how it is evaluated. These three types of change apply to perceptual measures. When other than alpha changes occur, interpreting measurement changes becomes the more difficult. Potent OD interventions may produce both beta and gamma changes, which severely complicates interpretations of findings reporting change or no change. Further, the distinctions among the three different types of change suggest that the heavy reliance on questionnaires, so often cited in the literature, should be balanced by using other measures, such as



interviews and unobtrusive records. Analytical methods have been developed to assess the three kinds of change, anti OD practitioners should gain familiarity with these recent techniques.

Case: The Farm Bank


The Farm Bank is one of the state’s oldest and most solid banking institutions. Located in a regional marketing center, the bank has been active in all phases of banking, specializing in farm loans. The bank’s president, Frank Swain, 62, has been with the bank for many years and is prominent in local circles. The bank is organized into six departments (as shown in Figure below). A senior vice president heads each department. All six of them have been with the bank for years, and in general they reflect a stable and conservative outlook.

The Management Information System


Two years ago, President Swain felt that the bank needed to “modernize its operations. With the approval of the board of directors, he decided to design and install a comprehensive management information system (MIS). The primary goal was to improve internal operations by supplying necessary information on a more expedited basis, thereby decreasing the time necessary to service customers. The system was also to be designed to provide economic operating data for top management planning and decision-making. To head this department he selected Al Hassier, 58, a solid operations manager who had some knowledge and experience in the computer department. After the system was designed and installed, Al hired a young woman as his assistant. Valerie Wyatt was a young MBA with a strong systems analysis background. In addition to bring the only woman and considerably younger than any of the other managers at this level, Wyatt was the only MBA. In the time since the system was installed, the MIS has printed thousands of pages of operating information, including reports to all the vice presidents, all the branch managers, and the president. The reports include weekly, monthly, and quarterly summaries and include cost of productions, projected labor costs, overhead costs, and projected earnings figures for each segment of the bank’s operations. The MIS Survey Swain was pleased with the system but noticed little improvement in management operations. In fact, most of the older vice presidents were making decisions and function pretty much as they did before the MIS was installed. Swain decided to have Wyatt conduct a survey of the users to try to evaluate the impact and benefits of the new system. Wyatt was glad to undertake the survey, because she had long felt the system was too elaborate for the bank’s needs. She sent out a questionnaire to all department heads, branch managers, and so on, inquiring into their uses of the system. As she began to assemble the survey data, a pattern began to emerge. In general, most of the managers were strongly in favor of the system but felt that it should be modified. As Wyatt analyzed the responses, several trends and important points came out: (1) 93 percent reported that they did not regularly use the reports because the information was not in a useful form, (2) 76 percent reported that the printouts were hard to interpret, (3) 72 percent stated that they received more data than they wanted, (4) 57 percent reported finding some errors and inaccuracies, and (5) 87 percent stated that they still kept manual records because they did not fully trust the MIS.

The Meeting


Valerie Wyatt finished her report, excitedly rushed into Al Hassler’s office, and handed it to him. Hassler slowly scanned the report and then said, “You’ve done a good job here, Val. But now that we have the system operating, I don’t think we should upset the apple cart, do you? Let’s just keep this to ourselves for the time being, and perhaps we can correct most of these problems. I’m sure Frank wouldn’t want to hear this kind of stuff. This system is his baby, so maybe we shouldn’t rock the boat with this report.” Valeries returned to her office feeling uncomfortable. She wondered what to do.

Case Analysis Form Name: ____________________________________________ I. Problems


A. Macro 1. ____________________________________________________ 2. ____________________________________________________ B. Micro 1. _____________________________________________________ 2. _____________________________________________________

II. Causes


1. _____________________________________________________ 2. _____________________________________________________



3. _____________________________________________________

III. Systems affected


1. Structural ____________________________________________ 2. Psychosocial __________________________________________. 3. Technical ______________________________________________ 4. Managerial _____________________________________________ 5. Goals and values __________________________________________

IV. Alternatives


1. _________________________________________________________ 2. _________________________________________________________ 3. ________________________________________________________

V. Recommendations


1. _________________________________________________________ 2. __________________________________________________________ 3. __________________________________________________________

Case Solution: The Farm Bank I. Problems A. Macro


1. Client system unprepared for change. 2. Client system unfamiliar with and unprepared for MIS.

B. Micro


1. Top-down approach (Swain’s) excluded staff from decision and preparation for MIS. 2. Survey should have preceded, not followed, MIS. 3. Hassler not assertive enough to fulfill Swain’s goals by keeping Swain informed. 4. Particulars in MIS need to be changed (limit info after determining needs, change format, etc.). 5. Valarie Wyatt has been charged by Swain to make survey but her boss, Hassler, has told her not to give the report to Swain.

II. Causes


1. Conservative nature of firm (and age of staff). 2. Lack of education regarding MIS. 3. Lack of planning regarding functions MIS would perform for managers and firm. 4. Hassler more interested in personal security than in fulfilling purpose for which he was hired.

III. Systems affected


1. Structural - Chain of command prohibited Wyatt from improving MIS through using results of report. 2. Technical - MIS needs new form and new limitations. These are not being carried out. 3. Behavioral – Wyatt’s “fulfillment” and satisfaction of job well done are restricted. Other staff’s expectations brought on by survey are frustrated by lack of follow-through. Swain hopes are not fulfilled. Hassler knows, somewhere, he is not fulfilling his role. Managerial decisions companywide are not being made in the best possible way, since information is not being managed in the most effective way possible. 4. Managerial – Hassler is uncomfortable about taking things up the chain. Possibly the president, Frank Swain, has intimidated subordinates in the past. Or Hassler does not want to rock the boat, has a “full plate”, or maybe is lazy. It is difficult to access motives of managers. 5. Goals and values – Excellence and organization improvement does not seem to be valued by most managers except possibly Wyatt.

IV. Alternatives


1. Wyatt could convince Hassler it’s in his best interest to show Swain results of survey. 2. Wyatt could go along with Hassler’s inaction. 3. Wyatt could go around Hassler and tell Swain.

V. Recommendations


Wyatt needs to submit the report to Swain since this is the person who assigned her to do the survey. She needs to explain tactfully to Hassler the importance of her giving Swain the report. Once the report is sent to Swain, The Farm Bank needs to embark on a strategy of solving the problems identified in the survey. The approach should be an integrated one involving the people who use the MIS with them identifying specific problems and the steps to correct the problems. Hassler needs to be involved in making the changes as well as Wyatt.

<Previous Lesson

Organization Development

Next Lesson>

Home

Lesson Plan

Topics

Go to Top

Copyright © 2008-2013 zainbooks All Rights Reserved
Next Lesson
Previous Lesson
Lesson Plan
Topics
Home
Go to Top