Continuous Participation in Stakeholder Assessment Studies Shows Link to Improvement in Performance

The initial findings from a recent research study conducted by Educational Benchmarking, Inc. (EBI) has revealed evidence of a link between continuous participation in the AACSB/EBI Full-time MBA and Undergraduate Stakeholder Assessment Studies and improvements in performance, as judged by students. Improved performance was based on continuous annual project participation over a four-year period. Full-time MBA programs, on average, demonstrated statistically significant improvement on all eight survey factors common across all four years of the study, including "overall satisfaction." Student perceptions of quality for undergraduate business programs, on average, demonstrated statistically significant improvement on nine out of 10 survey factors common across all four years of the study, including "overall satisfaction."

Attributes used to propel change
The AACSB/EBI Stakeholder Assessment Studies’ unique combination of mission-driven surveys, confidential custom comparisons with peer programs, targeted results analyses identifying critical factors to overall satisfaction, and monitoring of performance over time through longitudinal analysis of results are associated with improved performance.

Shedding new light on the old adage "you get what you measure," the initial results associated with the methodology used in the studies show promise as a catalyst for improved performance. Attributes of the studies include:

Credibility
Results from a stakeholder assessment study must have credibility among faculty and administrators who see those results. Such credibility comes from a rigorous methodology of design and analysis, best if developed externally and proven over time.

Comparison
Results must be presented within a comparative context to provide maximum value. When results are viewed in isolation, it is difficult to determine how performance compares with others trying to achieve the same result. The most meaningful comparisons are those with organizations identified as peers or competitors.

Confidentiality
When competitors are sure that the data they present will remain confidential, they are more likely to provide accurate information and are more likely to place a higher level of credence in the information and analysis provided by others. Program leaders and faculty are therefore more likely to initiate changes based on the results.

Comprehensive analysis
Studies that are clearly viewed as comprehensive diminish uncertainty. There is a persuasive sense that all the ground has been covered, all the questions have been asked, all necessary information is provided.

The results show the potential impact of an effective process. In the AACSB/EBI Stakeholder Assessment Studies administered to graduating full-time MBA and undergraduate business students, programs that have used the data in a reiterative process have seen substantive positive results.

Link Between Continuous Participation and Continuous Improvement
In conducting a study of data from four years of continuous participation in the full-time MBA and Undergraduate Student Stakeholder Assessment Studies, EBI found the following:

•Student perceptions of quality for 20 full-time MBA programs participating in the studies on an annual basis over the past four years, on average, demonstrated statistically significant improvement on all eight survey factors common to all four years of the study, including "overall satisfaction." (Results are based on the difference in the responses from students at all institutions over the period of analysis.)

•Student perceptions of quality for 41 undergraduate business programs participating in the studies on an annual basis over the past four years, on average, demonstrated statistically significant improvement on nine out of 10 survey factors common to all four years of the study, including "overall satisfaction."

Improvement in Factor Means Among Continuously Participating Full-Time MBA Programs
Table 1 shows the increase in factor means among the full-time MBA programs that participated continuously in the studies. Seventy-one percent of full-time MBA programs participating continuously in the AACSB/EBI studies experienced a statistically significant improvement in overall satisfaction with the MBA program when comparing 1997 results to 2000 results.

Improvement in Factor Means Among Continuously Participating Undergraduate Business Programs
Table 2 shows the increase in factor means in the Undergraduate Stakeholder Assessment Studies between 1997 and 2000. Among undergraduate programs participating continuously in the study, 33 percent improved by a degree that is statistically significant on the "bottom-line" factor of "Overall Satisfaction with the Program." Satisfaction with classmates improved among 75 percent of the schools participating continuously.

Continuous Improvement: The Iterative Process
It has been suggested that continuous feedback from participation in the annual student studies helps business programs undertake an iterative process of:

•Gaining initial feedback

•Using the initial feedback to guide change

•Gaining additional feedback to measure the impact of the change

•Using the new feedback to fine-tune the changes, or to make additional changes.

Separating the Wheat From the Chaff: Identifying What Really Matters
Guided by information gathered over time from a series of graduating students, schools should identify a student perspective regarding what they need to improve, develop and implement appropriate interventions, and assess the degree to which their efforts are successful. "For programs to achieve improved overall performance, they must focus their attention on factors that are most highly correlated with overall satisfaction," said Joe Pica, EBI managing director. "At times institutions may be inclined to focus their energies on the lowest-rated factors or on the factors where improvement seems most easily achieved. It is not always the case that the factors with the lowest performance, if improved, will have an impact on overall satisfaction," he said. Schools and programs that have participated continuously in the studies have succeeded in identifying those issues that matter the most to students and have taken project data as significant input to achieve improvement in overall satisfaction, according to Pica.

Another element essential to improved performance is the utilization of iterative feedback from longitudinal analysis to fine-tune or revamp intervention efforts. "Change initiatives will yield unintended as well as intended results," said Pica. "Improvement is a process of strengthening the intended positive consequences and diminishing the negative unintended consequences. Continued participation provides the feedback necessary to refine efforts and improve performance over time," he said.

The correlation between continuous participation in the AACSB/EBI Stakeholder Assessment Studies and improved performance provides evidence that benchmarking may be a catalyst for change and continuous improvement.