Business Simulations and Assessment

I recently attend the Faculty Conference on Learning sponsored by the Association to Advance Collegiate Schools of Business (AACSB), an entity "of educational institutions, corporations and other organizations devoted to the promotion and improvement of higher education in business administration and management."  AACSB sets the accreditation standards by which business schools are measured. The conference had a track on business games, so it was only natural that the issue of simulation efficacy and assessment was discussed.  These discussions affect simulation design and use, especially for those of us developing business game/simulations for higher education.

There's been lots of discussion, both research-based and speculative, about the efficacy of simulations in general and business simulation games in particular (some links are provided at the end of this article). One set of faculty panelists were proposing that business simulations could not only enhance learning and achieve learning objectives, but also quantify that learning by mapping outcomes to specific AACSB standards for assurance-of-learning.  I had personally never seen simulation outcomes mapped to academic standards in such a detailed manner, and the ramifications are significant.  If simulation outcomes were widely adopted by standards bodies as qualifying learning methods then designers of academic simulations would very likely start to map and target these outcomes to a greater degree.

The title of the session was "How Four Schools Use Business Simulation to Assess Learning", and the panelists were:

• Ernest Cadotte, Professor of Marketing, College of Business Administration, University of Tennessee at Knoxville
• Richard Riley, Professor, College of Business, West Virginia University
• Denise J. Luethge, Associate Professor, College of Business and Public Administration, Drake University
• William Patterson, Professor, College of Business Administration, Sullivan University

The simulation that these professors use is the Marketplace simulation produced by Innovative Learning Solutions.  Marketplace is one of the most comprehensive and impressive business simulations available today.  However, I should note here that Ernest Cadotte is one of the founders of the Marketplace simulation, and in general the AACSB conference did little to inform session participants when a speaker had a business relationship with the product that was being showcased (there were other examples of this as well).  Many of the faculty members attending are doing so in order to increase their minimal understanding of technology services and products available to them and they put faith in any teacher who stands before them and tells them something works.  It was interesting to see how this potential conflicts of interest was handled (or avoided, in this case) versus how I imagine it would be disclosed at a more corporate/professional conference setting.

That said, the claim they make regarding the mapping of Marketplace outcomes to AACSB standards is an interesting one. They started by stating that the responsibilities of educators have changed -- no longer are they merely focusing on the inputs to education, via designing courses that aim to provide skills necessary to be effective in business, but now they also must focus on the outputs of education, assessing the extent to which that skill development took place.  They acknowledged that with this change comes a change in accountability -- it is the educators, not the learners, who are on the hook to ensure learning takes place. 

They begin by employing a standard instructional design process: define learning objectives, design courses and curricula to meet those objectives, assess how well the objectives are being met, followed by adjusting either the objectives or the methods to better meet the learning objectives. Simulations are just one method -- a method Cadotte considers "combative training" where participants "pit their business skills against those of formidable opponents under the watchful eye of a training coach." He went on to review the basic "hybrid" format of the Marketplace simulation experience (meaning that some activities and analysis occur outside the simulation): students are put into teams, have to prepare and execute a business plan, have to adjust tactics to react against exogenous forces, and have to prepare a final presentation to a "board of directors".  Marketplace is "capstone" simulation, fairly complex and hence usually used over several weeks or even an entire term to allow students time to adjust to the learning curve of using the simulation itself.

Then much of the presentation focused on defining the larger skillsets developed by the simulation (such as "critical thinking") followed by specific claims of where/how the simulation activity fostered development of those skillsets (team discussion and debate, decision management with demanding customers and aggressive competitors, preparation of plans/reports, partner negotiations, etc.).  Then each presenter described in detail how the simulation skillsets would map to specific AACSB "Assurances of Learning." 

Some examples:

AACSB Assurance of Learning standard: Program graduates will be prepared to assume positions of greater leadership throughout their careers.
"Essence of Marketplace Simulation" addressing this standard:

• Competence, Vision, and Teamwork as aspects of Leadership are fostered throughout the simulation.
• The simulation requires strategic thinking and disciplined response.
• The simulation presents the same challenges that face corporate leaders.

AACSB Assurance of Learning standard: Program graduates will possess superior technology-based solutions to emerging business problems. Graduates will be able to use computer and information technology in solving problems and performing functions commonly seen in managing business and other organizations.
"Essence of Marketplace Simulation" addressing this standard:

• The simulation itself is delivered via the web and using it affords some measure of technology immersion.
• Simulation operation and analysis requires Excel and PowerPoint skills.
• The simulation provides some opportunity for statistical analysis.

AACSB Assurance of Learning standard: Program graduates will develop an appreciation for the global nature and impacts of the regional economy.
"Essence of Marketplace Simulation" addressing this standard:

• The simulation involves managing five regions of the world: US, Canada, South America, Europe & China.
• Target customers have differing characteristics depending on region.
• Each region has a different operating cost structure.

These are just a few examples.  Others included value chain management, using technology to measure business performance, global macro business and economic environment, business decision-making in a team context and/or in an "integrated" context, business communication skills, etc. The assessments differed by faculty and class level as well, but generally included some weighted measure of simulation performance, peer evaluation, various team presentations (final presentations as well as presentations to student or faculty "Venture Capital" directors, boards of directors, etc.), business plan assessments, etc. They also gave insights into why they thought this simulation worked well.  For instance, they thought that the competitive nature of business/MBA students and their desire to "win" (the ends) often clouded their thinking and action (the means) enough to ensure that learning points would surface.

The most interesting post-presentation discussion concerned faculty who questioned whether team output from simulations could be attributed to these AACSB Assurance of Learning standards since they had been designed for individual students.   The presenters claimed that this would not be a problem, but some audience members claimed to have been told otherwise by AACSB.  No AACSB representative made any official ruling on this during the presentation.  So this is clearly an area that faculty will be interested in receiving clarification.

It was the contention of these faculty that the Marketplace simulation provided tangible proof of learning outcomes that could be successfully mapped back to some primary learning objectives within the business school curriculum  As mentioned earlier, it needs to be noted that these faculty are somewhat biased in this claim.  Also, other factors affecting teachability and usability were not discussed.  For instance, during our research at Harvard Business School Publishing into the simulation practices and preferences of our customers we know that the Marketplace simulation can suffer from certain deficiencies, such as the administrative overhead required to run and especially advance the simulation from round to round. But the larger goal of trying to map simulation effectiveness to learning objectives (in effect to document the transference of learning from the simulation environment to the real world) is a valid effort.  Here is some additional information on the effectiveness of simulations:

• See earlier posts in this blog on an overview of simulations as well as simulations and future of learning.
• Harvard Business School Publishing has partnered with Forio Business Simulations to develop our online business simulation games.  Forio's website has a nice overview of simulation effectiveness that is broad and balanced.
• The June 2004 issue of the journal Simulation & Gaming (put out by the Association for Business Simulation and Experiential Learning) has a number of fantastic, academically-rigorous examinations of business simulation effectiveness.  In general this journal is a great resource, but this issue in particular has some articles that focus on this area:
  • "The Experiential Method and the Business Gaming Field" by Joseph Wolfe (creator of the Global Business Game -- Wolfe also presented at the conference on "Using Business Games - An Inventory of Best Practices" -- while the session abstract didn't mention his authorship fo the GBG, at least his bio did).
  • "A Survey of Simulation Game Users, Former-Users, and Never-Users" by Anthony J. Faria and William J. Wellington.  This is a phenomenal survey overview of the business simulation market as reported by business faculty.
  • "A Review of Scholarship on Assessing Experiential Learning Effectiveness" by Jerry Gosen and John Washbush. This is a very detailed analysis of the methods and results of research on simulation effectiveness.