Mining for Problem-solving Styles in a Virtual World

Brian M. Slator
Department of Computer Science
North Dakota State University
USA
slator@cs.ndsu.edu

Donald P. Schwert
Department of Geosciences
North Dakota State University
USA
donald_schwert@ndsu.nodak.edu

Bernhardt Saini-Eidukat
Department of Geosciences
North Dakota State University
USA
bernhardt_saini-eidukat@ndsu.nodak.edu

Abstract:  The Geology Explorer provides a multi-modal virtual environment that implements an educational game for teaching principles of geology. The game is a networked, multi-player, simulation-based, educational environment that illustrates our role-based pedagogical approach.  This takes the form of a synthetic, virtual world (Planet Oit) where students are given an authentic experience and the means and equipment to explore a planet as a geologist would.  Each student's experience includes elements of exploration of a spatially-oriented, virtual, world; practical, field oriented, expedition planning and decision making; and scientific problem solving (i.e. a “hands on” approach to the scientific method).  Students assume a role and learn about real science by exploring in a goal-directed way and by competing with themselves and with other players. 
	In this paper, data are reported from a 1999 study, in which students enrolled in a 
freshman-level Physical Geology course explored the planet for credit.  These data were collected in two forms: 1) a survey of student perceptions, positive and negative; and 2) a data mining analysis of student histories which reveals apparent categories of student problem-solving style.  Planet Oit can be visited at http://oit.cs.ndsu.nodak.edu/

Introduction
	The Geology Explorer (Saini-Eidukat, Schwert & Slator 1999; Schwert, Slator & Saini-Eidukat 1999) is a virtual world where learners assume the role of geologists on an expedition to explore a mythical planet: Planet Oit, which is designed to replicate the physical environments of Earth (and in the same orbit, but directly opposite the Sun). Learners participate in field-oriented expedition planning, sample collection, and “hands on” scientific problem solving. 
	To play the game, students are transported to the planet's surface and acquire a standard set of field instruments. They are issued an electronic logbook to record their findings and, most importantly, are assigned a sequence of exploratory goals. The students make their field observations, conduct small experiments, take note of the environment, and generally act like geologists as they work towards their goal.  A scoring system has been developed, so that students can compete with each other and with themselves. 
	The Geology Explorer is being developed as a synthetic environment using the freely available Xerox PARC LambdaMOO (Curtis 1997; Bruckman, 1997), which is an environment for creating text-based virtual worlds, to simulate a portion of Planet Oit.  Students armed with tools and instruments created as LambdaMOO objects land on the planet to undertake an exploration exercise.  They are given an authentic geologic goal, e.g., to locate and report the position of potentially valuable mineral deposits.  Accomplishing each of these goals entails mastering several geologic concepts and procedures, and demonstrates student mastery of the material.  The first module involves mineral exploration, where students are expected to plan an expedition, locate mineral deposits, and survive the somewhat hostile virtual environment in order to report on it (future modules on hydrology, metamorphic facies, etc. are underway).  The first version of the Geology Explorer is text-based; (Slator, Schwert, & Saini-Eidukat 1999); a graphical interface was launched in Fall, 2000.
	Planet Oit is designed to emulate the geologic features and processes of Earth. The first version is based on a realistic planetary design, consisting of a single, super-continent composed of roughly 50 locations (Figure 1), arranged so as to be both diverse and coherent.  A variety of Earth-like environments, ranging from tropical coastlines to volcanic calderas to glaciated peaks, allows for multiple geologic terrains to be explored. A museum of rocks and minerals is available at the landing site for use as a standard reference collection.  Coordination of navigation on the planet is made possible by using directions relative to Earth-like geographic poles (North, South, etc.).
	Implemented, as well, are almost 40 scientific instruments and geologic tools (streak plate, acid bottle, magnet, etc.), nearly 100 different rocks and minerals, and over 200 boulders, veins, and outcrops.  In the text-based version, students use a command language, which allows for navigation, communication, and scientific investigation while on the planet.  Command verbs dictate the student’s application of instruments (“streak,” “scratch,” “hit,” etc.) and senses (“view,” “taste,” “touch,” etc.).  Students can communicate, and therefore work, with one another through verbal commands.  An on-line "user card" listing all these commands and functions is available at: http://oit.cs.ndsu.nodak.edu/oit/usercard.html.
	Once the layout and artifacts of Planet Oit were implemented, the rules of the game were imposed over top.  After being transported to the planet’s surface, students are automatically assigned an initial exploratory goal and can acquire whatever equipment they wish.  The goals are intended to motivate the students into viewing their surroundings with a critical eye, as a geologist would.  Goals are assigned from a principled set, so as to leverage the role-based elements of the game by gradually leading students to more difficult and interesting goals.
	In order for a student to achieve a goal (and therefore earn points), they must address a multitude of tasks identical to those faced daily by field geologists.  These include the selection and use of proper field tools, navigation across the planet to the correct region, and interpretation of the tests applied to the problem.  As each goal is satisfactorily completed, the student is automatically assigned new goals requiring progressively higher levels of expertise and decision-making.  Through this practical application of the scientific method, students learn how to think, act, and react as geologists (see Duffy & Jonassen 1992).  This is a particular strength.  Student progress is tracked in terms of goals achieved  -- and students have a self-paced, anytime/anywhere learning experience.

Classroom Context
	Physical Geology at North Dakota State University is a large-enrollment (400+ students in one large lecture section), 3 semester hour course.  Aside from lecture, the course content is augmented by slides, a set of course lecture templates, a textbook, and a web resource site that includes self-quizzes, photographs, course news, and links to related resources.  Testing is by multiple-choice exams, with students submitting their results on optical scan sheets.  Nearly 100% of the students enroll in the course to complete either general education science requirements or specific course requirements within their majors.  Because the class is so large in enrollment, it demonstrates the obvious impracticality of field-training each student to think and behave as a scientist, and as a geologist. 
	In the Fall semester of both 1998 and 1999, the Geology Explorer was incorporated into the Physical Geology curriculum.  Data were gathered with a view toward 1) answering several questions about student use of technology and 2) student perceptions of, and satisfaction with, the Planet Oit simulation using an on-line evaluation questionnaire. 
	The primary goal was to investigate the effect on the student experience with Physical Geology as consequence of introducing the Geology Explorer prototype as a supplementary resource to classroom instruction for a non-major introductory course.  To do this, we implemented tracking routines on Planet Oit in order to get statistics for time-on-task, correlations for computer literacy and attitudes towards technology, effect on final grade.  We anticipated that these data would lead to a classification by learning style.

The Student Experience
	In the Fall, 1999, 81 students completed a Geology Explorer assignment, scoring the required 500 points, and then completed an on-line follow-up evaluation form.  This form is web-based and requires identification information (e.g. name, student ID number, and e-mail address), and is composed of 35 questions about the Planet Oit experience (the full form is on-line at http://oit.cs.ndsu.nodak.edu/~mooadmin/cgi-bin/oitform.html ). 
	In the post-test evaluation, 82.8% of the students said they somewhat agreed or strongly agreed they had learned something from the game, and only five students (5.4%) disagreed or strongly disagreed that they had learned something.  At the same time, only 9.7% somewhat or strongly disliked the concept of game, and 62.4% thought they might like to play the game again.  Meanwhile, the students perceived the game to be at an appropriate level of difficulty, with only 8.6% describing the game as much too complex, and 0 students believing it was much too simple. 
	For the 81 students who completed the assignment and the post-test evaluation, the average actual time on task was 3.47 hours with a range from 0.83 hours to 8.04 hours (while the average estimated time on task was 5.2 hours with a range from 1 to 12.5 hours).   Of this group, 11 (13.6%), underestimated time spent on the planet, 12 (14.8%), overestimated their time on task by 25% or more, and 58 (71.6%), accurately estimated the time they spent on the planet.

Learning Style
Through the course of the experiment, and by interacting with students both off- and on-line, we came to believe that identifiable learning and problem-solving styles were being employed by the students.  Some students appeared to take an analytical approach: frequently referencing the on-line help, conducting sequences of experiments, and making diagnoses leading to their scoring points in a deliberate fashion.  Other students seemed to take a pattern-matching approach: exploring far and wide in search of outcrops that seemed to match the description of what they were looking for, and then scoring points with relatively few experiments.  There was also a small but noticeable group taking a straight “brute force” approach, simply visiting location after location and identifying everything there, one after another, as their goal, eventually succeeding after many tries.  One monument to this approach was a student in 1998 who made 1,244 guesses on the way toward obtaining just five correct answers.
	To investigate the nature of the apparent trends, an analysis was conducted using logging data to count the number of “reports” (i.e. guesses), the number of locations visited, and the number of experiments conducted (e.g. hit, scratch, streak, etc.), for the same 81 students who had completed the game and the evaluation survey in 1999.  These data are summarized in Table 1.

					Reports		Moves		Experiments	
average		42.6		139.2		73.8	
st. dev		38.2		83.1		63.2	
min		5		19		0	
max		238		518		301	

Table 1:  Student Reports, Moves, and Experiments in 1999

	Using these values for average and standard deviation, we developed a classification of behaviors by looking for combinations of either much higher or much lower than average activities in terms of reports, moves, or experiments, or combinations of these.  These data are summarized in Table 2.  There are a total of 24 clusters, each marked with a code, which represent the three significant categories.
	The values in Table 2 indicate that a wide range of approaches are supported by the Geology Explorer, a testament to the user-centered and user-controlled nature of the simulation.  Further, almost half of the students can be classified as consistently efficient, not only economizing on their problem-solving effort, but doing so across all three dimensions.
	Meanwhile, over half were above normal in one or more dimension, with 17 making excessive reports (code “R,” the “brute force” approach); 19 making excessive movements around the planet (code “M,” the pattern matching approach), and 22 making more than the normal number of experiments (code “E”).  Note that three students were excessive on all three dimensions, and only two students were within 1/2 standard deviation on all three.

Consistently normal or below normal activity	Consistently normal or above normal activity	Mixed problem-solving activity	
rme	10	-ME	5	-Me	4	
r-e	8	--E	4	rM-	2	
r--	5	R-E	4	r-E	2	
-m-	5	R--	3	RmE	2	
-me	4	RME	3	---	2	
--e	4	RM-	3	Rm-	1	
rm-	4	-M-	2	-mE	1	
		rmE	1			
		R-e	1			
		Rme	1			
Total (49.4%)	40	Total (29.6%)	24	Total (21.0%)	17	

Table 2:  Learning / Problem-Solving Styles

Note: R = many reports; r = few reports; M = many moves; m = few moves; 
E = many experiments; e = few experiments. 
Example: “-Me” means normal reporting, many moves, below normal 
experiments (where normal is within one-half standard deviation from the mean). 

Discussion
We can only speculate, at this point, what these data mean.  We would seem to be seeing a great deal of apparent variability in student style which could point to basic differences, or which might simply be a function of pre-conceived notions on the part of the students as to how interactive software games usually work.  That is, a certain number of students might be tempted to “game” the system -- i.e. devote their energies toward learning to “beat the game” rather than learning the geological content the game is meant to convey. 
	Gaming the system in more-or-less constant concern in efforts of this type and a problem that we, as designers, must be constantly vigilant against.  However, it must be recognized these issues are not strictly confined to computing media.  Students are resourceful, and there is a long history of anecdotal accounts of students, for example, “cracking the code” of lab samples in order to pass a test.   Anyone who has taught laboratory sections for any length of time has similar stories on this theme.
	We can make only preliminary claims, at this stage, to a definitive classification of student problem-solving categories.  For example, above normal “M” in Table 2, indicating movement and exploration more than 1/2 standard deviation above the mean, might not directly indicate the “pattern matching” strategy mentioned above, but it could simply indicate a high degree of curiosity in some students.  Alternatively, it could mean (modesty forbids our promoting this alternative), that our virtual environment is so exciting and filled with wonders that students are exceedingly engaged and feel compelled to see everything they can (19 students were coded “M,” we note, over 23% of the total).
	By the same token, 22 students were “excessive,” if that's the right word, in terms of code “E,” for experimentation in Table 2.  This is hard to criticize on any level, as experimentation is, in one sense, what we hope to teach.  Perhaps these students were repeating experiments because our logging procedures were too inaccessible and they re-did experiments rather than refer to their logbooks.  If so, this is a failing of the software that we must try to address. 
	Lastly, we find that 17 students made excessive reports.  This was the indicator that led us to this data-mining investigation in the first place: the intuition that some students (in these data, 21%) were taking a “brute force” approach to their assigned goals.  Is this supported?   We suspect so.  But there are many open questions, the primary of which is how to encourage these students (if we're right in this supposition), to be more analytical in their approach.  One aspect we would like to track in the future is the effect of our software tutors in terms of steering the gamers, and the truly overwhelmed, towards more deliberative strategies. 

References
Bruckman, A. (1997). Finding One's Own in Cyberspace. In C. Haynes & J.R. Holmevik, (Eds.), High Wired: On the Design, Use, and Theory of Educational MOOs). Ann Arbor: University of Michigan Press. 

Curtis, P. (1997). Not Just a Game: How LambdaMOO Came to Exist and What It Did to Get Back at Me. in Cynthia Haynes and Jan Rune Holmevik, Editors: High Wired: On the Design, Use, and Theory of Educational MOOs. Ann Arbor: University of Michigan Press.

Duffy, T.M., & Jonassen, D.H. (1992). Constructivism: new implications for instructional technology. In Duffy and Jonassen (eds.), Constructivism and the Technology of Instruction. Hillsdale: Lawrence Erlbaum.

Saini-Eidukat, B., Schwert, D., & Slator, B.M. (1999). Designing, Building, and Assessing a Virtual World for Science Education. Proceedings of the 14th International Conference on Computers and Their Applications (CATA-99), April 7-9, Cancun.

Schwert, D.P., Slator, B.M., & Saini-Eidukat, B. (1999). A virtual world for earth science education in secondary and post-secondary environments: The Geology Explorer. International Conference on Mathematics/Science Education & Technology (M/SET-99), March 1-4, San Antonio, TX.

Slator, B.M., Schwert, D.P., & Saini-Eidukat, B. (1999). Phased Development of a Multi-Modal Virtual Educational World. Proceedings of the International Conference on Computers and Advanced Technology in Education (CATE'99), Cherry Hill, NJ, May 6-8.

Acknowledgements
	Development of the Geology Explorer is funded by the National Science Foundation under grants DUE-9752548, EAR-9809761, and EIA-0086142.  We also acknowledge the large team of dedicated undergraduate and graduate students in the computer and earth sciences who have made this project so successful.  Special thanks are due to John Bauer for Java graphical client development, to Rebecca Potter for graphical development, to Otto Borchert for simulation development, to Dean Vestal, Ned Kruger, Bryan Bandli, and Jane Willenbring, for geology content development and assessment, to Mark Tinguely, who saved our world when its universe imploded, and to Dave Schmidt for the name: Planet Oit.