Evaluating Pedagogical Quality of Learning Activities Using Fuzzy Evaluation Mappings: The Case of Pedagogical Games of Mathematical Proof ()
1. Introduction
Self-efficacy is a student’s set of beliefs in her or his capacity to successfully execute a behavior necessary to produce a positive learning performance attainment [1] [2] [3]. Students with high self-efficacy beliefs allocate more effort when they encounter learning difficulties compared to students with low self-efficacy who often don’t make any notable attempt to achieve a learning goal, as they a priori belief that they will fail to achieve it [4] [5].
A unique type of anxiety is mathematics and statistics anxiety. It is unique in this regard, as there are no widespread anxiety conditions for other specific learning content areas such as reading, writing, or history1. Mathematics and statistics anxiety is a negative emotional reaction to the manipulation of numbers and symbols that can be debilitating. It has been defined as “a feeling of tension and anxiety that interferes with the manipulation of numbers and the solving of mathematical problems in…ordinary life and academic situations [6] [7].” Recent research shows that a key factor negatively impacting mathematics anxiety is self-efficacy beliefs toward learning mathematics [8] [9]. This finding is in accordance with the observation that students with lower self-efficacy beliefs toward learning mathematics have higher levels of mathematics anxiety [10] [11] [12] [13] [14], a relation consistent across gender groups, grade-level groups, ethnic groups, and instruments measuring mathematics anxiety [15] [16].
To help students overcome anxious feelings toward mathematics educators recently started to incorporate active learning elements in their instruction with pedagogical successes reported in attitudes towards mathematics studies [16] [17] [18] [19]. Active learning in mathematics is a recent pedagogical development in response to a “call on institutions of higher education, ..., to ensure that effective active learning is incorporated into post-secondary mathematics classrooms”.2 Prior to this call a land marking paper showed in a meta analysis of 225 studies that active learning improves learning and course performance in undergraduate STEM courses relative to traditional lecturing [20]. This research further suggests that active learning does not harm, and may further benefit, already high-achieving students. It also shows that students in classes with traditional lecturing were more likely to fail compared to students in classes with active learning. Active learning helps students effectively engage in their learning process and improve their performance attainment [21]. The basic premises of active learning include instructional techniques reinforcing higher-order thinking, requiring students to co-create their own knowledge through active participation in the learning process [22] [23].
Game based learning is an emerging active learning instructional method in mathematics with several advantages [24]: it increases student engagement [25] [26], enhances self-efficacy [18], improves student achievement [27], motivation and resilience [28]. The literature on pedagogical games supporting teaching and learning of mathematics in higher education courses is scarce, and largely focuses on online settings [29] [30]. The game introduced in this paper, however, uses a “toy” to help students put hands on mathematics to physically flip and or rotate wooden objects to fit their counter-shape holes in a box. Students then write down their observations into a table from which mathematical proofs can be deduced. Even advanced mathematical concepts such as Abelian group properties of symmetries can be deduced from this method by playfully constructing a Cayley table. The activity considered here, enables students to derive proofs without calculations. There is currently little evidence in the literature about the effectiveness of using toys as an active learning activity to derive proofs “playfully”. This paper sets out to investigate this by developing a formal framework for constructing pedagogical games of mathematical proof and their pedagogical quality evaluations.
It is generally difficult to determine if a pedagogical game is devised with the explicit intention of meeting educational criteria, or whether researchers are appropriating existing games to test their possible educational impact [29] [31] [32]. Moreover, how would one compare the effectiveness of alternative pedagogies relative to each other? Naik (2017) suggests that the effectiveness of pedagogical games should ultimately be measured in terms of student learning outcomes [29]. In this paper, we consider a geometric approach based on fuzzy mappings to establish a pedagogical quality assurance system in which pedagogical effectiveness is only one dimension. We determine the quality of a pedagogical game by considering three main learning dimensions including: learning effectiveness, learning efficiency, and learning satisfaction. We utilize the concepts of fuzzy variables and metric functions to obtain a pedagogical quality assurance evaluation method.
The organization of this paper is as follows. Section two provides a formal framework for developing pedagogical games of mathematical proof. We apply this pedagogical framework in section four where we provide an example. Section three introduces the PGMP pedagogical quality assurance evaluation model using fuzzy variables. It provides a process for calculating a pedagogical quality measure supported by a consistency measure to judge pedagogical quality assurance of PGMPs. Section four provides the results of an in-class experiment utilizing both, the PGMP framework and its pedagogical evaluation model. Section five is a conclusion followed by an appendix.
2. Constructing a PGMP Learning Activity
We define a pedagogical game of mathematical proof expressed as
(1)
A pedagogical game of mathematical proof, hence, consists of a teaching model
defined by a set of pedagogies A, a set of rules G defining the game structure, and the mathematical proof
, consisting of a set of assumptions
, a true statement
or set of statements
, and the method of deduction
. Table 1 below summarizes the steps in the development of a PGMP as described in (1).
Implicit assumptions: Students have been introduced to basic abstract algebra and are familiar with the concepts of a group, Abelian group, and (non-)symmetry of mathematical objects such as those shown in Figure 1.
The theoretical model summarized in Table 1 requires educators to define the
sets
, and A. We use the notation
to indicate a set of mathematical statements
. For example
may represent a set of exercises, and
a specific statement
to be proven.
Step 1: Define
and
, and formulate
as in Equation (1). We define the premises
. That is
(2)
where
:
is an equilateral triangle.
:
is a box with a whole
in it such that a block
fits in
. See Figure 1.
: There are two transformations called rotation
and flip
.
: There is a composition
of transformations
and
acting on the block
.
Next, let there be a set of statements to be proven, where each statement
corresponds to an exercise.
(3)
where
:
fits in
in 6 different ways.
:
and
form a group
.
:
and
form subgroups
of
, for some
.
:
are Abelian.
:
and
form subgroups
of
, for some
.
:
are Abelian.
:
is not Abelian.
Step 2: Choose G over
. This requires to define the set of rules of the mathematical game.
(4)
where
:
is a right (or left) rotation of 120 degrees with fixed center in
.
:
is a right or left flip of 180 degrees with fixed center in
(imagine a line from any vortex through the center of
).
:
is a composition of transformations acting on
.
The definition of the set G must be in accordance with the guidelines set out in the pedagogy A. In our case, the elements of G are supposed to support a constructivist learning approach, where students actively co-create knowledge by physically rotating and flipping a block
to match a hole of same shape
in a box
to derive a table of all possible successful combinations of flips and rotations of
from which proofs are derived.
Step 3: We now define the pedagogy of our game of mathematical proof. With an increasing awareness that many undergraduates are passive during seminar sessions [33] [34], we define a learning activity that allow students to actively engage in co-creating knowledge. Each student is provided a wooden block (an equilateral triangle)
and a wooden box
with a hole of shape
in it. The student is then asked to use the building block to physically rotate and flip it to fit the hole and to therewith derive symmetries and present observations in form of a table from which proofs of propositions are then derived. We defined the set A which includes the following conditions.
1) Students are provided with a wooden box and a symmetric wooden object as shown in Figure 1.
2) Students are provided with a problem set which contains a set of instructions for each problem (Appendix).
3) Students work independently on the problem set (single player game).
4) Students are provided with 50 minutes of time to complete a set of tasks (Theorems 1 and 2 in Appendix).
5) Students can ask the instructor for definitions (loosing 5 points each).
6) Students compete with each other on time (loosing 5 points each rank) and points (each task is worth some points).
7) A total point score is calculated and appropriate reductions are made. The winner is announced to the class.
The pedagogical model
consists of a mixed pedagogy including elements from active learning and competitive learning both based within the constructivist paradigm with students co-creating their own knowledge playfully.
3. Pedagogy Quality Assurance Evaluation Method
In order to assess the usefulness of a PGMP as a learning activity we develop a pedagogy quality assurance evaluation method (PEM). We construct fuzzy variables [35] combined with a hierarchical model [36] to obtain a numerical value expressing pedagogical quality [37] [38] [39]. In our PEM, quality is defined as a weighted average of learning effectiveness, learning efficiency, and learning satisfaction as shown in Figure 2.
Let there be n evaluation factors represented by a set
(5)
where
is the ith evaluation factor in
and
a vector. There are m levels of appraisal grades represented by a set
(6)
where
is the kth appraisal grade in
on a Likert scale and
is a vector.
represents “strongly disagree” and gradually increasing to
representing “strongly agree”. A mapping
is a fuzzy evaluation mapping if for each evaluation factor
there is a mapping
(7)
where
is a fuzzy set associated with evaluation factor
. Alternatively, for every
the mapping
yields a row vector
, where
represents the fuzzy membership degree of appraisal factor i to grade k. The general fuzzy appraisal matrix
for all evaluation factors
and appraisal grades
is denoted by
(8)
For simplicity, we employ a triangular distribution function
in the construction of the mapping
characterizing the fuzzy measure values
for
and
[37] [38]. In order to obtain a comprehensive
Figure 2. Pedagogy quality assurance evaluation method.
overall quality assurance evaluation vector
we construct a weight vector
by the AHP method [36] assigning relative weight
to evaluation factor
for every
. The overall quality assurance evaluation vector Q is obtained by
(9)
where
(10)
Figure 2 outlines the overall methodology of the quality assurance evaluation process. Pedagogical quality is constructed as a measure of learning performance, defined by the degree of which students can successfully perform tasks within given time constraints, and learning satisfaction. A Post-Study System Usability Questionnaire (PSSUQ) ( [40], p.75-77) with respect to all three dimensions is employed to obtain survey data. The goal of this quality assurance system is to provide educators with a measure for the overall quality Q of a pedagogy.
We begin by constructing the fuzzy sets
from student survey (PSSUQ) and expert survey data. The later can be obtained via qualitative interview with subject experts. For simplicity, we assume a triangular distribution function
to construct the mapping functions characterizing fuzzy measure values
. We conduct a focus group with education experts to obtain the data points
for each evaluation factor
in the product quality evaluation factor set U and appraisal intervals
. Then for all
(11)
where each
is some value provided by the student survey data (PSSUQ). From Equation (11) we use the values
for
to calculate the normalized appraisal vector
.
(12)
with appraisals grades
for evaluation factor i.
can then be normalized in the usual way yielding the values
.
Some factors
according to our product quality definition provided above contribute with different importance to the overall measure of pedagogical quality. We now discuss the construction of the vector W using the AHP method [41]. The AHP method requires experts to make pair-wise comparesons between evaluation factors
and assigning numerical values
for
to them. This yields a square matrix
(13)
of relative weights
with the following properties:
1)
, for
.
2)
, for all
.
3) If
for
, then
, for
.
4) If
is more important than
then
.
These properties yield a positive definite and reciprocal matrix with 1’s on the main diagonal. The experts only need to provide data for the upper triangle of the matrix, that is
data points. Property one above suggests a relation
(14)
One can use Saaty’s method to compute W [41]. The final step is to check that human judgments (expert data, A) are consistent. For that purpose, we calculate a consistency index (CI) measuring how a given matrix A compares to a purely random matrix in terms of their consistency indices. Let CI be the consistency index
(15)
where n is the size of the pair-wise comparison matrix A and
represents the maximum eigenvalue of the matrix. Let RI be the average random index which can be computed as shown in [41]. A matrix A is consistent if and only if
and that we always have
when matrix A is a positive reciprocal one. If CI is less than 0.1, the numerical judgments will be considered acceptable [41].
4. Results from an In-Class Experiment
An in-class experiment with 10 third year undergraduate economics students taking an advanced mathematics course was conducted at the University of Exeter. The student cohort attended two different one hour sessions plus 15 minutes introduction to the learning activity. In session one, they conducted the learning activity “Theorem 1” and after 7 days, session two covered learning activity “Theorem 2”3. The precises activities are discussed in section three and the appendix of this paper. Student performance was incentivized with a competitive game element, distributing points to correct answers and penalizing students with negative points who requested definitions from the tutor. A total score was determined and the best performer was announced to the cohort.
Students reported Subjective Task Time, measured on a Likert scale 1 - 10 with 1 long subjective time perception and 10 short subjective time perception. Both Subjective Task Completion and Subjective Task Time was reported in a USQ survey [40], with Task Completion determined by a tutor who marked the completed tasks to obtain an overall assessment score for each student. Together, Task Time and Task Completion determine a weighted measure of Learning Performance.
A focus group consisting of five mathematics education experts was conducted to determine the bounds of the metric function in Equation (11). As a rule, experts had to jointly agree about the metric bounds, which were determined in a verbal 30 minutes debate after being introduced to the specifics of the experiment and PGMP. Expert decisions were purely based on the description of the PGMP and they were not privy to student survey results. The same cohort of experts also provided numerical values (matrix (17)), based on a pairwise comparison of the variables “fun to play and usability”, “instruction quality”, and “game quality” of the PGMP to yield the matrix A in Equation (13).
Table 2 shows the individual average task success and self evaluated task time data for both sets of exercises resulting in the proofs of theorems 1 and 2. Associated with each theorem, an USQ survey collected data on task completion and task completion time for six tasks (see Appendix) [40]. On average task success was 8.308 (Learning Effectiveness) with associated average task time 8.6 (Learning Efficiency) for theorem 1. For theorem 2, average task success was 8.617 (Learning Effectiveness) with associated average task time of 8.057 (Learning Efficiency). average task success was higher for theorem 2, students completed the six tasks While associated with theorem 1 faster. From this table an average
Table 2. Individual task success and task time for theorems 1 and 2.
Learning Performance of 8.454 for theorem 1 compared to 8.337 for theorem 2 suggests that students performed better on proving theorem 1.
Table 3 reports individual subject data from the PSSUQ survey on three variables: average “fun to play and usability” (PGMP play) 4.250, average instruction quality (InstQual) 4.129, and average game quality (PGMPQual) 4.100. From this table, an overall average Learning Satisfaction value of 4.160 is obtained.
We now determine the parameters of the fuzzy evaluation mapping given in Equation (11). For simplicity, we assume a triangular metric. From a focus group consisting of five mathematics education experts we obtain the boundary values of each fuzzy variable reported in Table 4. The experts agree to apply the same metric boundaries for Task Success and Task Time, but propose different metric
Table 3. Individual pedagogy satisfaction measured by PGMP play, InstQual, and PGMPQual.
Table 4. Expert metric evaluation matrix.
boundaries for PGMP Satisfaction. Figure 3 graphically shows the suggested metric for PGMP Satisfaction and Figure 4 graphically represents the metric for Task Success and Task Time and Table 5.
TableA1 (appendix) and Table5 report the individual fuzzy values for Task Success for Theorems I and II. TableA2 (appendix) and Table6 report the individual fuzzy values for Task Time for Theorems I and II.
From Table 5, Table 6 and Table 3 we obtain the matrix
(16)
The following matrix A is constructed with input from experts:
(17)
Figure 4. Task success, and task time metric.
Using the average of normalized columns (ANC) method
(18)
where
are the entries of the matrix A we obtain a normalized matrix
(19)
Averaging
over rows yields a vector
defined by
(20)
Hence
. We calculate the overall evaluation vector
(21)
That is
. We defuzyfy Q to obtain an overall measure of quality
using
(22)
where
and
is the appraisal vector in [38] representing “very poor”, “poor”, “medium”, “good”, “excellent”.
We now calculate the consistency of our result. That is
(23)
(24)
(25)
5. Conclusions
This paper designs a pedagogical game of mathematical proof to help non-STEM mathematics educators to produce pedagogically effective active learning exercises within the constructivist paradigm using toys and without tedious calculations supporting anxious students in deriving mathematical proofs playfully. The purpose of this learning activity is to help students improve self-efficacy and reduce mathematics anxiety.
An in-class experiment shows that our pedagogy based on PGMPs satisfies a pedagogical quality criterion. We derive this conclusion from our Pedagogy Quality Assurance Evaluation Method, which we introduce using fuzzy variables. Our model uses Learning Effectiveness, Learning Efficiency, and Learning Satisfaction as inputs to derive a weighted measure of pedagogical quality. The advantage of this measure of learning activity evaluation is that it relies on data from both, students and experts.
The data from the experiment show that students were effective in playfully producing mathematical proofs using a hands-on approach using toys demanding them to physically rotate and flip geometric objects, and write up their observations in a table. The table, together with a set of sequential instructions effectively guided students through the reasoning required to produce a complete proof. The success was not only reflected in high performance attainment but also in the survey responses.
To improve on the robustness of our main result, future work should consider different metric functions such as trapezoids, bell-shaped, and sigmoid functions to specifically take into account other pedagogical variables. An empirical experiment with a large group of mathematics education experts would provide more accurate metrics.
Obtaining a more specific measure of our learning activity of reducing mathematics anxiety would be a valuable result for many educators. A robust measure could be obtained from a field-experiment with a much larger subject pool incorporating established instruments for measuring mathematics anxiety [23] [24] [33] into our model.
Acknowledgement
This work was supported by the Research-Inspired Inquiry-Led (RIIL) projects fund, as part of the Exeter Education Incubator, and by the NUBS School Research Fund.
Appendix
Definition 1. A symmetry is any transformation of
which leaves
in an equivalent state (−5 pts)
Theorem 1
: There are 6 symmetries.
Task 1. Label the edges of the triangle
with the capitals A, B, and C in counterclockwise direction. Similarly, label the edges of the corresponding hole
in the box
so that
matches
,
matches
, and
matches
. [20 pts]
Task 2. Check that if starting at position
matches
(AA) a rotation
moves to the new position
matches
(AB). We express this compactly as
. Check that if starting at position
matches
(AA) with fixed (AA) a flip
leaves
matches
unchanged, but moves
on
(BB) to
on
(BC) and
on
(CC) to
on
(CB). We use the notation
to express this property. [15 pts]
Task 3. Construct a table to show all symmetries of
and
only using rotations
and its compositions
. [15 pts]
Task 4. Fix the edge (AA) and use a flip
in isolation and in combination with rotations
to obtain the symmetries of
. Follow this procedure to obtain the symmetries associated with
and
and collect them in a table. [15 pts]
Task 5. Collect all symmetries from above tables to define the set
. [15pts]
Task 6. Use the results of tasks 1 - 5 in given order to formally write up the proof of theorem 1. [20 pts]
The expected result from task 6 is summarized in the proof bellow.
Proof. Label the vertices of
counterclockwise starting from the top by A, B and C. Similarly on
label the initial vertices of
counterclockwise starting from the top by A, B and C, such that the labeling for
matches the labeling on the
, i.e., AA, BB and CC.
We introduce two operations; a counterclockwise rotation around the center of
and a flip of
at a fixed vortex (A, B, orC). Let
denote the counterclockwise rotation of
from i.e. initial A to final B such that A on
moves from position A to position B on the
. This is compactly expressed as
. Let
be a flip with fixed vortex A such that initial B moves to final C and initial C moves to final B while A remains unchanged. We compactly express this as
.
The composition of two operations is denoted by
. Then by definition of symmetry we obtain:
These are the symmetries obtained from only applying rotation to
. Notice that there are two ways to achieve
; i) apply rotation three times, or ii) do not take any action
at all. Let’s generate the symmetries associated with flipping.
We collect all symmetries in a set
.
Definition 2. A Latin square is an
array filled with n different symbols, each occurring exactly once in each row and exactly once in each column. (−5 pts)
Definition 3. A group is a set
with a binary operation
(−2 pts each),
for all
satisfying:
1) Closure:
.
2) Associativity:
.
3) Identity element: There exists and element
such that
.
4) Inverse: For every
there exists some
such that
and
.
Definition 4. A Cayley table describes the structure of a finite group by arranging all the possible products of all the group’s elements in a square table (−5 pts).
Theorem 2.
:
, and
, form a group
.
1) Construct a table of symmetries (Cayley table) and show that it is a Latin square. [20 pts]
2) Using the definition of a group show the closure property (use the Cayley table). [15 pts]
3) Using the definition of a group demonstrate the associativity condition (use the Cayley table). [15 pts]
4) Using the definition of a group find the identity element (use the Cayley table). [15 pts]
5) Using the definition of a group show that each element has an identity (use the Caleyly table). [15 pts]
6) Write up a formal proof using the results of tasks 2 - 5. [20 pts]
Proof. To show the closure property we construct a Cayley table for
. From TableA3 one can also see that it defines a Latin square. The closure property follows from the Latin square property. Associativity follows from the fact that the composition of bijections from a set to itself is associative. By inspection of the Cayley table we have:
Table A2. Task time evaluation theorem I.
Table A3. Cayley table for the six symmetries of
.
It follows that
is the identity element since for any
. Each of our symmetries
has an inverse
such that
. We have
, with inverse
since
, with inverse
since
, with inverse
since
, with inverse
since
, with inverse
since
, with inverse
since
. ¨
NOTES
1https://www.cne.psychol.cam.ac.uk/what-is-mathematics-anxiety.
2https://www.ams.org/publications/journals/notices/201702/rnoti-p124.pdf.
3The learning activities correspond to
and
of section three. The other statements can be used as further active learning activities within the PGMP model.