An Online Evaluation of Problem-Based Learning (PBL) in Chung Shan Medical University, Taiwan - A Pilot Study

624
Online Evaluation of PBL in CSMUJia-Yuh Chen et al
Original Article
An Online Evaluation of Problem-based Learning (PBL) in Chung Shan Medical

University, Taiwan A Pilot Study
Jia-Yuh Chen,6MD, PhD, Meng-Chih Lee,2MD, PhD, Hong-Shan Lee,3PhD, Yeou-Chih Wang,4MD,
Long-Yau Lin,5MD, PhD, Jen-Hung Yang,1MD, PhD,
Abstract
Introduction: Problem-based learning (PBL) embraces principles of good learning and
teaching. It is student-directed, fosters intrinsic motivation, promotes active learning, encourages peer teaching, involves timely feedback, and can support student self and peer assessment.
The most important function of the assessment process is to enhance student learning, to improve
the curriculum and to improve teaching. Materials and Methods: To improve the PBL tutorial
in Chung Shan Medical University, we developed an online evaluation system containing the
evaluation forms for students, tutor, self and peer. The Cronbach reliability coefficients were
0.9480, 0.9103, and 0.9198 for the Student, Tutor and Self and Peer Evaluation Form, respectively.
The online evaluations were mandatory to both students and tutors, and the information was
completely anonymous. Results and Conclusions: The survey response rates of the online
evaluations ranged from 95.6% to 100%. The online evaluations provided a documented
feedback to the students on their knowledge, skills and attitudes. Correspondingly, tutors too
received feedback from students in evaluating their performance on the appropriateness and
effectiveness of tutoring the group. Although there was an initial lack of coordination regarding
responsibilities and how to use the online system for both students and the Faculty, the system
enabled us to look into how effective our PBL course had been, and it provided both process and
outcome evaluations. Our strategy for evaluating the success of PBL is only at its initial stage;
we are in an ongoing process of collecting outcome data for further analysis which will hopefully
provide more constructive information to the PBL curricula.
Ann Acad Med Singapore 2006;35:624-33
Key words: Facilitator, Formative evaluation, PBL tutors, Small group
Introduction
The goal of problem-based learning (PBL) is to motivate
students to develop self-learning skills in a small group.
PBL embraces principles of good learning and teaching. It
is student-centred, student-directed, fosters intrinsic
motivation, promotes active learning, encourages peer
teaching, involves timely feedback, and can support student
self and peer assessment.1-4 The students are encouraged to
use self-directed learning skills to analyse a given clinical
scenario, to identify the problems, to formulate key-learning
objectives, and to collect whatever additional information
they think will be needed to address those objectives. More
importantly, all this takes place within a group setting, so
that each individual member of the group contributes to the

learning process at every stage. At the end of each tutorial,
time is set aside for the tutors to provide feedback to
students and vice versa to improve the performance of both
students and tutors. From the PBL tutorial, students learn
to develop attitudes to facilitate cooperative PBL team
work, to develop communication and group skills, to
encourage peer teaching, to involve timely feedback, and
to support self and peer assessment on an ongoing basis.5
In Asia, PBL curricula may not be easily adopted for 2
reasons: (1) Class sizes range between 100 and 250 students,
requiring a large number of tutors. (2) The students are
generally younger than the US cohort and have less
Department of Dermatology
Department of Family Medicine
3
School of Public Health
4
Department of Surgery
5
Department of Obstetrics and Gynaecology
6
Department of Paediatrics
Chung Shan Medical University and Hospital, Taichung, Taiwan
Address for Reprints: Jen-Hung Yang, Department of Dermatology, Chung Shan Medical University and Hospital; 110, Sec. 1, Chien-Kuo N. Road, Taichung,
Taiwan 402, ROC.
Email: [email protected]
2
Annals Academy of Medicine
experience and maturity for self-directed learning. Strong

support from the academic administrators (dean and other
staff responsible for implementation of the curriculum) in
the introduction of PBL into the curriculum and careful
training of both Faculty and students appear to be key
factors to ensure the successful implementation of PBL in
Asian medical schools.6 The PBL curricula are now widely
distributed in a large proportion of Asian medical schools,
although most of them are undergoing a hybrid PBL
curriculum (hybrid = traditional lectures + PBL).
PBL in Chung Shan Medical University (CSMU)
PBL was introduced to CSMU in 2002. A hybrid
curriculum which composed of approximately 20% PBL
and 80% traditional lectures (dominant teaching method)
was adopted. Implementation of PBL focused on clinical
medicine at Year 5 class (M5, preclerk), then distributed to
other organ-system based modules at Year 2, 3, 4 classes
(M2, M3 and M4). These days, PBL programmes have
been applied in medical ethics and basic science, and
courses at Nursing School in CSMU.
PBL tutorial process was modified from the Maastricht
seven jump process as shown in Table 1.3 A typical PBL
tutorial consisted of a group of 8 students and a tutor, who
facilitated the session. All tutors have been trained and
certified to implement the PBL tutorial in M5 class. Students
were divided into 18 small groups, 7 to 8 students in each
group, 2 to 3 sessions to each tutorial and 4 to 6 PBL tutorial
cases were scheduled in a semester.
The aims of the online evaluation system were (1) to
provide feedback to students, tutors and administrations in
improving the learning in PBL tutorial, and (2) to provide
an academic research opportunity to study the effectiveness
of the PBL curriculum in CSMU. Within each tutorial
group, the tutor was expected to assess the students in the
tutorial process, and, correspondingly, the students provided
feedback to each other and to the tutor on a regular basis.
The online evaluations were not intended or used to replace
the immediate feedback during the PBL tutorial.
One of the most consistent findings in evaluations of
PBL is a more humane learning environment that promotes
collegial interactions. Albanese7 concluded that this was
a worthwhile goal in and of itself. Feedback is an essential
component of learning in PBL tutorial. Through feedback,
individuals recognise areas of deficiency in knowledge or
skills and seek to remedy these.8
Materials and Methods
Design of the Evaluation Forms
We designed evaluation forms for students, tutors, and
student self and peer assessment (Appendixes 1 to 3). The
questionnaire items (phrased in Chinese) were designed
September 2006, Vol. 35 No. 9
625
after literature review and interviews with expert Medical

Educators in National Taiwan University, National YangMing University, Taipei Medical University, Fu-Jen
University and CSMU. The Student Evaluation Form has
15 items representing 4 categories: (1) critical appraisal (2)
utilisation of learning resources (3) group work, and (4)
attitudes and communication skills (Appendix 1). The
Tutor Evaluation Form has 6 items, and was evaluated by
the students to assess the tutors performance in the PBL
tutorial (Appendix 2). The Student Self and Peer Evaluation
Form has 11 items (4 items for self evaluation and 7 items
for peer evaluation), and was also evaluated by the students
to provide peer interactions during the PBL small group
tutorial (Appendix 3). Each form also has space for
comments.
Validity and Reliability of the Evaluation Forms
Content experts from National Taiwan University,
National Yang-Ming University, Kaohsiung Medical
University and CSMU reviewed the validity of the evaluation
forms. For the study, the Evaluation Forms used the Likert
5-point scale. Internal consistency and reproducibility were
examined by Cronbach analysis (SPSS 13).9
Participants
The evaluations were mandatory for both students and
tutors. All 136 students in M5 class (preclerk) during the
2005 academic year, and all 47 tutors were enrolled into the
study. The tutors were all clinicians trained and certified to
implement the PBL tutorial. The students were divided into
18 small groups, and the average number of students in
each group was 8. Among the 18 groups, 5 of them had 2
or 3 tutors, and the remaining 13 were facilitated by a single
tutor for each group throughout the semester. All participants
were asked to complete the online evaluation forms within
a week after the end of each case.
Online Evaluations
Each participant (students and tutors) completed the
evaluation forms online at the end of each case. Information
was completely anonymous and individuals comments
cannot be traced back to individual students in the website.
These evaluations can be seen and checked any time by the
students and the tutors of the same group to fulfill the
requirement for easy accessibility and feasibility. The web
site address for PBL in CSMU is http://www.csmu.edu.tw.
Results
Evaluation forms were completed by students and tutors
after each completed case and forwarded to the PBL
coordinator. The results of the evaluations were available
to each tutorial group including the students and the tutors.
The survey response rates ranged from 95.6% to 100%.
626
Validity and Reliability of the Evaluation Forms

The Cronbach reliability coefficients were 0.9480,
0.9103, and 0.9198 for the Student, Tutor and Self and Peer
Evaluation Form, respectively (Table 2). Cronbach
reliability coefficient >0.7 indicated that the evaluation
forms were reliable for internal consistency.9
Evaluation on Student Performance
At the completion of each tutorial case, each student was
awarded a Satisfactory (agree) or Unsatisfactory (disagree)
grade on the domains outlined in the PBL Student Evaluation
Form. The scores 5 and 4 indicated very satisfactory and
satisfactory, score 3 indicated no preference, and the
scores 1 and 2 indicated very unsatisfactory and
unsatisfactory, respectively. An item rated as satisfactory
was defined as more than 80% of the respondents scored
4. Student performance was assessed primarily through
participation in the PBL small group tutorial. There were
15 items in the form representing 4 subscores, critical
appraisal utilisation of learning resources, group work, and
attitudes/communication skills. In general, the tutors were
satisfied with the students performance, 13 of the 15 items
in the Student Evaluation Form were ranked high scores (4
and 5). However, the scores of 2 items (7 and 14) were
below 4, which indicated that the students were not skillful
in utilising the internet or other resources to gather
information; in addition, they were also unfamiliar with the
Table 1. PBL Tutorial Process
1.
Elect chair and scribe, and presentation of the scenario
2.
Clarify terms and unknowns
3.
Discuss what is known/preliminary ideas (Brainstorming)
4.
Determine objectives/learning issues
5.
Self-directed study
6.
Share new information/knowledge and (Re) formulate
7.
Reflect, feedback and evalution
The PBL tutorial process was modified from the Maastricht seven jump
process
Table 2. Reliability of the Evaluation Forms
Forms
Cronbach value
Student Evaluation Form
0.9480 (15 items)
Critical appraisal
0.9098 (5 items)
Utilisation of learning resources
0.8250 (3 items)
To share with peer
0.7870 (3 items)
Humanistic attitudes and skills
0.8088 (4 items)
Tutor Evaluation Form
0.9103 (6 items)
Knowledge, attitude and skills

Self and Peer Evaluation Form
0.9198 (11 items)
Internal consistency and reproducibility by Cronbach analysis (SPSS 13)
use of appropriate presentation tools (e.g., PowerPoint)

(Figs. 1a, 1b). However, the skills of the students in these
2 aspects revealed a tendency to improve in the following
period (Fig. 1c).
Evaluation of Tutor Performance
At the completion of each case, the tutor was assessed by
the same tutorial group students. The average ratings for
tutor performance were high (4 and 5) in all 6 items of the
form, and the satisfactory rate was over 90% (Figs. 2a, 2b).
Students highly appreciate our faculty performance
(including knowledge, skill and attitude) in the PBL tutorial.
The results showed that the students were highly satisfied
with the tutors facilitation during the small group tutorial
regardless of whether they were expert or non-expert
tutors.
Student Self and Peer Evaluation
For self-evaluation, 3 of the 4 items were ranked as
satisfactory, and for peer evaluation, 6 of 7 items were
ranked as satisfactory (Figs. 3a, 3b). The students were not
very satisfied with their overall performance (item 8) and
with regard to how systematic and well-organised the
discussion was conducted (item 4). However, students felt
they had improved and revealed self-confidence in the
following period (Fig. 3c). In general, the students showed
an attitude of autonomy, self-criticism, and an increase of
self-confidence from case 1 to case 4 tutorials (Fig. 3c).
They were earnest and highly cooperative in the evaluations.
Moreover, the results showed that students acknowledged
that PBL was a useful tool for developing strong working
alliances in PBL group dynamics, and had a strong sense of
identification and confidence in their ability to perform
their own task.
Discussion
To improve the PBL tutorial, we developed online
evaluation forms to access students, tutor, self and peer.
The evaluations were done after the closing of each case
session (formative assessment) and the online evaluations
were mandatory for both students and tutors. The online
evaluation provided a documented feedback to the students
on their knowledge, skills (e.g., use of resources, problem
analysis and solving, group work skills) and attitudes (e.g.,
independence, respect for the contribution of others,
leadership, and self-reflection). In addition, tutors received
feedback from students on their performance. The online
evaluation system for PBL enables us to see how effective
our course has been. The efficacy and reliability of the
evaluation forms are useful in providing both process and
outcome evaluations (Cronbachs >9).
In general, tutors are satisfied with the students
performance, 13 of the 15 items in the Student Evaluation
Form were ranked high scores (4 and 5). However, the

scores of two items (7 and 14) were below 4, which indicate
that the students are not familiar with utilisation of the
learning resources and the skills in presentation. Apparently,
to enhance e-learning skills and to learn how to appraise the
information are very important for students. However, we
found that there was a parallel drop in the scores for items
7 and 14 for Case Number 3, perhaps this might be due to
the time of PBL tutorial conducted near examination
period. Unexpectedly, a small proportion of students could
not use presentation tools (e.g., PowerPoint) effectively,
and the reason might be due to the lack of enthusiasm in
participating in the small group discussion. In comparison
to the assessment by students themselves and the peers, the
students showed an attitude of autonomy, self-criticism,
and an attitude of self-confidence. They were earnest and
highly cooperative in the evaluations.
To the contrary, students were highly satisfied with the
tutors performance regardless of whether they were subjectmatter experts during the PBL tutorial or not. This may be
attributed to the following two factors: Firstly, all of the
tutors were trained and experienced. Secondly, the
relationship between the students and the tutor in a small
group was mutually interactive. Our results suggest that the
ideal PBL tutor would function as a group facilitator rather
than a subject-matter expert. In other words, the role of a
tutor is to facilitate student learning rather than to convey
knowledge.10 It is widely accepted that subject expertise is
not an essential requirement for a PBL tutor, the use of
expert or inexpert tutors does not appear to affect learning.2
Caplow et al11 reported that, to optimise learning in the
group context, students preferred tutors who are minimally
challenging and non-judgemental in their roles as knowledge
experts, not directive in their roles as facilitators, and not
directive in their roles as coaches or models of clinical
reasoning.
A central organising principle of PBL is the use of small
tutorial groups. This instructional format is comparable to
the cooperative learning group method in which learners
cooperate in framing problems, gathering resources and
elaborating knowledge through discussion, teaching peers,
evaluating information and sources, and answering
questions.2 The students acknowledged that PBL is a
powerful tool for developing strong working alliances
early in the group dynamics process. For work to progress
efficiently, the group has a strong sense of identification
and confidence in their ability to perform their task.
It is now recognised that acquiring skills in self-evaluation
is a prerequisite for the continual learning of every doctor.
The online evaluation system provides an opportunity to
develop the skills and helps in facilitating appraisal of
performance.5 In addition, the system also provides a
627
narrative space for participants to express their comments

or suggestions, which are invaluable to improve our PBL
curricula.
However, there were barriers in the implementation of
the online evaluations in CSMU. There was an initial lack
of coordination regarding responsibilities and how to use
the online system for both students and the Faculty. After
some modifications during the period, our system is
becoming more accessible and feasible to the students and
tutors. However, there were still a few students who failed
to complete the evaluation forms in a timely manner, which
may indicate a reflection of the students professionalism.
Conclusion
In summary, our online evaluation system for PBL
curriculum is valid and reliable. Although there was an
initial lack of coordination regarding responsibilities and
how to use the online system for both students and the
Faculty, the formative online evaluation of PBL enabled us
to see how effective our courses had been, and provided
both process and outcome evaluations. Our strategy for
evaluating the success of PBL represents only an initial
stage; we are in an ongoing process of collecting outcome
data for further analysis which will hopefully provide more
constructive information to the PBL curricula.
Acknowledgements
The authors would like to thank Dr Hao-Chang Lo, Ms Yi-Fang Liang and
Meng-Chia Kuo for their assistance in implementing the online evaluations
in Chung Shan Medical University, Taiwan.
REFERENCES
1. Barrows HS. A taxonomy of problem-based learning methods. Med
Educ 1986;20:481-6.
2. Albanese MA, Mitchell S. Problem-based learning: a review of literature
on its outcomes and implementation issues. Acad Med 1993;68:52-81.
Erratum in: Acad Med 1993;68:615.
3. Wood DF. Problem based learning. BMJ 2003;326:328-30.
4. Kilroy DA. Problem based learning. Emerg Med J 2004;21:411-3.
5. Das M, Mpofu D, Dunn E, Lanphear JH. Self and tutor evaluations in
problem-based learning tutorials: is there a relationship? Med Educ
1998;32:411-8.
6. Khoo HE. Implementation of problem-based learning in Asian medical
schools and students perceptions of their experience. Med Educ
2003;37:401-9.
7. Albanese M. Problem-based learning: why curricula are likely to show
little effect on knowledge and clinical skills. Med Educ 2000;34:729-38.
Erratum in: Med Educ 2001;35:419.
8. Parikh A, McReelis K, Hodges B. Student feedback in problem based
learning: a survey of 103 final year students across five Ontario medical
schools. Med Educ 2001;35:632-6.
9. Cronbach LJ. Coefficient alpha and the internal structure of tests.
Psychometrika 1951;16:297-333.
10. Barrows HS. The Tutorial Process. 2nd edn. Springfield, Illinois: Southern
Illinois University School of Medicine, 1998.
11. Caplow JA, Donaldson JF, Kardash C, Hosokawa M. Learning in a
problem-based medical curriculum: students conceptions. Med Educ
1997;31:440-7.
628
Case 1
Fig. 1a.
Case 1
Case 3
80%
Case 2
80%
Case 4
80%
80%
Fig. 1b.
Fig. 1. There are 15 items in the Student Evaluation Form representing 4 subscores: critical appraisal, utilisation of learning resources, group work, and attitudes/
communication skills. The percentages of average scores for each item in cases 1 to 4 are shown in Figures 1a and 1b. The scores of 2 items (7 and 14) were
below 4 (Fig. 1a). The skills of the students in these 2 items revealed a tendency for improvement in a following period (Fig. 1c).
629
95
Satisfactory rate (%)
90
Item 7
85
80
75
Item 14
70
65
0
Fig. 1c.
Case Number
Case 1
Case 1
Case 3
Case 2
80%
80%
Case 4
Fig. 2a.
80%
Fig. 2b.
80%
Fig. 2. There are 6 items in the

Tutor Evaluation Form. The
percentages of average scores for
each item in cases 1 to 4 are shown
in Figures 2a and 2b. The satisfactory rate was approximately
90% in all 6 items (Figs. 2a and
2b).
630
Case 1
Case 1
Fig. 3a.
Case 3
80%
80%
Case 2
Case 4
80%
Fig. 3b.
80%
88
Satisfactory rate (%)
86
84
82
Item 4
80
Item 8
78
76
0
Case number
Fig. 3c.
Fig. 3. There are 11 items in the Student Self and Peer Evaluation Form. Items
1 to 7 are for peer evaluation, and items 8 to 11 are for self evaluation. The
percentages of average scores for each item in cases 1 to 4 are shown in
Figures 3a and 3b. The students were not very satisfied with their overall
performance in a systematic and well-organised way (items 4 and 8).
However, students felt they had improved and showed self-confidence in the
following period (Fig. 3c).
631
Appendix 1. Problem-based Learning (PBL) Student Evaluation Form

(Tutor-to-Student)
1 = strongly agree. 2 = agree. 3 = no preference. 4 = disagree. 5 = strongly disagree.
I. Critical appraisal
1
2
3
4
5
Clarifies, defines and analyses the problem from the scenario.

strongly agree
1
2
3
4
Student is capable of proposing hypotheses and issues.
strongly agree
1
2
3
4
Identifies learning objectives.
strongly agree
1
2
3
4
Demonstrates viewpoints of initiative and curiosity.
strongly agree
1
2
3
4
Demonstrates constructive thinking process.
strongly agree
1
2
3
4
strongly disagree
strongly disagree
strongly disagree
strongly disagree
strongly disagree
II. Utilisation of learning resources

6
7
8
Utilises relevant resource materials effectively.

strongly agree
1
2
3
4
5
strongly disagree
Utilises internet or evidence-based medicine to get appropriate information.
strongly agree
1
2
3
4
5
strongly disagree
Applies knowledge to new situations to solve problems and to reach decisions.
strongly agree
1
2
3
4
5
strongly disagree
III. Group work

9
Organised and prepared for small group sessions.

strongly agree
1
2
3
4
10 To share thoughts and opinions with peer actively.
strongly agree
1
2
3
4
11 To share all sources for picture, text and other information.
strongly agree
1
2
3
4
strongly disagree
strongly disagree
strongly disagree
IV. Attitudes and Communication Skills

12 The oral expression is clear enough to be understood.
strongly agree
1
2
3
4
5
strongly disagree
13 To provide and accept constructive feedback.
strongly agree
1
2
3
4
5
strongly disagree
14 To use presentation tools (eg. Power-Point) effectively.
strongly agree
1
2
3
4
5
strongly disagree
15 Contributes to group harmony (listens to conflicting opinions and tolerates shortcomings of others).
strongly agree
1
2
3
4
5
strongly disagree
Comment:
To describe the strengths and weaknesses of the reviewed student and to assist him/her to be a more effective learner.
632
Appendix 2. Problem-based Learning (PBL) Tutor Evaluation Sheet

(Student-to-Tutor)
No.
Item
Understands the objectives of the PBL process, and assists group focus on learning issues and
objectives.
strongly agree
1
2
3
4
5
strongly disagree
Provides timely feedback and completes evaluations.

strongly agree
1
2
3
3
4
strongly disagree
Encourages critical thinking and a thorough look at available material.

strongly agree
1
2
3
4
5
strongly disagree
Attends session as planned, and control of session for discussion is adequate.

strongly agree
1
2
3
4
5
strongly disagree
Guides the PBL course adequately (asks challenging questions; avoids mini-lecturing).
strongly agree
1
2
3
4
5
strongly disagree
Shows enthusiasm as a tutor, and assists in creation of a comfortable learning climate.

strongly agree
1
2
3
4
5
strongly disagree
Comments: Please provide any other suggestions or comments about the tutor.
1. What are the tutors main strengths? What are the tutors main shortcomings?
2. What do you expect the tutor to do to improve in the next tutorial session?
Appendix 3: Problem-based Learning (PBL) Evaluation Sheet

(Student Self and Peer Evaluation)
No.
1
2
3
4
5
6
7
Item
Group participation is good.
strongly agree
strongly disagree
Peer interaction is good.

strongly agree
strongly disagree
strongly disagree
The proceeding in the tutoring session is well-managed.

strongly agree
1
2
3
The content of discussion is systematic, well-organised and substantial.

strongly agree
1
2
3
4
5
strongly disagree
The group members all strive to search and collect information.

strongly agree
1
2
3
4
strongly disagree
Students are keen on PBL.

strongly agree
strongly disagree
Most of the group members acquire knowledge to meet learning objectives.

strongly agree
1
2
3
4
5
strongly disagree
In the tutoring session, my performance is well in general.

strongly agree
1
2
3
4
strongly disagree
The tutoring session facilitates the interaction among classmates.

strongly agree
1
2
3
4
strongly disagree
10 The tutoring session greatly increases my own knowledge.

strongly agree
1
2
3
4
strongly disagree
11 This tutoring session greatly impacts my way of learning.

strongly agree
1
2
3
strongly disagree
8
9
Please provide some suggestions or comments for the group members:

1. What else should be improved?
2. As compared with the previous facilitated session, what has and has not improved?
633

An Online Evaluation of Problem-Based Learning (PBL) in Chung Shan Medical University, Taiwan - A Pilot Study

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

An Online Evaluation of Problem-Based Learning (PBL) in Chung Shan Medical University, Taiwan - A Pilot Study

Uploaded by

Copyright:

Available Formats

624

Online Evaluation of PBL in CSMUJia-Yuh Chen et al