Higher order

rd10
thinking skills
By Margaret Forster
Margaret is the Research Director
of ACER’s Assessment and Reporting
research program.

In today’s world it is necessary, but not

Higher order literacy and numeracy skills are now adopted as a
sufficient, for students to achieve minimal
starting point in most international assessment programs.
competence in areas such as reading, writing
International Adult Literacy Survey
and numeracy. Beyond the achievement of
Reading literacy is no longer defined merely in terms of a basic
minimal competence, students also need to
threshold of reading ability which everyone growing up in developed
develop what are often called ‘higher order’ countries is expected to attain. Rather, literacy is now equated with
an individual’s ability to use written information to function in society.
thinking skills including critical literacy, critical
(Kirsch, 2001, p.4)

numeracy and cross-curricular competencies.

Programme for International Student Assessment
A useful conceptualisation of higher order Reading literacy is an individual’s capacity to understand, use and
thinking skills distinguishes two contexts in reflect on written texts, in order to achieve one’s goals, to develop
one’s knowledge and potential, and to participate in society.
which these skills are employed: contexts (OECD, 2003, p.15)

where the thought processes are needed to

International Life Skills Survey
solve problems and make decisions in everyday
Numeracy involves abilities that include interpreting, applying and
life; and contexts where mental processes are communicating mathematical information in commonly encountered
situations to enable full, critical and effective participation in a wide
needed to benefit from instruction, including range of life roles.
comparing, evaluating, justifying and making (ETS, 2004, p.14)

inferences (Wheeler & Haertel, 1993).The Programme for International Student Assessment

ability to employ higher order thinking skills in Mathematical literacy is an individual’s capacity to identify and
understand the role that mathematics plays in the world, to make
both these contexts is seen as essential in a
well-founded judgements and to use and engage in mathematics in
rapidly changing world and the first context in ways that meet the needs of that individual’s life as a constructive,
concerned, and reflective citizen.
particular is being adopted as a starting point
(OECD, 2003, p.15)

for international assessment programs.

rd11
In Australia, school systems are increasingly olds in 46 countries2. Here the approach to information, forming a broad general
focusing attention on higher order thinking assessing the higher order thinking skills is understanding of the text, interpreting it,
skills. For example, the curriculum in based on a dynamic model of lifelong reflecting on the content and form of texts
Tasmania is organised around five cross learning in which new knowledge and skills in relation to their own knowledge of the
discipline constructs or ‘essential learnings’. necessary for successful adaptation to a world, and arguing their own point of view.
The ‘learnings’ are intended to culminate in changing world are continuously acquired
In the case of mathematical literacy, tasks
the lifelong outcomes of inquiring and throughout life. Rather than assessing
address three dimensions. First, the content
reflective thinkers, effective communicators, ‘school’ knowledge, PISA aims to measure
of mathematics, as defined mainly in terms
self-directed and ethical people, responsible how well students perform beyond the
of broad mathematical concepts underlying
citizens, and world contributors. They are school curriculum.
mathematical thinking (such as chance,
also intended to be future-oriented and
Although the domains assessed in PISA so change and growth, space and shape,
connected to the real world, to focus on
far (reading literacy, mathematical literacy reasoning, uncertainty and dependency
depth rather than breadth, to focus on
and scientific literacy) are closely related relationships), and only secondarily in
connections and transfer of learning, and to
to subjects learned at school, PISA relation to ‘curricular strands’ (such as
reflect and embody values and purposes’.
concentrates on the value of the skills number, algebra and geometry).
In the Tasmanian framework, higher order
acquired, through applying them in real-life
thinking is situated in the broader context Second, the process of mathematics as
situations3. The definitions of the assessment
of higher order learning with the five defined by general mathematical
domains make this clear.
‘essential learnings’ listed as: communicating, competencies. These include the use of
personal futures, social responsibility, world Reading literacy is defined as ‘an individual’s mathematical language, modeling and
futures, and thinking. capacity to understand, use and reflect on problem-solving skills. The idea is not,
written texts, in order to achieve one’s goals, however, to separate out such skills in
It is easy to articulate a commitment to the
to develop one’s knowledge and potential, different test items, since it is assumed that a
development of higher order thinking skills
and to participate in society’. Mathematical range of competencies will be needed to
but more demanding to translate this
literacy is defined as ‘an individual’s capacity perform any given mathematical task. Rather,
commitment into action. How do we
to identify and understand the role that questions are organised according to the
develop a curriculum that both addresses
mathematics plays in the world, to make type of thinking skill needed (simple
subject knowledge and emphasises higher
well-founded judgements and to use and computations or definitions; connections to
order thinking? And how do we monitor
engage with mathematics in ways that meet be made to solve straightforward problems;
and assess the development of students’
the needs of that individual’s life as a mathematical thinking, generalisation
higher order thinking ability? In the case of
constructive, concerned and reflective and insight).
assessment, the first challenge is to define
citizen’; and scientific literacy is defined as
exactly what it is we wish to focus on (how Third, the situations in which mathematics is
‘the capacity to use scientific knowledge to
we define higher order thinking), the second used, ranging from private contexts to those
identify questions and to draw evidence-
is to select an appropriate method for relating to wider scientific and public issues.
based conclusions in order to understand
gathering evidence of students’ skills (what In the case of scientific literacy, tasks address
and to help make decisions about the
form of assessment we use). three dimensions also. First, scientific
natural world and the changes made to it
Two areas of work undertaken at ACER through human activity’ (OECD, 2003, p15). concepts, which are needed to understand
provide two different approaches to the certain phenomena of the natural world and
In the case of ‘reading literacy’ students
assessment of higher order thinking skills the changes made to it through human
perform a variety of tasks with different
through paper and pen tests: tests of activity. Second, scientific processes, focusing
kinds of text selected to reflect the range of
performance ‘beyond the school curriculum’, on the ability to acquire, interpret and act
written forms that individuals will encounter
and tests of ‘general academic abilities’. upon evidence. Third, scientific situations
in adult life. Written forms include both
These approaches reflect two different selected mainly from people’s everyday lives
continuous prose and ‘non continuous’ text
contexts in which higher order thinking skills rather than from the practice of science in
such as lists, forms, graphs, and diagrams.
are employed. a school classroom or laboratory, or the
The range of texts also samples texts
work of professional scientists. As with
constructed for different purposes such as
Performance beyond the school mathematics, science figures in people’s lives
curriculum a novel, personal letter or biography written
in contexts ranging from personal or private
for people’s ‘private’ use; official documents
Tests of performance ‘beyond the school situations to wider public, sometimes
or announcements for ‘public’ use; a manual
curriculum’ assess the thought processes global issues.
or report for ‘occupational’ use; and a
that are needed to solve problems and textbook or worksheet for ‘educational’ use. Examples of a PISA scientific literacy task
make decisions in everyday life. At ACER and a PISA mathematics literacy task are
these skills are addressed in the Programme Students are not assessed on the most basic
provided on the facing page.
for International Student Assessment (PISA), reading skills. Rather, they are expected to

an international study of the skills of 15-year- demonstrate their proficiency in retrieving

rd12
An example of a PISA Mathematics Unit with one of the unit questions4

For health reasons people should limit their efforts, for instance during
sports, in order not to exceed a certain heartbeat frequency.

For years the relationship between a person’s recommended maximum heart

rate and the person’s age was described by the following formula:
Recommended maximum heart rate = 220 – age

Recent research showed that this formula should be modified slightly.

The new formula is as follows:
Recommended maximum heart rate = 208 – (0.7 x age)

Example question

A newspaper article stated ‘A result of using the new formula instead of the old
one is that the recommended maximum number of heartbeats per minute for
young people decreases slightly and for old people increases slightly.’

From which age onwards does the recommended maximum heart rate increase as
a result of the introduction of the new formula? Show your work.

An example of a PISA Science Unit with one of the unit questions5

Another way that Peter gathers information to improve road safety is by the use of a TV camera
on a 13 metre pole to film the traffic on a narrow road.The pictures tell the researchers such
things as how fast the traffic is going, how far apart the cars travel, and what part of the road the
traffic uses.Then after a time, lane lines are painted on the road.The researchers can then use the
TV camera to see whether the traffic is now different. Does the traffic now go faster or slower?
Are the cars closer together or further apart than before? Do motorists drive closer to the edge
of the road or closer to the centre now that the lines are there? When Peter knows these things
he can give advice about whether or not to paint lines on narrow roads.

Suppose that on one stretch of the narrow road

Peter finds that after the lane lines are painted the
traffic changes as below.
Speed Traffic moves more quickly
On the basis of these results it was decided that
Position Traffic keeps nearer edges of road
the lane lines should be painted on all narrow
Distance apart No change
roads. Do you think this was the best decision?
Give your reasons for agreeing or disagreeing.
Agree
Disagree
Reason

rd13
General academic abilities
The second approach to the assessment
of higher order thinking skills focuses on the
mental processes needed to benefit from
Example A
instruction (such as comparing, evaluating,
justifying and making inferences). At ACER
these skills are assessed in tests of general
academic abilities. These tests aim to
measure how well students reason.

Two broad domains are sampled in these Example B

tests: reasoning in the ‘natural world’
(mathematics, science and technology); and
reasoning in ‘human world’ (humanities, arts
and social sciences). While these broad
domains reflect a basic division in school
curricula, the tests focus on the styles of
thinking elicited when we reason in the two
domains rather than the kinds of material The image above suggests that we:

that are considered, or school knowledge. A cannot escape conformity.

These styles of thinking can be seen as two B can effectively disguise ourselves.
ends of a continuum. Reasoning in the C appear to be anonymous but
natural world elicits styles of thinking that are unique.
typically can be described as ‘objective’ or
D appear to be unique but
‘scientific’. They include problem solving, and
are anonymous.
using logical analysis to draw inferences.
The finger-print is known around the In this parade, Alfonzo (white
Categorisation, representation,
world, and its suggestions and uniform) is second from the front,
transformation of information, relationships
significance can be interpreted from a third from the right. Barthollo is in
and objects are important. Reasoning in the
range of different cultural another, similar parade. He is fourth
human world elicits styles of thinking that
perspectives. The image plays the from the front, third from the back,
typically can be described as ‘subjective’.
apparent anonymity of a faceless, second from the left and third from
They include the interpretation of subjective
nameless human figure against the the right.
human constructs, and holistic judgements
about meaning. Vocabulary and socio- unique personal identity embodied by How many people are there in
cultural understandings are important. the fingerprint. Identifying the correct Barthollo’s parade?
response depends on the candidate’s
At the extreme ends of the continuum the A 20 B 24 C 28
ability to perceive this paradox within
styles of reasoning elicited are clearly D 42 E more than 42
the image and then work out its most
differentiated. For example, at the human
likely meaning. Distractors A and B The correct response to this item is B.
world end ‘interpretive reasoning’ is elicited
are wrong because neither captures
(see Example A), whilst thinking typical of In this item, the stimulus describes
this central feature of the image; in
the ‘natural world’ (mathematics, science and how a person’s position in a group is
fact, the uniqueness of a fingerprint
technology) elicits styles at the other end of designated. Then a description is given
suggests rather the opposite of
the continuum (see Example B). of a person’s location in another
conformity or disguise. D is wrong
group. The problem is to determine
Of course, the separation of styles of because the relationship between
the size of this second group. There is
thinking within contexts is not watertight. uniqueness and anonymity is reversed;
enough information for the candidate
For example, reasoning in both the natural it is the uniqueness that is
to see that if the person is fourth
and the human world includes ‘critical fundamental and real rather than
from the front and third from the
thinking’; and a single piece of stimulus can the anonymity.
back, then there must be six rows
address both the natural and human world
marching forward. The other
and can elicit objective and subjective styles
information shows that there are four
of reasoning.
columns. Thus there are twenty-four
And, of course, the assessment of higher people in all.
order thinking skills can occur at different
levels of specificity. The two examples
illustrate the measurement of skills at a
broad level of generality that samples a

rd14
range of skills. For example, a test of
reasoning in the human world might draw
on ‘understanding’ (recognition of explicit
and implicit meanings through close reading
of words and phrases, and global
interpretations of text); ‘interpretation’
(interrelating, elaborating and extending
concepts and ideas and drawing
conclusions); and ‘critical thinking’ (making
discriminations and judgements in the realm
of plausible reasoning and/or making
assessments and value judgements).

These general academic abilities could be

conceptualised, measured and reported at
a greater level of specificity. For example, it
would be possible to measure and report
students’ ability in the specific area of
‘empathy’ (narrowly defined) rather than the
more general area of ‘interpersonal
understandings’, or ‘socio-cultural
understandings’, or the still more general
area of ‘reasoning in the human world’.

The two different approaches to the

assessment of higher order thinking skills
described here – PISA tests of performance
‘beyond the school curriculum’, and tests of
‘general academic abilities’ – reflect two
different contexts in which higher order
thinking skills are employed. Ongoing
challenges for work in this area include
continued refinement of our understanding
of higher order thinking and continued
exploration of the contexts in which higher
order thinking might be developed, assessed
and monitored. ■
References:
Kirsch, I. (2001). The International Adult Literacy Survey
(IALS): Understanding what was measured. ETS Research
Report. Princeton: Education Testing Service.
ETS (2004). Adult Literacy and Life Skills Survey: Numeracy.
www.ets.org/all/survey.html
New Releases
OECD (2003). The PISA 2003 Assessment Framework –
Mathematics, Reading, Science and Problem Solving
Knowledge and Skills. Paris: Organisation for Economic
Co-operation and Development.
from ACER Press
Wheeler, P. and Haertel, G. (1993). Resource Handbook
For enquiries contact:
on Performance Assessment and Measurement. Berkeley,
CA:The Owl Press.
Customer Service
1 For further information on essential learning see the 347 Camberwell
Camberwell Road,
Road, Camberwell
Camberwell VIC
VIC 3124
3124
347
Tasmanian Education Department website:
Toll Free
Toll
T Free 1800 338
1800 338 402
402 Tel
Tel
T (03) 9835
(03) 9835 7447
7447
www.education.tas.gov.au
2 For further information on PISA see Fax
Fax (03) 9835
(03) 9835 7499
7499 Email
Email <[email protected]>
[email protected]
r
www.pisa.oecd.org <www.acerpress.com.au>
www.acerpress.com.au
3 Competencies more obviously across disciplinary

boundaries also have a growing importance in PISA.

PISA 2000 analysed students' approaches to learning
and beliefs in their own abilities, motivation and
engagement and other aspects of student attitudes,
under the heading "self-regulated learning" and in
2003, PISA assessed students’ ability to solve problems.
4 OECD, 2003, p.64
5 OECD, 2003, p.144

rd15