Professional Documents
Culture Documents
Aptis Scoring System Layout Final PDF
Aptis Scoring System Layout Final PDF
KAREN DUNN
Technical Report
Aptis Scoring System
TR/2019/001
Karen Dunn
Assessment Research Group, British Council
ISSN 2057-7168
© BRITISH COUNCIL 2019
www.britishcouncil.org/aptis
PAGE 1
APTIS SCORING SYSTEM
KAREN DUNN
CONTENTS
1. INTRODUCTION 3
2. REPORT SUMMARY 3
3. OVERVIEW OF APTIS SCORE REPORTING 3
3.1 Scores and levels for individual skill areas 3
3.2 Overall CEFR level 4
4. APTIS AND STANDARDS 4
5. THE COMMON EUROPEAN FRAMEWORK OF REFERENCE
FOR LANGUAGES (CEFR) 5
6. ALIGNING APTIS WITH THE CEFR 7
6.1 CEFR levels and the Core component 7
6.2 Comparability with other tests 7
7. CEFR LEVEL ALLOCATIONS FOR INDIVIDUAL SKILL AREAS 8
8. CEFR LEVEL ALLOCATIONS REPRESENTING OVERALL L2 ENGLISH ABILITY 10
8.1 Relationship between the overall CEFR and the overall numerical score 10
9. INTERPRETING CEFR LEVELS AND NUMERICAL SCORES AWARDED
TO CANDIDATES 11
9.1 Individual skill areas 11
9.2 Overall English language performance 11
REFERENCES 12
LIST OF TABLES
Table 1: Cronbach’s Alpha mean values for major Aptis variants 2016–17 5
Table 2: CEFR level coverage for major Aptis variants 6
Table 3: Illustration of overall CEFR level calculation 10
Table 4: Illustrative combinations of overall numerical score and
overall CEFR level allocations 10
LIST OF FIGURES
Figure1: The Aptis Core component feeds into CEFR level allocation for all four skill areas 9
Figure 2: Illustration of the “grey area” in which candidate CEFR level allocation is
contingent on Core component performance 9
Acknowledgements
For their contribution to the content development and refining of early drafts, acknowledgements
are made to Judith Fairbairn and David Akast of the British Council's Global Assessment team.
PAGE 2
APTIS SCORING SYSTEM
KAREN DUNN
1. INTRODUCTION
A test’s scoring system is one of three elements that lies at the heart of the test in the socio-cognitive
model of validation (O'Sullivan, 2015). With this core role in mind, the Aptis scoring system has been
developed to provide meaningfulness in the use and interpretation of scores for all stakeholders.
This document explains the key features of the scoring approach for an audience with an interest
in the mechanisms and technicalities of Aptis scores.
2. REPORT SUMMARY
This report describes the scoring and reporting processes in the Aptis suite of tests, as well as
outlining the rigorous standards by which Aptis is constructed and aligned with the Common European
Framework of Reference for Languages (CEFR) (Council of Europe, 2001).
An overview of Aptis score reporting is given in Section 3. This is supported by information about
setting and maintaining standards in Section 4 and the relationship between Aptis and the CEFR in
Section 5. Information about the process by which Aptis is benchmarked to the CEFR and how cut
scores are established is found in Section 6, with further details on CEFR level allocations for
individual skill areas and overall L2 English ability in Sections 7 and 8 respectively. Finally in
Section 9, there is a discussion on the use and interpretation of Aptis test scores, both on the
numerical scale and as CEFR level.
3. OVERVIEW OF
APTIS SCORE REPORTING
Aptis test performances for the four skills components – Listening, Reading, Speaking and Writing –
are reported on two scales: as a numerical score and a CEFR level. Test performance in the Grammar
and Vocabulary component, the “Core” component, is reported as a numerical score only. Candidates
who sit all four skills components, plus Grammar and Vocabulary, are additionally awarded an overall
numerical score and an overall CEFR level which summarises their L2 English language ability.
PAGE 3
APTIS SCORING SYSTEM
KAREN DUNN
For the skill components, a CEFR level is assigned according to the score obtained in each skill.
Key points to note in this score allocation process are as follows:
1. Each skill component has designated set of score ranges which determine CEFR level
allocation. The precise scores required to achieve a specified CEFR level differ for each skill,
i.e. the range of scores that will achieve a B1 is different for Writing compared to Listening.
This means the numerical scores are not directly comparable between skill areas.
2. For a given skill component, if a candidate’s score falls into the borderline area immediately
below a CEFR cut-score boundary, CEFR allocation for that skill is refined based on their
performance in the Core component. A strong Core performance will see them pushed up to
the next CEFR level. It is possible therefore for two candidates to be awarded the same
numerical score and a different CEFR allocation for the same component if they have scored
differently in the Core component.
PAGE 4
APTIS SCORING SYSTEM
KAREN DUNN
Table 1: Cronbach’s Alpha mean values for major Aptis variants 2016–17
For the productive skills (Speaking and Writing) a measure of “inter-rater” reliability is given. This is
based on the examiner performance on a series of “control items” within the system, and indicates
how consistently the examiners are marking. Each examiner’s workload will comprise approximately
5% control items, i.e. responses taken from a pool of candidate performances that have been
scrutinised by senior examiners. All examiners are required to mark within tolerance of the agreed
mark, and if not, they are required to undertake further training.
The control items are regularly refreshed so that an examiner will not encounter the same control item
more than once. The inter-rater reliability is calculated based on the agreement of examiners on the
marks awarded for the control items. For the 2016–17 Annual Operating Report, a mean inter-rater
reliability value of .89 was recorded for Speaking and .85 for Writing for the Aptis general variant.
A1 – A2 (Basic User)
B1 – B2 (Independent User)
C1 – C2 (Proficient User)
1 https://www.coe.int/en/web/common-european-framework-reference-languages/uses-and-objectives
PAGE 5
APTIS SCORING SYSTEM
KAREN DUNN
The CEFR levels give guidance about language in use across the main language skills. These are
fully elaborated in the CEFR (Council of Europe, 2001), with additional scales and descriptors
in the Companion Volume (Council of Europe, 2018). Descriptive information, illustrative of the
type of performance to expect at the different proficiency levels is given in a range of scales for
communicative activities and strategies, including overall reading and listening comprehension,
spoken and written production, and spoken and written interaction2. Scales for linguistic competencies
address aspects of grammatical and vocabulary range and accuracy but only in very broad terms.
These are deliberately under-specified in the CEFR, as the CEFR was intended as a common
framework applicable to multiple languages. The CEFR makes clear that aspects of grammatical
and lexical knowledge need to be fleshed out in local applications for different languages (Council
of Europe, 2001, p. 33).
The range of CEFR levels it is possible to be awarded in Aptis depends on the test variant taken.
Aptis General, Teachers and Teens do not differentiate at the higher end of the scale between C1
and C2 level because the tasks included in the test are designed and written to challenge English
language learners with a range of abilities between A1–B2. If a candidate does exceptionally well in
one of these Aptis test variants, it is clear that their proficiency level exceeds B2, and will be in the
C range. It is recommended in these cases for candidates to sit the Aptis Advanced test, which
includes targeted tasks that can discriminate between performances at C1 and C2 level.
See Table 2 for a summary of the CEFR level coverage across major Aptis variants.
A0 A1 A2 B1 B2 C(1)2 C2
General ü ü ü ü ü ü N/A
Teachers ü ü ü ü ü ü N/A
Teens ü ü ü ü ü ü N/A
Advanced A0–A2 1 ü ü ü ü
2 The original CEFR documentation contains 57 different scales detailing communicative activities, strategies,
and competencies, with further scales added in the Companion Volume.
PAGE 6
APTIS SCORING SYSTEM
KAREN DUNN
For further details of the full alignment procedure please refer to O’Sullivan (2015).
Aptis has benefited from being able to base test specification, task development, and rating scale
development from the outset on the CEFR. It is important to note that standard setting itself, while
essential, is only one part of a comprehensive alignment process. In addition, ongoing research
continues to contribute to the validation of the CEFR alignment (e.g. Dunlea et al, 2018). The Aptis
approach to best practice in project delivery across not only exams but all of the areas of activity of
the British Council is driven by an evidence-centred approach.
PAGE 7
APTIS SCORING SYSTEM
KAREN DUNN
The decision as to where the cut scores will fall in an Aptis test component is reached via a process of
standard setting (see Section 6 above), in which the decision is made regarding which score indicates
that a particular level of English language proficiency has been reached for each successive CEFR
level. Cut scores in Aptis are not the same across each of the skills components for each of the CEFR
levels. This is because the content and design of the tests has been independently evaluated and
aligned with the language descriptors set out in the CEFR. In order to accurately reflect a candidate’s
level of English language ability in any given skill area, each skill is addressed separately and the cut
scores relevant for each CEFR level are based on the data and evidence collected in the alignment
process. For Speaking and Writing skills, this reflects the quality of the language that is produced to
achieve a score that meets each threshold. For Listening and Reading, this reflects the level of
understanding indicative of a given CEFR level. This means that the exact point on the skill-specific
score scale at which a person demonstrates a B2 level of proficiency, for example, will not be the
same in the Reading component as it is for Listening.
In cases where a candidate’s score for a given component falls in the borderline between two levels,
Aptis uses score information from the Core component to refine the final CEFR level allocation. In this
way, the Grammar and Vocabulary score feeds into the decision-making process for the awarding of
CEFR levels across all skills: Listening, Reading, Speaking and Writing. This increases the fairness
and accuracy of CEFR level allocation (McCray & Dunn, forthcoming).
The rationale for doing this is as follows:
§ In allocating a test score, there will always be a degree of error in the test’s reflection of the
candidate’s true ability in any given skill. This is known as “standard error of measurement” in
testing theory. The aim is to minimalise this error during test development, but it is impossible
to eradicate fully.
§ Grammar and vocabulary have been shown theoretically to be key processes in models of
receptive and productive L2 language ability (e.g., Field, 2013; Grabe & Kaplan, 1996;
Khalifa & Weir, 2009; Levelt, 1989), and empirically good predictors of language proficiency
(e.g., Shiotsu, 2010). A solid foundation in grammar and vocabulary knowledge can therefore
be viewed as relevant across all skill areas.
§ Scores achieved in the grammar and vocabulary component provide a valuable additional
source of evidence in finalising CEFR level allocation in the Aptis test for all skill areas,
especially for the crucial thresholds between B1 and B2, B2 and C (McCray & Dunn,
forthcoming).
PAGE 8
APTIS SCORING SYSTEM
KAREN DUNN
Figure1: The Aptis Core component feeds into CEFR level allocation for all four skill areas
Reading
Core
Writing Grammar & Speaking
Vocabulary
Listening
The process of determining CEFR level cut scores in Aptis is the same, regardless of the test variant
taken (General, Teachers, Teens, Advanced). All candidates are required to sit the Core component,
alongside the combination of skills tests they are entered for. The scale scores will be recorded and
reported, however, if any of these scale scores fall just below a cut score threshold, i.e. within the
margin of error, then the system refers to the candidate’s Core component score to determine which
side of the threshold is most likely to accurately reflect the candidate’s ability. If the candidate has a
strong performance in the Core component, they will be given the benefit of the doubt, and awarded
the higher CEFR level. This process is illustrated in Figure 2.
Figure 2: Illustration of the “grey area” in which candidate CEFR level allocation is contingent
on Core component performance
B2
CUT SCORE
B1
This process means that in the operational reality, it is fully possible for two candidates to achieve the
same numerical score, e.g. 40/50 for the speaking test, yet for one of the candidates to be awarded a
B1 and the other a B2. This is because 40/50 is an Aptis ‘grey zone’ score for Aptis General Speaking
test. The candidate who was awarded a B2 will have performed better in the Core component than the
candidate awarded a B1.
PAGE 9
APTIS SCORING SYSTEM
KAREN DUNN
Table 4: Illustrative combinations of overall numerical score and overall CEFR level allocations
PAGE 10
APTIS SCORING SYSTEM
KAREN DUNN
In comparing ability within a single individual skill area, it is however a valid approach to refer first to
the CEFR level and then to the numerical score for a more detailed comparison. So, for example, in
comparing candidates who both achieved B2 in the Speaking component, it is valid to refer to the
numerical score each candidate achieved on this component to establish which is the strongest at this
skill. The numerical scores for individual skills can also be used to give an indication of improvements
in Aptis test performances at the component level by the same person across different sittings in
cases where an increase of a full CEFR level might not be possible. Equally, this information can be
used to investigate the benefit of different teaching interventions for groups of students, since it is
more nuanced than the CEFR level information.
PAGE 11
APTIS SCORING SYSTEM
KAREN DUNN
REFERENCES
Council of Europe. (2001). Common European Framework of Reference for Languages.
Cambridge: Cambridge University Press.
Field, J. (2013). Cognitive validity. In L. T. A. Geranpayeh (Ed.), Examining listening (pp. 77–151).
Cambridge: Cambridge University Press.
Grabe, W., & Kaplan, R. B. (1996). Theory and practice of writing: an applied linguistic perspective.
London: Longman.
Khalifa, H., & Weir, C. J. (2009). Examining Reading: Research and practice in assessing second
language reading. Cambridge: UCLES/Cambridge University Press.
Levelt, W. J. M. (1989). Speaking: From intention to articulation. Cambridge, MA, US: The MIT Press.
McCray, G., & Dunn, K. (Forthcoming). Validity and Usage of the Aptis Grammar and Vocabulary
(Core) Component.
Shiotsu, T. (2010). Components of L2 Reading: Linguistic and Processing Factors in the Reading
Test Performances of Japanese EFL Learners. Cambridge: Cambridge University Press and
Cambridge ESOL.
PAGE 12
Aptis Scoring System
Author??
www.britishcouncil.org/aptis
2019