Differentiation in Stage 6 Assessment

UNSW, where Ignite the Spark 2018 took place

About a week ago I attended the Ignite the Spark 2018 Conference. This Professional Learning was mainly focused on assessment in relation to giftedness but what was most interesting across the conference was that, underneath a rhetoric built around excitement for assessment and the need for teacher quality, the real subtext of most of the sessions seemed to be that the HSC Exams have become obsolete and that the requirements handed down from NESA and the DoE aren't represented by this increasingly irrelevant process.

But, like it or not, the HSC exams remain largely unchanged and this means that Stage 6 has become an increasingly challenging transition phase between all the requirements of 21st century learning covered by Years K-10 and the need to perform well in a 19th century-styled examination at the end of Year 12. 

Enter Lisa O'Neill, the Head Teacher of HSIE at St Mary's Senior High, who presented a brilliant seminar at Ignite the Spark that centred on how we can better meet the challenges set forth by the demands of the HSC in 2018 and beyond by differentiating our assessment of Year 11 and 12. In her presentation Lisa identified our tendency as teachers to focus on preparing students for exams, and the way that our assessment tasks lead towards the HSC exams, and questioned the capacity of these assessments in actually helping these students to learn. We can't eradicate the exam but there is a concerted need to shift our thinking so that students become more skill-focused. NESA seems to be acknowledging this through the mandating of only 1 formalised examination per subject, per year. There is an undeniable tension, though, between the push for assessment in Stage 6 to become less focused on preparing for the HSC exams and the HSC exam-focused culture that already exists in NSW schools.

Here are some things we need to consider:

• NESA syllabuses include a requirement that each KLA assesses their students using assessment as, of, and for learning. I've tended to interpret this as meaning that not all assessment tasks should be based on establishing data that assesses student ability (assessment of learning).
• About 60% of domestic university student enrollments from 2014-2017 were non-ATAR, which suggests that universities are becoming less concerned with the HSC examination when it comes to evaluating who they admit into their courses.

Getting back to Lisa O'Neill's Differentiation in Stage 6 Assessment presentation, she takes her lead from John Hattie in approaching differentiation as something that "relates more to addressing students' different phases of learning from novice to capable to proficient rather than merely providing different activities to different (groups of) students" (from Hattie's Visible Learning for Teaching). O'Neill also piqued my interest by using this to interpret the as, of and for dichotomy of assessment in a more holistic sense - that each individual assessment task should encapsulate all three of these forms of assessment. 

I think I'm down with that. 

Linking her work to the Professional Standards, O'Neill highlighted Standard 1, IE. 'Know Your Students', and clarified her interpretation of differentiation as having one broad task that allows access points for all students. In a nutshell, assessment:

• Should provide students with multiple ways to achieve.
• Can provide access points for students moving through a spectrum of novice to capable to proficient.

O'Neill presents on the Assessment Cycle of Love

 Here are some other recommendations made by O'Neill:

1. Use what she refers to as the 'Assessment Cycle of Love', a diagrammatic tool for designing assessment, to identify gaps in your tasks. Are the outcomes, marking criteria, feedback, and process of assessment all as per NESA advice? Are assessment as, of and for learning all functional within the task?
2. Staff evaluation should be an integral part of the process, with consideration of whether a task has allowed students to set learning goals, and what skills students were taught while undertaking the task.  
3. Know the data - use SMART, SCOUT, our own assessment tasks, and the Educator Calculator (a data tool for assessment that can be found at the DoE Centre for Education on Statistics and Evaluation).
4. Consider the need for access points in tasks so students don't become disheartened when an assessment task has shown them that they can't do anything in preparation for the HSC exams. They might be able to access a band in the marking criteria, but can they access a skill set?
5. Include feedback in the marking criteria. Feedback is pivotal for Stage 6 assessment and all criteria should be a form of assessment for learning (IE. What can teachers and students alike gain from reading this criteria in preparing for future learning?)
6. Can we shift the student mindset away from marks and towards skills? Have students make an appointment for feedback conferencing and use this time for students to annotate their own response. Identity their areas of weaknesses ahead of time and get them to look for these weaknesses in the task so that they can establish learning goals.

Some Useful Links

"Crunching the Numbers" - A report from the Mitchell Institute on the use and usefulness of the ATAR.
Student Conferencing and Feedback  - providing students with 1:1 feedback to facilitate personal growth.
Assessment For Learning - an alternative approach in using separate diagnostic assessment with students.

Ignite the Spark 2018 Conference: Ways of Assessing Giftedness

On Friday I attended the Ignite the Spark Conference at UNSW in the hope of learning some new stuff about assessment. Associate Professor David Blaazer, the Deputy Dean for UNSW Arts and Social Sciences, opened up the conference after a reflective Welcome to Country delivered by a representative of the Dharawal people, and this led into an address from DoE Director of Secondary Education Lila Mularczyk, who spoke enthusiastically about the conference's role in highlighting the need to tie our assessment practice to current policy. 

Assessment is a funny beast though.

All teachers do it so it's not KLA specific, and the need for Professional Development around assessment can sometimes get relegated elsewhere due to its non-denominational nature. With this in mind, it's always interesting to see who shows up to these kinds of conferences. Suffice too say - there were a lot of teachers here and the presenters represented many different sectors of the education system - from primary to secondary, private to public, and including counsellors, psychologists, NESA representatives, DoE corporate, and other agencies who help care for our kids.  

The keynote address (which this blog focuses on) was delivered by Dr. Jae Yup Jared Jung, and focused on his research relating to assessment in gifted education, which represented a systematic review he has published in the Journal of Advanced Academics (28, 163-203). NESA considers 'differentiation' to include adjusting and modifying assessment activities in order to cater for both individual students and larger groups, and the idea of assessing students to locate giftedness is accordingly mandated. It should be noted, however, that there is no real standardisation for catering to the gifted in NSW and this is why Dr Jung's research is so incredibly relevant.

Dr. Jung demonstrates what a non-verbal ability test looks like.

The Research

In his search of research literature from 2005 to 2016, Dr. Jung essayed each of the major academic databases and the reference lists of retrieved articles found in the top research journals. By doing this, he found the following:

• 148 articles in total that dealt with the topic of assessment in gifted education.
• A clear majority within this of 128 articles that deal with identification of gifted students.
• Only 14 articles that covered evaluations of gifted programs.
• And just 8 articles that deal with learning growth. 

The majority of the presentation went on to cover the ways that gifted students are identified (since this is where the majority of research focuses). Dr Jung identifies two major categories of common assessment: Objective (or Traditional) assessments, which are quantitative and result in a number used to identify ability, and Subjective (or Non-Traditional), which involve qualitative judgements made by teachers, psychologists, counsellors, parents, etc. 

Objective Assessments

1. IQ Tests

These involve problem solving, utilising logic, and demonstrating recognition of patterns or relationships. Examples of this kind of testing includes the Wechsler Intelligence Scale for Children (WISC-V), the Stanford Binet 5 test, the Kaufmann Brief Intelligence Test, the Cognitive Abilities Test (CoGAT), the Australian Council for Educational Research General Abilities Test (AGAT), Raven's Progressive Matrices (RPM), and the Naglieri Nonverbal Ability Test II (NNAT-II).   

Benefits: The WISC-V is designed to measure five different areas (verbal comprehension, fluid reasoning, visual spatial ability, working memory, and processing speed) and therefore allows for the recording of strengths and weaknesses in said areas. The Kaufmann Brief Intelligence Test is an abbreviated IQ test that can be given by teachers rather than psychologists, and the CoGAT and AGAT can be administered in groups rather than individually (which is practical for schools). The Raven's and Naglieri methods are nonverbal ability tests that assess students through the use of visual-spatial stimuli and can be useful because they reduce the reliance on a student's prior educational history or ability to speak English as a first language (or at all).

Concerns: The WISC-V and Stanford Binet 5 tests must be administered by appropriate accredited professionals (psychologists) and are designed for 1:1 testing. These can therefore be quite expensive; $1000-2000 per individual student.

Quote: "Aw, c'mon! Tell me which Poplar Tree is the tallest!!!" The Window Effect refers to the idea that if we assess whether students can achieve to a certain point then we can't see what they could do beyond this point.

2. Above Level Tests

Achievement tests designed for older students in higher grades, with 'higher ceilings' (see cartoon above). These tests are difficult for the majority of testees to 'top out', and assist in the search for talented individuals whose abilities may go beyond the scope of most assessment tasks. Examples include American-style examinations such as the SATs, ACTs, and EXPLORE tests. 

Benefits: Allows for the identification of outliers: the small percentage of people whose ability does not fit on the regular scale of assessment. This style of testing is also designed for specific age groups, so it allows for more targeted assessing of students.

Concerns: Results can be disheartening for the majority of students who undertake the test, so this data usually needs to be kept confidential and for teacher purposes only.

3. Computer Adaptive Tests

Online testing instruments that adapt questions to suit student ability, and scale each question based on prior performance in the same test so that the student is constantly working within their zone of proximal development. Abbreviated as 'CATs'.

Benefits: Can be a very efficient way to gather data as the computer generates reports automatically. It also differentiates the test for each individual student, which means no two tests are the same.

Concerns: CATs can be calibrated too closely to a 'pass-fail' dichotomy that doesn't allow for a diverse data set. These tests can also only measure as far as the student goes - when a student or the test finishes we may still not know what the student is truly capable of.

Subjective Assessments

1. Nominations

Having 'gifted' students nominated for recognition allows for us to consider the unique perspectives of various groups familiar with gifted students. One such group are parents, who are a subjective source of information but can also provide lots of information that we don't or can't have access to otherwise, such as milestones like the moment the child first spoke or formed a sentence.

Benefits: Nominations allow for assessment of different types of giftedness, and can also be a useful way of incorporating the input of a student's individual teachers (therefore making them part of the consultative process, which is useful and supportive in the formation of any kind of school-based gifted program).

Concerns: This process relies on human judgement and therefore lacks the psychometric rigour of I.Q. testing and similar methodologies.

2. Performance-Based Assessment

These are assessments that look more like subject-specific assessment tasks. This style of assessment could be: asking students to act in a prescribed way to create a product or a response, such as a written composition, or a teacher observing students in problem-based learning activities.

Benefits: The familiarity of this kind of testing will appeal to teachers already used to gathering data in this way. This method is also fairly open-ended and can elicit a wide range of results.

Concerns: There may be a lack of clarity in how it should be scored as each teacher will mark to the criteria in their own particular way. It can also be more time-consuming than objective methods of assessment as each individual student requires an individualised evaluation of their work.

3. Dynamic Assessment

Of all the things Dr. Jung spoke about in his keynote, I found his discussion of Dynamic Assessment the most interesting. I probably found this the most interesting thing about the whole conference and I can see myself looking forward into it in the immediate future.

Dynamic Assessment is a process that involves interaction between the teacher and the student, with a focus on how the student responds to educational interventions. It's not about the student's response to the task but about measuring how a student improves or reacts to a task. There are two main kinds: 

• The first is the 'Sandwich Format', in which a student is pretested, given an intervention, and then post-tested. In the post-test the level of improvement in the student is measured in comparison tot he pretest. 
• The second kind is the 'Cake Format', in which a student is given a task with a series of items and assistance is provided only when the student encounters difficulties. The amount of instances, or types of intervention, are then measured to show how much growth the student exhibited, or how much/little help they needed in independently adapting to the demands of the task.

Benefits: This process deliberately acknowledges the inequality of educational opportunity among students - as each student comes to us with their own unique context this means that their giftedness may have been either nurtured in the past or completely ignored. The result is that there isn't an even playing field when it comes to trying to 'see' the natural level of giftedness in a student. Using Dynamic Assessment minimises the role of performance in previous assessment tasks as this form of assessment focuses on observing the student's capacity in adapting to difficulty.

Concerns: A high degree of standardisation is needed when training teachers to administer this process of assessment. As such it can be time-consuming, resource and staff-intensive, and involves a lot of effort.

Dr. Jung

Further Observations of Dr. Jung

1. Multiple Criteria Identification

With so many different methods of assessment available to identify gifted students the consensus seems to be that multiple criteria should be used together. This allows for the minimisation of bias and multiple opportunities for students from diverse backgrounds to demonstrate giftedness. So, with this in mind, Dr. Jung asserts that we should use both objective and subjective methods. Some other things to take into account:

• We need to ensure that we're using instruments appropriate to our context - this means look at current student numbers, demographics of the region, etc.
• Identification needs to take place as early as possible and there should also be training provided for the teachers identifying the students.
• If using multiple criteria, the data should be collected and analysed concurrently so no students get left behind.

2. Disadvantaged Groups in Gifted Programs

Dr. Jung also expressed several concerns in relation to the under-representation of students from disadvantaged backgrounds. He posited that this happens due to several factors:

• Inappropriate/unfamiliar content in assessment tasks.
• Pre-defined conceptions of giftedness could be inherent in assessment tasks.
• Pre-conceptions of teachers could lead to unconscious bias.
• Socio-economic factors.
• Lack of a mechanism for universal assessment of gifted students in NSW.

This last point is a big issue with giftedness in general. Not all schools test for giftedness and the ones that do may do so in very different ways, with specific ideas of what giftedness means. Many schools have 'self-select', 'opportunity', or 'academic-focus' classes but the criteria used to create these classes often have little to do with natural giftedness and more to do with prior performance.

Nevertheless, as we move forward into the 21st century, I have strong hopes that NSW will inch its way closer to some kind of standardised mandate to identify and better support those who may be hiding in plain sight.

Dr. Jung's research can be found here, but you will need access to academic journals in order to read it in full.