jump to navigation

Who benefits from online marking of NAPLAN writing? October 26, 2017

Posted by Editor21C in Directions in Education, Education Policy and Politics, Primary Education, Secondary Education.
Tags: , ,
add a comment

By Susanne Gannon

In 2018 most students in most schools will move to an online environment for NAPLAN. This means that students will complete all test sections on a computer or tablet. Test data that is entirely digital can be turned around more rapidly so that results will be available for schools, systems and families much faster.

The implication is that the results can be put to use to assist students with their learning, and teachers with their planning. While this appears to address one of the persistent criticisms of NAPLAN – the lag between testing and results – other questions still need to be asked about NAPLAN. Continuing concerns include high stakes contexts and perverse effects (Lingard, Thompson & Sellar, 2016), the marketization of schooling (Ragusa & Bousfueld, 2017), the hijacking of curriculum (Polesel, Rice & Dulfer, 2014) and the questionable value of NAPLAN for deep learning (beyond test performance).

Almost ten years after its introduction, NAPLAN has been normalised in Australian schooling. Despite some tweaking around the edges, the original assessment architecture remains intact. However, the move to online delivery and automated marking represents a seismic shift that demands urgent attention.

Most student responses in NAPLAN are closed questions. In the new online format these include multiple choice, checkbox, drag and drop, reordering of lists, hot text, lines that can be drawn with a cursor and short answer text boxes. These types of answers are easily scored by optical recognition software, and have been since NAPLAN was introduced.

However the NAPLAN writing task, requiring students to produce an extended original essay in response to an unseen prompt, has always been marked by trained human markers. Markers apply a detailed 10 point rubric addressing: audience, text structure, ideas, persuasive devices, vocabulary, cohesion, paragraphing, sentence structure, punctuation and spelling. In years when narrative writing is allocated, the first four criteria differ however the remaining six remain the same. Scores are allocated for each criterion, using an analytic marking approach which assumes that writing can be effectively evaluated in terms of its separate components.

It is important to stress that online marking by trained and highly experienced teachers is already a feature of high stakes assessment in Australia. In NSW, for example, HSC exams are marked by teachers via an online secure portal according to HSC rubrics. The professional learning that teachers experience through their involvement in such processes is highly valued, with the capacity to enhance their teaching of HSC writing in their own schools.

Moving to online marking (called AES or Automated Essay Scoring by ACARA, also called machine-marking, computer marking or robo-marking) as NAPLAN proposes is completely different from online marking by teachers. While the rubric will remain the same, judgement of all these criteria will be determined by algorithms, pre-programmed into software developed by Pearson, the vendor who was granted the contract. Algorithms cannot “read” for sense, style, context or overall effectiveness in the ways that human experts can. All they can do is count, match patterns, and apply proxy measures to estimate writing complexity.

ACARA’s in-house research (ACARA NASOP Research Team, 2015) insists on the validity and reliability of the software. However, a recent external evaluation of ACARA’s Report is scathing. The evaluation (Perelman, 2017), commissioned by the NSW Teachers’ Federation from a prominent US expert, argues that ACARA’s research is poorly designed and executed. ACARA would not supply the data or software to Perelman for independent examination. However it is clear that AES cannot assess key aspects of writing including audience, ideas and logic. It is least effective for analytic marking (the NAPLAN approach). It may be biased against some linguistic groups. It can easily be distorted by rewarding “verbose high scoring gibberish” (Perelman, 2017, 6). The quality of data available to teachers is unlikely to improve and may lead to perverse effects as students learn to write for robots. The risk of ‘gaming’ the test is likely to be higher than ever, and ‘teaching to the test’ will take on a whole new dimension.

Human input has been used in ACARA’s testing of AES in order to train and calibrate the software and in the future will be limited to reviewing scripts that are ‘red-flagged’ by the software. In 2018 ACARA plans to use both human and auto-marking, and to eliminate humans almost entirely from the marking process by 2019. In effect, this means that evaluation of writing quality will be hidden in a ‘black box’ which is poorly understood and kept at a distance from educational stakeholders.

The major commercial beneficiary, Pearson, is the largest edu-business in the world. Educational assessment in the UK, US and now Australia is central to its core business. Details of the contract and increased profits that will flow from the Australian government to Pearson from the automated marking of writing are not publicly available. Pearson has already been involved in NAPLAN, as several states contracted Pearson to recruit and train NAPLAN markers. Pearson have been described as a “vector of privatisation” (Hogan, 2016, 96) in Australian education, an example of the blurring of social good and private profit, and the shifting of expertise from educators and researchers to corporations.

Writing is one of the most complex areas of learning in schools. NAPLAN results show that it is the most difficult domain for schools to improve. Despite the data that schools already have, writing results have flatlined through the NAPLAN decade. Negative effects and equity gaps have worsened in the secondary years. The pattern of “negative accelerating change” (Wyatt-Smith & Jackson, 2016, 233) in NAPLAN writing requires a sharper focus on writer standards and greater support for teacher professional learning. What will not be beneficial will be furthering narrowing the scope of what can be recognised as effective writing, artfully designed and shaped for real audiences and purposes in the real world.

NAPLAN writing criteria have been criticised as overly prescriptive, so that student narratives demonstrating creativity and originality (Caldwell & White, 2016) )are penalised, and English classrooms are awash with formulaic repetitions (Spina, 2016) of persuasive writing NAPLAN-style. Automated marking may generate data faster, but the quality and usefulness of the data cannot be assumed. Sustained teacher professional learning and capacity building in the teaching of writing – beyond NAPLAN – will be a better investment in the long term. Until then, the major beneficiaries of online marking may be the commercial interests invested in its delivery.

References

ACARA NASOP Research Team (2015). An evaluation of automated scoring of NAPLAN Persuasive Writing. Available at: http://nap.edu.au/_resources/20151130_ACARA_research_paper_on_online_automated_scoring.pdf

Caldwell, D. & White, P. (2017). That’s not a narrative; this is a narrative: NAPLAN and pedagogies of storytelling. Australian Journal of Language and Literacy, 40(1), 16-27.

Hogan, A. (2016). NAPLAN and the role of edu-business: New governance, new privatisations and new partnerships in Australian education policy. Australian Educational Researcher, 43(1), 93-110.

Lingard, B., Thompson, G. & Sellar, S. (2016). National Testing in schools: An Australian Assessment. London & New York: Routledge.

Polesel, J., Rice, S. & Dulfer, N. (2014). The impact of high-stakes testing on curriculum and pedagogy: a teacher perspective from Australia. Journal of Education Policy, 29(5), 640-657.

Ragusa, A. & Bousfield, K. (2017). ‘It’s not the test, it’s how it’s used!’ Critical analysis of public response to NAPLAN and MySchool Senate Inquiry. British Journal of Sociology of Education, 38(3), 265-286.

Wyatt-Smith, C. & Jackson, C. (2016). NAPLAN data on writing: A picture of accelerating negative change. The Australian Journal of Language and Literacy, 39(3), 233-244.

 

Associate Professor Susanne Gannon is a senior researcher in the School of Education and Centre for Educational Research at Western Sydney University, Australia.

What if…my teacher was an app??? February 26, 2012

Posted by Editor21C in Directions in Education, Educational Leadership, Engaging Learning Environments, Primary Education, Secondary Education.
Tags: , , ,
2 comments

from Associate Professor Susanne Gannon

Susanne Gannon writes on the increasing trend in schools towards online learning, and argues that teachers are very important in utilisng new technologies in education; indeed that “the teacher is the key”.

Last November the Wall Street Journal reported in ‘My teacher is an app’ on an increasing trend in the United States for high schooling to be delivered entirely online. In particular states, this phenomenon has been more marked than others. The article reports that Florida requires all public school students to take at least one class online, Idaho will soon require two, while Georgia allows public school students to take entire courses on their iphones or Blackberries. Entirely online high schools are proliferating. Partly these moves are justified by the respective authorities in terms of preparing students for the cyber university, but no doubt they are also partly responses to crises in education that see some states as performing particularly poorly in standardised testing, and to ongoing issues around recruitment and retention of quality teachers for disadvantaged students. While the affordances of connected classrooms and other networking technologies have proved very useful for Australian schools with small numbers of students in particular curriculum areas, or remote and rural schools, and for teacher professional learning, we are – thus far – untouched, as far as I know, by large scale relocations of learning from the ‘real’ to the virtual. The other insidious move that is apparent in the WSJ article is the corporate control of online learning, and through a sort of sleight-of-hand – by supplying services that under-resourced schools may be unable to deliver themselves – an incursion of for-profit companies into not-for-profit public schooling markets.

What about teachers in online schools? They are designers, presumably, of the learning materials that students are accessing on their various devices. At least the corporations are employing teachers, the article tells us. Teachers are built into the modules. One student reported that he listened to ‘most of’ an online lecture that ‘his’ English teacher was presenting, explaining the concept of a protagonist, to the 126 ninth graders who were logged for that session. I am unclear about when and where the teacher will be available to check their understanding and application of the concept to texts that they might be reading (except in a pre-prepared quiz or test) for that module. I suspect that next year’s students will hear the same voice and the same lecture that they hear this year. On the audio podcast there will be no signs of aging, no changes in mood, no changes in teacher. If the teacher is seconded to higher duties, takes leave or is hit by a bus, the students won’t suffer a casual or replacement who may not know them as well as their own teacher. But this teacher is unlikely to know them anyway. To give them their due however, some of the schools do offer email or phone support to students, though delays in response of up to three days are common, or ‘the occasional video conference’. Some offer field trips and live classrooms at a school building for students who prefer that – although most learning will still be online and self-directed. No doubt the profit margins are greatly enhanced when teacher salaries are minimised in the educational delivery equation. The article notes that one school district in Idaho sees the wholesale outsourcing of education to online providers as ‘a creative solution’ to the state’s budget crisis. Other inadvertent consequences of students, and the funding that follows them, shifting out of public schools includes the slashing of LOTE curriculum and cancellation of the school play. Results from these experiments in terms of improved student results are erratic.

It is clear that ICTs are transforming education. A recent mapping exercise we completed as part of the Strategic Secondary Education Research Program (SSERP) for Greater Western Sydney, a partnership between the University of Western Sydney and two large regions of the NSW Department of Education and Communities, found that ICTs and Web 2.0 technologies play ‘a fundamental role’ in planning, delivery and access to innovative educational programs. We plan to begin supporting a cluster of schools to research their practice in this area through 2012. These technologies were taken up within constructivist, collaborative, learning centred pedagogies, enabling new relationships between teachers and students, and extending teachers’ skills and knowledges. Teachers are not replaced by technologies but rather, effective and powerful use of technologies is reliant on teachers and how they are able to make best use of the affordances of technologies to enhance learning. My UWS colleague Jane Hunter’s recent blog entry on 21st Century Learning emphasises how innovative use of technologies is reliant on ‘imitable teachers’ – highly skilled, enthusiastic, imaginative, expert at integrating technologies seamlessly and purposively into classrooms – who can open up learning in classrooms. Jane’s research demonstrates that the teacher is the key.

I began writing this blog just as the new school year was beginning around Australia, and a day after watching the Four Corners program Revolution in the classroom. Anticipating the Gonski review of school funding, the program focused on school based management and drew comparisons between the relative autonomy experienced by private and public schools. However what struck me in the footage was, firstly, the uncompromising focus each school placed on teacher effectiveness and on student learning. Secondly, it was obvious that innovative leadership had created cultures of collaboration and critical reflexive practice in each of the schools, supported by coaching, mentoring and peer support programs for teachers at all stages of their careers and enabling teachers to research and evaluate their own practice. These were also characteristics of the innovative DEC secondary schools that we mapped in 2011 in Western and South Western Sydney, where many of these innovations were enabled by National Partnership funding. Finally, the other striking feature for me of the footage from each of the schools in the Four Corners program was the relationships that were evident between teachers and students. A year 9 student says that it is the ‘teacher-student relationship’ that has made all the difference: ‘it’s about the teacher understanding the student and the way they should be taught’. A year 12 student says that teachers ‘really care’, they’ve shown ‘massive support’ and have pushed and supported students to make sure that ‘you do the best that you can’. This is not a vague sort of ‘teacher-student relationship’ – although connection, respect, interest and engagement are all necessarily part of it – but it is a purposeful relationship that includes a focus on learning about each student and understanding their learning needs. Getting the best out of every student is the goal at each of the three schools that are profiled and this is premised on knowing each student not only through carefully analysed assessment strategies, that are then used to adjust pedagogy, but also on knowing the student as a particular and complex person. The focus on performance that was evident in the Four Corners program carries with it a risk, however, that the ‘tight coupling’ of education to teaching and learning outcomes that can be measured in mandated assessment and other data driven performance measures may reposition anything else as superfluous. The final scenes at the schools cut between the teacher calculating HSC bands and rankings relative to other schools to the warmth of her hug and ‘I’m so proud of you’ for the student who has transcended his ‘well below average’ record in English. An app will never do this.

The provocation of  ‘My teacher is an app…’ points to an ongoing need to maintain the emphasis on teacher quality as the crucial factor in student learning. Teacher quality is not an inherent characteristic of an individual, not a feature of personality, but an ongoing learning journey throughout a career. It is not solely an individual characteristic. Nor is it solely to do with cognitive domains. School leadership structures and an ethos of inquiry and collaboration that supports teachers to continue investigating and improving their practice alongside each other are crucial for this. Research I conducted in a sample of DEC English faculties in our region that had sustained good results in English Extension 2 indicated that strong and collaborative leadership with shared responsibility for improving student learning and teaching quality characterised these faculties. Interviews with high quality graduate English teachers in their first five years of teaching in a range of schools also suggested that the context of leadership in their school and faculty made the difference between their capacities to develop rapidly into effective and excellent teachers and to build the resilience and commitment to keep them within the profession. The most effective teachers are committed to their students as people and as learners and they are incorporating all sorts of technologies into the teaching and learning activities that they design for their students and they are opening spaces where students can surprise them with what they can do with technology. Despite the simulation of care that the voice of ‘Siri’ from an iphone might give, I don’t see any time soon when ‘My teacher is an app…’ will be any kind of solution to the complexities of contemporary education.

Susanne Gannon is an Associate Professor of Education at UWS. She is Director of Programs for Adult and Postgraduate Education and Academic Course Adviser for the Master of Education (Leadership). She loves her ipad and is a great fan of many apps including i-university, however she is increasingly annoyed by the spamming of her faculty by emails marketing poor quality apps for teacher education, and remains wary about the erratic quality of many educational apps. She is involved in a number of ICT innovations in teacher education, including the national Teaching Teachers for the Future Project and a laptop trial project at UWS. She is a member of the SSERP project team at UWS. She is also the current editor of the journal English in Australia, the next issue of which focuses on English teaching and new technologies (guest edited by Kelli McGraw and Scott Bulfin). The views expressed in this opinion piece are entirely personal and do not represent those of the School of Education at UWS.

“Now that NAPLAN is over I can start to teach?” June 12, 2011

Posted by Editor21C in Directions in Education, Education Policy and Politics, Primary Education, Secondary Education.
Tags: ,
1 comment so far

from Dr Robyn Gregson

In her first post, Robyn Gregson reviews international approaches to testing student learning and argues that NAPLAN* testing in Australia is having a negative impact on pedagogy and assessment, and contributes to a de-professionalising of teachers in schools.

“Now that NAPLAN is over I can start to teach?” is the cry of many that has been heard in the corridors and staffrooms of both primary and secondary schools.

Mirroring the USA and UK experience, Australian education reform has been driven by political agendas that seek to assign accountability for educational outcomes. Subsequently we have the introduction of National and International testing that will provide comparisons in learning outcomes across states and countries (Perso, 2009). Three such tests are the Programme for International Students Achievement (PISA), Trends in International Mathematics and Science Study (TIMSS) and Progress in International reading literacy Study (PIRLS). PISA tests reading literacy within ‘real life’ settings whereas TIMSS focuses on mathematics and science curriculum-based proficiency benchmarks (Kell & Kell, 2011). PIRLS is a comparative study of the literacy skills of 4th graders. TIMSS and PRILS are grade based while PISA is age based.

NAPLAN (National Assessment Program –Literacy and Numeracy) is the Australian version for Years 3, 5, 7 and 9, introduced in 2008. This test was intended to provide valuable information about basic skills, what students know and don’t know and what teachers need to focus on in their classrooms (Anderson, 2009). The data from this national test was to be used to support educational planning for individuals, classes, and schools, and would inform school systems and the wider community about how schools in their local areas compared. While the PISA, PIRLS and TIMSS provide interesting data they have not led to significant changes at classroom level. However NAPLAN is linked to the myschool website, and funding.

Since the introduction of NAPLAN, pedagogical practices in classrooms have been driven by the need for students to do well in the tests. Schools, teachers and students are being judged by the levels that students attain. Teachers are torn between the use productive pedagogies and authentic assessment that support academic progress, and with preparing their students for high stakes national testing. Teaching to the test is longer just a concern, but a reality (Luke & Woods, 2007). A study of the views of teachers by Dimarco (2009) reports that teachers are using their professional judgment to support student success in national testing that has led to a narrowing of the curriculum, teacher deskilling and attrition, corruption of testing procedure and test scores with no evidence that that the testing has led to improve student learning outcomes.

What has become apparent is that teachers are tailoring their teaching and assessment practices to match those of the national tests. There is concern about keeping students interested and engaged while preparing them for the testing. However there is much debate over the effectiveness of such practices. Teachers are concerned by the negative effects such as student and teacher stress, disaffection of curriculum, narrowing of curriculum and a shift from higher order skills to lower order forms of literacy.

Until recently research literature reported that a more positive relationship between pedagogy and assessment had developed because of the shift from assessment of learning to assessment for learning. In the latter, the focus of assessment was on helping students to learn from assessment as well as use the feedback given to improve not only what they know and understand, but how they learn. While educational research focuses on the benefits of constructivist and emerging 21st century theories of learning, the reality of many classrooms in both primary and secondary schools is that teachers are not utilising the types of pedagogy and assessment tasks that promote learning (Tierney, 2006).

What has emerged from recent literature is the destabilsation of the teaching profession with concerns about teacher motivation and engagement of students. The role of national testing is currently under surveillance with anecdotal evidence suggesting that teachers are yet again changing their teaching and assessment practices to align with national testing strategies.

References:   Dimarco, S. (2009). Crossing the divide between teacher professionalism and national testing in middle school mathematics. Australian Mathematics Teacher; 65 (4) pp.6-10.   Kell, M. & Kell, P (2010). International testing: measuring global standards or reinforcing inequalities. The International Journal of Learning, 17 (12) pp 293-306.   Luke, A. & Woods, A. (2007). Accountability as testing: Are there lessons about assessment and outcomes to be learnt from no child left behind? Literacy Learning: The middle years, 6 (3), pp.11–19.   Perso, T (2009). Cracking the NAPLAN code: numeracy and literacy demands. AMPC 14(3) pp.14-18.   Tierney, R.D. (2006). Changing practices: influences on classroom assessment. Assessment in Education 13(3) pp.239-264.

* To source other posts about NAPLAN on 21st Century Learning use our search engine at the top of the page.

Robyn Gregson is a Lecturer in Science education and  literacy for learning in the School of Education at the University of Western Sydney, Australia. She teaches in our Master of Teaching (Secondary) program.

Does NSW still need a School Certificate in the 21st century? September 13, 2010

Posted by Editor21C in Directions in Education, Education Policy and Politics, Secondary Education.
Tags: , ,
3 comments

from Professor Margaret Vickers

In this post, Margaret Vickers examines the changing context of secondary education in NSW and the potential for the recently announced review of the NSW School Certificate to modernise the secondary school accreditation.

The basic architecture of the NSW system of secondary education was established almost fifty years ago, and some of its key features are still with us today. One of these features is the Year 10 School Certificate. Under the Wyndham plan, NSW established a four-year program of comprehensive education leading to the School Certificate, followed by a two-year academic program designed to prepare a talented minority for the Higher School Certificate and University admission. Despite the huge economic and social changes of the past five decades, NSW still divides secondary education into two phases and concludes the junior phase at Year 10 with state-wide formal assessments. In contrast, between 1968 and 1985, every other state in Australia abolished the Year 10 certificates.

The reasons behind this nation-wide shift away from Year 10 qualifications reflect major shifts that can no longer be ignored, and it is possible that NSW will soon find itself moving into closer alignment with the other states. In May this year a Review of the School Certificate was announced by the NSW Board of Studies, and its terms of reference open up all sorts of possibilities. In every state and territory across Australia, the school leaving age has already been raised. National targets have also been set, proposing that 90 percent of all young people should complete Year 12, or an equivalent qualification such as an apprenticeship. From 2010, young people in NSW will also be required to remain in school or in some form of structured training until they are 17 years old.

This creates a contradiction. Since the average age of a NSW student at the end of Year 10 is 15 years and 9 months, most students will now be staying on long after they have completed the School Certificate. However, a considerable number of these young people may not complete Year 12. What will they do, between the end of year 10 and the day they leave? What will they gain in return for the additional months they are staying on? It seems that there may now be a need for an exit qualification that recognises what students have actually accomplished at the time they do leave school.

Several other factors have also contributed to the need for a new approach. First, with the NAPLAN in place at Year 9, there is less justification for a formal academic assessment at the end of Year 10. Second, young people are seeking greater flexibility in the timing of their studies, with some undertaking senior academic or vocational courses in Year 10. Thus, the boundary between the junior and senior phases is being blurred. Third, there is a growing trend towards the recognition of out-of-school achievements (in employment or community service) as part of a new certificate of school achievement. While there will be no change to the School Certificate in 2010, all sorts of possibilities are opening up for the years ahead. If you want to read more about this, or participate in the debate about the future, go to:  www.boardofstudies.nsw.edu.au/schoolcertificate/sc_review

 

Margaret Vickers is Professor of Education in the School of Education and Centre for Educational Research at the University of Western Sydney, Australia. She specialises in secondary education policy and post-secondary education pathways.

The dilemma of computers in schools July 22, 2010

Posted by Editor21C in Directions in Education, Early Childhood Education, Primary Education, Secondary Education.
Tags: , , ,
1 comment so far

From Dr. Joanne Orlando

In this post, Dr Orlando reflects upon the considerable investment Australian governments are making in placing computers in Australian schools, and suggests that the provision of the hardware is just a first step if we are to see a positive impact on student learning.

 

If you spent millions of dollars putting computers into schools, what do you think would happen? Do you think students will achieve better results in their learning? Do you think teachers will change their practice and adopt new ways of teaching? Do you think that governments will change the teaching and learning they expect to take place in schools?

Computers have been associated with developing the 21st century skills students will need for the future workplace. Late last year the Federal government implemented the Digital Education Revolution into Australian schools. This four-year initiative was the government’s response to capitalising on the benefits that computers potentially hold for student learning. In NSW this initiative has a budget cost of over $400 million (Federal government funding) and $25 million (NSW State government funding). It focuses on providing every Year 9 student and teacher in every Government secondary school in Australia with a laptop for the next four years.

Research to date however suggests that initiatives which focus primarily on resourcing schools with computers are not successful. Placing computers in schools does not seem to enhance student learning and achievement.

Teachers’ lack of performance or unwillingness to change is often given for as the reason for lack of success of initiatives which focus on resourcing schools with computers. This is however a one-dimensional response to a complex and important issue. It also does not provide insight into how we can actually make the most of using computers to enhance student learning and meaningfully prepare our students for the future workforce.

Schools are multifaceted and complicated environments. Expecting one new addition (computers) to change schools and raise standards is simplistic and unhelpful. Considering the complexity of the environments that these computers are being placed in will give us greater insight into reflecting on why teachers choose to use computers in their teaching in the ways they do (or don’t).  It will also support signposting where future planning might take place to facilitate future advancement and growth of the use of computers in schools that will support reaching the goal of enhancing student learning.

For example, a current and significant issue for schools and teachers is that while the government is resourcing schools with laptops they are simultaneously increasing pressure on teachers in terms of accountability. This is evident with the publication of league tables in high profile newspapers and the MySchool website. This accountability is linked to nation-wide standardised test scores. There is sufficient research to show that standardised tests encourage teaching which is traditional and teacher-centred. This of course is at odds with the creative software available on school computers and the use of these for the development of 21st century skills. Add to the mix other significant factors such as the diverse student abilities and experiences, and recent and significant changes to the curriculum.

The resourcing of schools with computers is a substantial beginning step for enhancing student learning. A second step now, if we really aim to use computers to their full potential to support student learning, is to contextualise computers within the schools and classrooms where they have been placed. Let’s not rely on the rhetoric about what computers are able to do. We need to shift our focus to thinking about and planning for how computers might be used in ways that are meaningful to our students and our teachers within today’s schools.

Joanne Orlando is a lecturer in Early Childhood Education at the University of Western Sydney, Australia.

%d bloggers like this: