Students’ Stellar Gains from Maths Invaders Online
Introduction – Maths Invaders Online, Adaptive Learning Mode
Maths Invaders Online is an online learning website published by EdAlive, marketed to school and home users in Australia and some other countries, to support instruction in mental arithmetic across school years 1-10 maths curricula. The content is split into 11 curriculum areas: counting, addition, subtraction, multiplication, division, fractions, decimals, percentages, squares and square roots, directed numbers, and powers. Not all areas are included within a year level while other years with larger content have two and occasionally 3 subsets of a curriculum area. Within a year the units are ranked by difficulty. There are in total 118 units of work and, with an additional level each year reserved for revision (not used in this data), a grand total of 128 units or levels each mapped to the curriculum. In excess of 350 000 questions are behind these levels.
Maths Invaders has four modes of usage for students, two of which employ the Automatic EdAlive Adaptive Learning system to continuously monitor individual performance to determine the range of levels sampled for the next set of 12 questions to be presented. Adaptive Learning is used in the “Questions Only” mode and as the vehicle for question presentation in the “Space Rescue” game embedded within Maths Invaders Online. The EdAlive Adaptive Learning system was launched in August 2019.
Using the EdAlive Adaptive Learning system there are four possible outcomes for a question – No Attempt, Correct, Skip or Incorrect. No Attempt is recorded when a user runs out of time on a question set (12 questions in 40 seconds). No Attempt also occurs when a user stops using – due to the end of class time or possibly leaving a game when they are dissatisfied with it. Students can elect to Skip a question at any time. While there are observed instances of Skip being used indiscriminately, generally it is applied to more difficult questions – students learn to Skip questions which will take more time to solve, so that they are less likely to exhaust the time allowed for the set. Overall the frequencies of the four question outcomes are: No Attempt 21%, Correct 47%, Skip 24% and Incorrect 8%.
Anecdotal evidence gathered from class visits and ad hoc inspection of the Maths Invaders Online Adaptive Learning internal reporting has indicated that students using the system make strong gains in maths skills. This study aimed to confirm that observation and quantify any student improvements in maths performance with Maths Invaders Online in Adaptive Learning mode.
By mid-February 2020, EdAlive had accumulated some 4 million anonymised question records from Maths Invaders Online adaptive use. 1899 users had completed at least 300 correct answers, and some had exceeded 3000, this selection covering almost 2.6 million question records. These data were further refined for analysis by restricting the starting maths level to school Year 4 or above, up to the highest performing students, who were in the Year 8 range. This reduced cohort displayed a more homogeneous response rate amenable to a pooling of data. These data covered 1.5 million question records, representing 1227 individuals with at least 300 correct answers completed in Maths Invaders Online adaptive mode.
A small degree of data editing occurred where a student answered questions at a level more than 4 school years below previous performance. Less than 0.5% of Correct answer records were excluded by this means, affecting less than 0.25% of users.
For each user admitted to the analysis, the sequential data was commuted into blocks at the completion of each 100 correct answers. This was applied regardless of session breaks or longer intervals (for example school holidays). Records for each block included the number of questions in each of the four outcome categories and the average adaptive level applying to each category. An estimate was also made of the screen time required by the user to complete the block. This required an arbitrary decision about the length of break between question sets as to what constituted a pause (including the game play phase of Space Rescue) and what was discontinuation of use. The final block for each student comprised 100 to 199 correct answers so as to avoid small data blocks.
It was expected that students would take some time to adjust to the online environment, and before their true maths ability is displayed – a “burn-in” phase as the user gains familiarity with the interface, different styles of question and factors such as time limits. While the length of burn-in phase could be debated, it was soon clear that Maths Invaders Online performance was climbing steeply in the early stages and so no immediate distinction could be drawn between burn-in and educational gains. At this stage the first 100 correctly answered questions was used to set the starting level for a student.
Age data was incomplete, with 62% of 1899 individuals having an inferred class age based on the class level reported by the teacher, including composite classes. The most common age was 11 (32%), followed by 10 (24%), 9 (18%), 12 (17%), then 8 (3%), 13 (3%), 7 (2%), and small numbers of 6 and 16-year-olds. The average class age was 10.4 years.
Figure 1. Maths performance benefits from effort in Maths Invaders Online
Figure 1 shows the average performance of 1227 individuals with at least 300 correct answers completed in Maths Invaders Online adaptive mode, reducing to 109 individuals with 1100 correct answers. As the quantity of data continued to decline sharply, any further extent of the curve was considered unreliable. Of these students, 62% had age information, with the average class age of 10.7 years, and the range covering 6 to 16 years.
The graph depicts the average adaptive level of correct answers for all students. Expressed in maths age, the gains exceed 12 months after 10 data blocks of 100 correct answers – each with an estimated screen time of 25.5 minutes.
Figure 2. Advances in maths levels without loss of accuracy.
Figure 2 additionally shows the average accuracy of answers across all students for each block, as Correct/(Correct + Incorrect), charted on the right hand axis. This measure was highly consistent across the progression – as questions were being asked at a more difficult level, student accuracy was maintained within a narrow range.
The study has demonstrated that Maths Invaders Online has a positive effect upon arithmetic ability, and this has been shown to be an effect of considerable value. If a 30 minute session was conducted weekly for one 10-week school term, then gains in maths age of 12 months have been achieved; compared to the expected rate via conventional instruction of 12 months’ gain over 4 school terms.
Importantly, the research shows that using Maths Invaders Online in Adaptive Learning mode is beneficial across all ages and maths abilities – from Year 1 to Year 10. Students make gains whether they are already performing at average, below average or above average level for their age. Students who have suffered a setback now have a realistic chance to catch up to their peers. Early signs are that students starting at Year 1 to Year 3 maths level can make even greater progress – this research will be presented in the near future.
The rate of Maths Invaders Online usage in the data varied widely from occasional short bursts to several hundreds of questions per week. Overall this wide range had no discernible impact on the correlation between volume of work and gain in maths age. As more data becomes available from a longer timeframe, further light may be shed upon optimal usage patterns.
Automated Adaptive Learning is strongly suggested as the key driver of this progress. Student performance is continually assessed, and their individualised need for reinforcement controls the rate of progression. Students in need of greater reinforcement at a level will have it delivered so they are not overwhelmed by subsequent units of work and importantly, their rate of progression is not directly exposed to peers. Students able to grasp a concept more readily are allowed to progress without boredom. For both of these groups, they will remain engaged with the content for longer, building not only their maths ability but crucially their confidence.
As much as Automated Adaptive Learning is a benefit to students, it is also a huge help to teachers, much reducing the need to set work according to individual needs. This will be a yet greater benefit for teachers in small schools and composite classes, even more so because students can play Space Rescue on an even footing whatever their maths level. As students consolidate and master their learning at all levels, they will be more receptive to future classroom instruction and responsive teachers will be able to factor this into their lesson plans.
Many students, teachers and parents become nervous in the face of NAPLAN, but the evidence here suggests that strategic use of Maths Invaders Online can give students greater ability and confidence in their ability. Far from creating a hothouse environment, Maths Invaders Online will contribute to calm and productive students who enjoy learning.
This research and analysis was completed by Dr Lindsay Brash.