Learning analytics
The aim was to find out which activities the children performed and how these were related to competencies. It was important for us – accordingly to John Hattie – to make the learning of the children visible. (visible-learning.org)
In order to do so, we had to find a way to track the children’s activities. As the project proposal suffered budget cuts that concerned mainly the IT equipment, we had to refrain from the original idea that most of the activities would happen online and we had to find a way to track tasks that were performed offline.
We came up with the decision to deliver the learning material as well as the assignments in the form of “tasksheets” and print a QR-code on each sheet.
Hand-it-in-App
The Hand-it-in-App is an online application that allowed children to “hand in” their work by scanning the QR code of the related task sheet.
When following the QR code on a specific task sheet, kids reached a page that contained the digital version of the task. In the user interface, the children could identify themselves by choosing their personal icon (sticker). Then they could provide results (answer questions, upload pictures, provide links to videos), write feedback and tick a smiley to indicate how much they liked the task.
Competency framework
We decided to use an existing competences framework instead of inventing a new one.
As the boxes covered quite a large variety of topics and transversal skills, we chose the “21st century skills” framework of the University of Melbourne (http://www.atc21s.org/, http://www.p21.org/), which is very general and holistic. We liked its approach to not only include knowledge but also skills and attitudes. Our boxes covered about two thirds of its categories, and we added six more to cover all our boxes.
The 15 most often used competencies were:
1. Follow instructions (e.g. tutorial video or instruction manual)
2. Develop, implement and communicate new ideas to others effectively
3. Infer, query evidence, conjecture alternatives and draw conclusions
4. Use technology as a tool to research, organize, evaluate and communicate information
5. Improve handicraft skills
6. Analyze how parts of a whole interact with each other to produce overall outcomes in complex systems. Examine ideas, identify and analyse arguments
7. Ability to concentrate for extended as well as short periods of time
8. Use various types of reasoning (inductive, deductive, etc.) as appropriate to the situation
9. Be open to non-familiar, unconventional and innovative solutions to problems and ways to solve problems
10. Confidence when speaking in public
11. Prioritize, plan and manage work to achieve the intended group result
12. Logical thinking (programming) – formulate algorithms in various forms
13. Monitor, define, prioritize and complete tasks without direct oversight
14. Know when it is appropriate to listen and when to speak
15. Work effectively in diverse teams
However, as not all the tasks were actually performed (the locations were free to select which tasks they wanted to offer), the list of actually addressed competencies was smaller. The top 15 of this list are shown in the following table. Qualitative feedback of teachers confirmed an overall increase of these competencies in their groups.
The 15 most addressed competencies in locations:
1. Ability to read and understand different texts, adopting strategies appropriate to various reading purposes (reading for information, for study or for pleasure) and to various text types
2. Sound knowledge of basic vocabulary, functional grammar and style, functions of language
3. Ability to concentrate for extended as well as short periods of time
4. Know a wide range of idea creation techniques (such as brainstorming)
5. Know when it is appropriate to listen and when to speak
6. Speak with clarity and awareness of audience and purpose. Listen with care, patience and honesty
7. Act responsibly with the interests of the larger community in mind
8. Follow instruction (e.g. tutorial video or instruction manual)
9. Analyze how parts of a whole interact with each other to produce overall outcomes in complex systems. Examine ideas, identify and analyse arguments
10. Be open to non-familiar, unconventional and innovative solutions to problems and ways to solve problems
11. Improve handicraft skills
12. Ask significant questions that clarify various points of view and lead to better solutions
13. Be open to new and worthwhile ideas (both incremental and radical concepts)
14. Create new and worthwhile ideas (both incremental and radical concepts)
15. Develop, implement and communicate new ideas to others effectively
Evaluations
As we knew for each task which competencies it addressed, we could easily produce charts that showed an overview of student’s addressed competencies, like the one in the following table.
The other evaluation shows which tasks were already done by which students:
This helped the teachers to keep an overview of what was done and showed students how much they have already done and how much they still could do. (Sometimes these sheets were put on the wall.)
Limitations
Although this system of tracking activities and evaluating competencies theoretically worked quite well, it showed some major weaknesses:
• Incompleteness of data: All three locations often had to deal with unstable or unavailable internet. If the hand-it-inapp was not available, then the teachers would have to manually take notes and enter the data later, as soon as the internet was available again. In this process, data was often lost.
• Too little added value: In the end, the evaluations did not produce enough added value to justify the additional effort of the hand-it-in-app (for teachers as well as for students) – especially under weak internet conditions (see above). We had several ideas how to improve the visualization of the learning progress in order to improve student’s motivation but did not have time to put it into practice.
Due to these limitations it was not possible to meaningfully link the data with the rest of the evaluation. However it strengthened the interpretation of the other outcomes.