Introduction
Expert evaluation was embedded throughout the Space Awareness project to rigorously test its effectiveness and identify its impacts on key target audiences, especially school teachers (and their pupils). The evaluation effort thus explored whether resources and activities were relevant and applicable to the various target audiences, including identifying key strengths and impacts.
Our approach
The Space Awareness evaluation consisted of a multi-pronged strategy to both investigate the immediate impacts of the project activities, as well as create a baseline and follow-up opportunity for exploring longer-term impacts. Importantly, the evaluation framework was based on a set of common agreed intended outcomes for the various project activities. Much of the evaluation data collection was conducted by the project partners and dissemination nodes as they distributed the Space Awareness resources to local teachers and other educators, based on protocols and tools developed within the evaluation framework. The combination of a centralised framework and distributed data collection has resulted in consistent and comparable data collected from across Europe and beyond regarding activity impacts in areas ranging from the educational resources to face-to-face teacher training workshops to the Massive Open Online Courses (MOOCs). Additionally, a complementary qualitative fieldwork strategy, combined with dedicated online efforts and an international teacher competition, allowed a more in-depth exploration of specific issues within the final year of Space Awareness. For example, through these more qualitative approaches we were able to better explore the impacts of the Space Awareness tools on specific target groups such as girls, ethnic minorities and those from socioeconomically disadvantaged backgrounds. Teachers and other educators were also provided with their own assessment tools, consisting of both formative and summative techniques to help assess pupil learning in engaging and informative ways.
An important aspect of the evaluation framework involved the implementation of two large-scale Europe-wide surveys which enabled a baseline measurement of existing attitudes and opinions relating to space science. Covering both current pupils’ and existing space scientists’ experiences and perspectives, these surveys have for the first time allowed specific investigation of aspirations and influences relating to space science careers.
A further substantial achievement within the evaluation effort was implementing a longer-term strategy, involving inviting educators to register their contact details for potential involvement in follow-up research at a later date. Invitations to participate were distributed at every stage of the evaluation data collection processes, resulting in a broad pool of registered teachers and other educators that will enable dedicated longitudinal follow-up to occur in due course (subject to appropriate funding). This is a rare and highly unique feature of the Space Awareness programme.
Further details regarding the various evaluation tools, protocols and analyses are contained within the individual activity reports indicated below, including copies of all surveys and other documentation used during data collection.
The key documents here focus on the top-level contents of each report. For further details please click on the links associated with each report to access the full versions.
Key Documents
This official project Deliverable outlines the main findings of the Space Awareness project evaluation. The evidence collected in all the other individual activity reports is synthesised in order to identify over-arching reach and participation patterns, perceived quality, and the project’s success in achieving the specified Space Awareness intended outcomes. For the benefit of future similar projects, key elements of success and the main challenges are outlined, leading to both strategic and practical recommendations relating to future policy and practice in space science education within Europe and beyond. Additionally, we hace prepared a shorter Executive Summary document for D7.2 for those who are interested in quickly reiviewing the headline findings.
Space Science: The view from European school students
A survey of European pupils’ views of space science was conducted in 2016 and early 2017 as part of the Space Awareness project. This summary report provides an overview of the survey and its main findings, particularly with regard to age, gender and country. Additional details of the data and analyses are included in a technical annexe.
Looking Back: Space scientists’ reflections on career influences, paths and choices
Within this survey participants were asked to reflect retrospectively on what influenced their decisions relating to space and technical career choices. A total of 415 individuals working in space science from over 30 countries responded to our online survey, representing a wide range of backgrounds and roles. This report summarises the key findings, and pays attention to the timings and initial influences of respondents’ interest in space science, as well as how that interest was maintained. There is also a section highlighting key patterns within their reported career paths, including how their career interests developed, and what choices and priorities were influential.
Inspiring the Educators: Summary of feedback relating to the teacher training workshops delivered by individual partners and nodes
34 teacher training courses were delivered between September 2016 and June 2017. Within this document the data collection and analysis procedures are explained, utilising both participant and workshop organiser perspectives, and focused mainly on questionnaire surveys. The main results are delineated according to the courses delivered, participant backgrounds, course ratings, outcomes achieved, and any evidence of the skills developed within the courses being embedded into the participants’ ongoing professional practice. Finally, a series of recommendations are offered in order to inform the development of future similar training workshops.
Deep Impacts: A summary report of the evaluation findings from the Space Awareness Massive Open Online Courses (MOOCs)
The evaluation of the MOOCs combined detailed quantitative surveys (distributed pre-, post- and after a 2-month delay in order to explore likely impacts from the course) with an in-depth qualitative analysis of a single course, Navigation Through the Ages. (Full details of the latter analysis are available in the Engaging Teachers report indicated below.) Respondents’ backgrounds and participation patterns are explored in depth within this report, including a remarkable 30% course completion rate (corresponding to approximately six times the international norm). The evidence relating to the very high perceived quality, and success in achieving all of the intended participant outcomes, is also presented. In particular, comparison of pre- / post- / delayed- survey responses enabled affirmation of distinct knowledge gains and key shifts in participants’ attitudes, values, skills and behaviours which appear to be directly linked to the Space Awareness project.
Engaging Teachers: Navigation Through the Ages MOOC: Summary of qualitative evaluation via participant observation
Participant observation was used within the Navigation Through the Ages MOOC to augment the survey data and collect qualitative evidence in relation to the intended learning outcomes. Evidence was synthesised according to the major Space Awareness Intended Outcome themes: Feel, Value, Understand, Do and Skills. These data augment the findings from the aforementioned Deep Impacts report by adding images, statements, reflections and narratives to better reflect the participant voice within the evidence presented.
Educate & Inspire: Summary of feedback relating to the Space Awareness educational resources
Feedback was provided on 43 separate educational activities via 212 separate responses to an electronic questionnaire representing an excellent breadth of contributions across central, eastern, northern and southern Europe. This report describes the data collection methodology as well as the respondents’ backgrounds. In addition to an overall rating, key quality indicators were used to evidence the respondents’ generally positive reception of the resources developed. Other indicator statements enabled evaluation of the project’s success in achieving the intended outcomes for both teachers and students. Finally, specific recommendations are identified in order to inform the development of similar future projects.
Winning Hearts and Minds: Highlights from pupil and teacher competitions run within the Space Awareness programme
This document presents inspiring case studies demonstrating the wonder, enthusiasm, and fundamental benefits to participants’ attitudes, values, knowledge, skills and behaviours as a result of their involvement in the Space Awareness project. Each easy-to-read case study presents a snapshot of what can be achieved by teachers and pupils across the world who were inspired by Space Awareness resources and activities.
Networking Space: Analysis of online and social media traffic relating to the Space Awareness project
Based on data collected using standard (free) analytics tools, the main trends regarding the Space Awareness website and Facebook traffic are outlined. This includes an overview of key visitor usage patterns and demographics, evidence of the growth in participation over the duration of the project, and the reach and traffic flow.
Voices of experience: Education practitioner conversations at the Mission to Mars Summer School Marathon Greece, 2 - 7 July 2017
This report is designed to synthesise and reflect upon the experiences of practicing teachers and educators from across Europe. It is specifically designed to help contextualise the wider evaluation of the Space Awareness project resources and activities. The Space Awareness project targeted educators as users of project activities and resources and the conduits of change. This report explores some of the broader patterns from the point of view of educators as practitioners. They are the experts on the context they work within, and their voices are valuable evidence of the issues at stake. The key findings consider what works inside the classroom, what are the key challenges faced by teachers, and what networks, resources and/or continuing professional development opportunities are sought in order to support teachers within their professional practice.
Finally, although much of the content is superseded by the above reports, it is perhaps also useful to share the mid-term official project Deliverable associated with the Evaluation Framework. It included a brief introduction to the Space Awareness project generally, as well as the Evaluation Framework more specifically. The Space Awareness Intended Outcomes were also introduced, along with the ethical considerations and formal approvals processes undertaken to ensure appropriate data collection and management. Finally, the initial Evaluation Toolkit was introduced, comprising a coherent and consistent set of tools and instructions to enable robust evaluation of the main project activities as they were implemented by nodes and networks across Europe and internationally.
Image: Space Awareness Evaluation in Numbers
Credit: Karen Bultitude
