The online questionnaire to school principals included questions on three sets of variables: background characteristics of schools, outcome variables, intermediate processes and inspection measures. The questionnaire can be found here: Survey to principals (english version).
Background characteristics of schools
The questionnaire starts with a number of questions on background characteristics of schools and principles that are expected to be relevant for responses of schools to school inspections, such as the location of the school in a rural or urban area, the composition of the student population, the experience of the principal and his/her tasks, and resources in the school. Items on the background characteristics of schools were inspired by items from the TALIS and PEARLS surveys. The items were adapted to fit the context and variables of this study.
Outcome variables
The second part of the questionnaire contains questions on the outcome variables of our study: capacity to improve, effective school and teaching conditions and unintended consequences of school inspections. Questions about the first two variables are framed in terms of the time principals have spent during the previous academic year to improve the school’s functioning in these areas (using a 5-point scale ranging from ‘much less time’ to ‘much more time’), as well as the school’s status and functioning on these variables (5-point scale ranging from ‘strongly disagree’ to ‘strongly agree’).
Our outcome variable ‘capacity-building’ refers to the school’s capacity to enhance the professional learning of teachers and to transform large-scale reform into accountable student-oriented teaching practices. A school is thought to be capable of change if it is competent to implement innovations initiated either by the government (or the Inspectorate of Education) or by the school itself (as a result of the outcomes of self-evaluation), and if necessary, to match both types of innovations. In this context, Geijsel et al. (1999) stress the importance of the school as a learning organization which consciously attempts to expand its learning capacity to optimize its effectiveness (Geijsel et al., 1999). Participation of teachers in decision-making, cooperation between teachers, and transformational leadership are considered to be important conditions of learning organizations and are therefore included as sub-variables in our study.
Items to measure these sub variables were inspired by the Dutch School Improvement Questionnaire (see Geijsel et al, 2009). The questions were reworded to also ask schools about the extent to which any changes were made in these variables during the previous academic year.
The second outcome variable of ‘effective school and teaching conditions’ refers to the effective practices on the school level related to school organization and management, including educational leadership, a productive climate and culture and achievement-oriented school policy. These conditions are expected to contribute to and facilitate effective teaching and instruction and as a result lead to higher student achievement. Teaching/instruction conditions include what a teacher does to create effective learning environments and to boost learning (Scheerens et al., 2009). Opportunity to learn and learning time, achievement orientation, clear and structured teaching, a Challenging teaching approach and an orderly learning environment are considered to be important aspects of effective school and teaching conditions and are therefore included as sub-variables in our questionnaire.
Items measuring opportunity to learn and learning time, achievement orientation and an orderly learning environment were inspired by Scheerens et al (2010); items measuring clear and structured teaching/clear and activating instruction, a challenging teaching approach/clear and activating instruction and an orderly learning environment were adapted from the ICALT questionnaire which was developed by Inspectorates of Education in several European countries to measure the quality of teaching and learning, using a shared framework of indicators.
The variable ‘unintended consequences’ included questions on the extent to which school inspections lead to a narrowing of curricula and instructional processes in the school, the extent to which principals experience inspections as an administrative burden and manipulate documents and data they send to the Inspectorate. These questions are only administered to schools that received an inspection visit in the previous year. Questions on intermediate processes were inspired by the the NfER survey ‘evaluation of the Impact of Section 5 inspections’ (2007). Questions were also inspired by the questionnaire for teachers to evaluate educational board/school inspectorate (..). Questions from both studies were adapted to fit the context and variables of this study.
Intermediate processes
The third part of the questionnaire included questions about the intermediate processes that precede our outcome variables: setting of expectations, acceptance and use of feedback, promoting self-evaluations, choice/voice/exit of stakeholders.
- Setting of expectations refers to the extent to which schools use the criteria and descriptors set out in the inspection frameworks to define their own standards of a ‘good school’ and to incorporate these standards in their daily work.
- The acceptance and use of feedback refers to the feedback provided during or as a results of inspection visits and the school’s acceptance and use of this feedback for school improvement.
- Promoting self-evaluations relates to the implementation of internal systems of evaluation and self-review as a result of school inspections.
- Choice/voice/exit of stakeholders includes questions about actions of stakeholders (such as parents, local policymakers or school boards) to use the standards and the feedback of the Inspectorate of Education (as described in the public report) to voice their opinion to the school, to choose a school or even to move students from a poor school.
These questions are only administered to schools that received an inspection visit in the previous year. Questions on intermediate processes were inspired by the NfER survey ‘evaluation of the Impact of Section 5 inspections’ (2007). Questions were also inspired by the questionnaire for teachers to evaluate educational board/school inspectorate (..). Questions from both studies were adapted to fit the context and variables of this study.
Inspection measures
The fourth and final part of our questionnaire included indicators on the frequency and types of inspection visits that have taken place in schools, the methods used by Inspectorates of education to collect information (e.g. through lesson observations, interviews, document analysis), the standards used to assess schools, the feedback provided to schools, the threshold used to come to an overall assessment of the school, the consequences of school inspections (rewards, sanctions, interventions), the type of reporting of the inspection assessments (e.g. in public school reports or through league tables) and potential additional tasks of school inspectors (e.g. in managing or governing schools). We also included a number of questions on the school principal’s general satisfaction with inspection measures. These questions were inspired by the the NfER survey ‘evaluation of the Impact of Section 5 inspections’ (2007). Questions were also inspired by the questionnaire for teachers to evaluate educational board/school inspectorate (..). Questions from both studies were adapted to fit the context and variables of this study.
