Lesson Observation Data Analyst

Scope and Description of the Task

In recent years, Lebanon has become the country with the highest refugee per capita in the world. Officially, there are more than 1.5 million registered Syrian refugees in Lebanon, a third of which are school going aged children. In 2013, the Government of Lebanon opened a second-shift in its public education system to cater for the enormous numbers of refugee students. In 2017/2018, non-Lebanese students outnumbered Lebanese students in the public education system. The responsibility for educating all these students resides with the Lebanese Ministry of Education and Higher Education (MEHE) who, alongside external donors, launched the Reaching All Children with Education II (RACE II) programme in 2016.

The World Bank (WB) and Government of Lebanon (Gold), especially the Ministry of Education and Higher Education (MEHE), designed Support to RACE II (S2R2) to deliver on significant parts of RACE II as the strategy relates to the formal education sector. S2R2 is designed to complement other work developed to support RACE II, such as activities planned and implemented with UNICEF, UNHCR and UNESCO.

DFID Lebanon’s contribution to the education sector includes support to the multi-donor trust fund as well as the provision of a technical assistance team to MEHE. In March 2018, a three-year technical assistance project was awarded to Oxford Policy Management, which will expand an existing TA facility to serve additional MEHE priority areas to achieve the objectives of RACE II through MEHE’s Support to RACE II (S2R2) programme. The TA team will support the GoL to deliver the immediate objectives of the S2R2 programme while building the capacity, systems and processes which will underpin a medium-term sector development strategy.

Background: 

The assignment will provide support to MEHE’s Direction d'Orientation Pédagogique et Scolaire (DOPS) department to work on the format, method, analysis and utilisation of teacher lesson observation data collected by DOPS counsellors.

This data is currently collected using a number of different tools which have differing objectives and have been designed at various points through a number of donor-funded projects. DOPS is now considering whether to amalgamate these tools into a unified lesson observation tool with a single and comparable mode of analysis. The alternative is to draw the tools together under a single framework to complement one another as a coherent package of tools. The appointed consultant will analyse the case for unification of tools in terms of what is lost (eg. year-on-year comparability looking backwards) with the benefits (eg clarity for counsellors, single analysis platform etc). In any case, the final package of lesson observation tools would be trained upon, piloted and iterated for optimal reliability, validity, accuracy and ease of use.

Whether the tools are amalgamated or grouped, the appointed consultant will consult with the DOPS software developer to design, pilot and roll out a suitable interface for the collection of data from the lesson observation tool. This should make sure that the quality and quantity is data collected is sufficient while weighing up the data entry burden on DOPS counsellors.

The data collection platform should be designed in way that makes data visualisation, analysis and uptake as straightforward and useful as possible. The platform for analysis and visualisation should be the Visit Information Management System (VIMS) which has already been developed within DOPS. 

The specific tasks under this role include providing technical assistance to DOPS to:

• At the request of the Director of DOPS, review the current lesson observation tools used by DOPS counsellors
• Assess the functions of each lesson observation tool and whether they are fit-for-purpose individually or as a combination
• Design package of observation tools (creating links between existing tools or devising new one) depending on the objective of that individual observation
• Prioritise the highest quality datapoints within these observation tools
• Specify details of an online interface to enter this data into the VIMS database, striking a balance between making it ease-of-use with collection of high quality data
• Ensure the data from lesson observations can be used in:
  • future VIMS scheduling
  • detailed analysis of year-on-year trends (longitudinal analysis at single school)
  • school to school comparisons (cross sectional system analysis)
• Work with VIMS software developer to ensure that standard reports can be straightforwardly run and that new reports can easily be designed
• Work with VIMS software developer to design data access protocols for entry, users, super-users, administrators etc
• Provide detailed statistical analysis of lesson observations, including trends by location, socio-economic status of school or teacher profile
• Set out the most relevant reports to flow down from the lesson observation data
• Display lesson observation data in a manner that tracks trends and is east to assess trends over time and other prominent features, working closely with the VIMS software developer
• Design of appropriate feedback loops to improve teachers’ classroom practice as a result of individual or group analysis of lesson observation data

DELIVERABLES AND TIME FRAME:

Phase 1: 6 Days 

Desk-based assessment the existing lesson observation tools

Consult with DOPS counsellors on their experience using the tools

Observe these lessons in use in classroom contexts

Make recommendation to Director of DOPS on next steps for the tools

If necessary, redesign/realign lesson observation tools

Phase 2: 30 Days 

Design the data collection programme for existing/revised tools, including number of fields, estimation of time taken, modality of entry (tablets/smartphones etc) and the form of entry (quantitative vesus descriptive data entry).

Work with VIMS Software developed to ensure alignment  and building upon existing system features

Pilot the final lesson observation tools in schools, summarise feedback and make adjustment where necessary

Train DOPS counsellors on the underlying theory, specific content and entry protocols of the final tools

Roll out tools for use by DOPS counsellors and collect/correct and issues with its implementation

Phase 3: 4 Days 

Desktop assessment of current reporting abilities of VIMS

Design data reporting protocols and the format of final reports from point of initial tool finalisation

Quality assure reports can be generated as designed

Recommend any improvements to report protocols/ tool content

Set out procedures for robust statistical analysis of datasets when sufficient data has been collected

Design specifications for data visualisation dashboard to give useful real time information on the progress of lesson observations

The expectation is that the majority of time spent on the programme will be spent in the Ministry of Education and Higher Education working with counterparts in DOPS, or on field trips.

Profile and Skills: 

• Advanced university degree (minimum Masters, preferably a PhD) in education, evaluation, statistics, or other relevant field.
• Minimum of 10 years’ experience in quantitative/qualitative classroom evaluation methodologies and their analysis.
• Specific experience with extracting value from lesson observation tools.
• Demonstrates experience in supporting teachers to improve their practice based on collection of evidence and feedback models.
• Expertise in the design of suitable data collection methods, including use of technology to streamline the burden of doing so
• Experience working with technology to collect large datasets, visualize them and run appropriate reports
• Understanding of data storage principles to allows for evaluation of aggregated data from a wide variety of viewpoints
• Fluency in Arabic and English. French is advantageous.
• Organizations and individuals are invited to apply

Working Relationship:  The consultants will work in close coordination with the Director of DOPS and build on existing practice within DOPS, especially drawing on the VIMS. The Team Leader OPM-TA is responsible for day to day management of the consultants and will oversee their deliverables.