Te Kotahitanga is a New Zealand school reform project aimed at improving the pedagogical contexts in mainstream classrooms in which the indigenous Māori students have traditionally been marginalised. It does this by assisting teachers to implement an Effective Teaching Profile. Part of this process uses an observation tool to monitor the degree to which participating teachers are incorporating the interactions and relationships described in the Effective Teaching Profile into their day-to-day teaching. Given the central importance of these tasks, the Te Kotahitanga team undertook to test the observation tool for measurement reliability and validity. In order to undertake this study, the team conducted synchronous observations amongst trainers (the Professional Development Coordinator and Regional Coordinators) to ascertain their level of consistency when using the tool. The team then conducted synchronous observations between trainers and 38 in-school facilitators in the 12 schools involved in Phase 3 of the project. In total 41 teachers were observed and over 200 Māori students were involved in these observations. This study suggests that the tool can produce consistent and reliable results when observers have been effectively trained.