Executive Summary (Click here for the full report)

This executive summary provides a high-level overview of the Billy Graham Youth Foundation (BGYF) boxing program evaluation. The BGYF’s vision is to develop vibrant, independent and responsible youth who contribute to their communities. The program originated at the Naenae Boxing Academy and in recent years has been expanded to Cannon’s Creek and Ashburton. As the program and the BGYF have gained momentum and aspire to expand further, there is a need to formalize the program to ensure high quality, fidelity and consistency across present and future sites. To this end, the BGYF has commissioned an external evaluation led by Dr. Barrie Gordon at Victoria University of Wellington.

Because the boxing program is framed as a values-based program with the intention of developing character and teaching life skills, the evaluation was conducted with a primary focus on these elements of the program as compared to sport performance. Therefore, the theoretical framework of the evaluation was rooted in the literature from positive youth development, social and emotional learning, as well as the SAFE (sequenced, active, focused, and explicit) guidelines for best practice in youth programs. Special attention was also paid to issues of culture and diversity in the planning of the evaluation, data analysis and reporting.

The evaluation design was comprehensive, involving multiple data sources and a case study approach to assessing the three program sites (Naenae, Cannon’s Creek and Ashburton). Participants in the evaluation included youth (boys and girls) as well as a range of adult stakeholders (coaches, program managers, board members, police, parents, and other community members). Data sources included interviews with adults, focus groups with youth, systematic observation of boxing lessons, field notes, document review, valid and reliable youth surveys, and a review of pre-existing testimonials from parents and youth participants. Data sources were analyzed separately and then triangulated with each other to identify common patterns, key strengths and areas of opportunity. These are addressed in the following section as key findings, organized around the key focal points of the evaluation.

It should be noted that the exploration of longitudinal outcomes through the Integrated Data Analysis (IDI) system in association with the Ministry of Social Development has not been completed at the time of presenting this report. There were significant challenges in identifying quality information on participants who have attended the Naenae academy over the last twelve years. This information is now available and the results of the IDI analysis will be available in 2018.

Key Findings

Clear Focus on Values

A review of program materials, documents and testimonials from youth participants and parents indicate the program is clear and consistent in its commitment to values education through sport as opposed to competitive boxing. Perceptions and experiences in the programmes appear strongly positive and consistent with the stated values and mission. The challenge facing BGYF is how to replicate, in a consistent fashion, programs at new sites with new coaches, etc.

Relationships

Relationships are at the center of the NBA model. These include both internal and external relationships. All the data supported the positive nature of the variety of relationships established by the three Academies. The relationships between the participants and coaches are crucial and the data was unequivocal that these were positive and highly valued by all involved. In the PSRQ survey, for example, an overwhelming percentage of participants identified that they either very- strongly agreed or strongly agreed to the statement that they respected their coaches. Not one participant from across the three Academies disagreed with this statement.

Success at Multiple Levels

As the boxing program’s evolution has been based in the values and personal commitment of an individual, the approach has been quite organic. The program was not conceived or designed to generate specific outcomes such as violence prevention or academic achievement. On the contrary, stakeholders define and report success in a number of areas. Many of these success appear to emerge very early for participants. For example, many stakeholders report immediate improvement in self-confidence, physical fitness, boxing skills, friendships and understanding program values. With continued attendance (often prompted by the aforementioned successes), stakeholders report youth beginning to transfer the attitudes, values and life skills learned in the program to other areas such as home, school and community settings (e.g. more positive attitudes about police and staying in school). A high proportion of the adult stakeholders see and/or hope for more longitudinal success stemming from participation. For example, many believe that lessons learned and changes made as a result of this program enable youth, families and possibly the broader community to break free from negative cycles and patterns to enjoy more positive life outcomes (e.g. education, health, employment). The breadth and depth of perceived benefits reflect extremely well on the program. However, as the program moves forward and expands, it will be important to identify some common metrics that will enable the BGYF to set expectations for quality, assess program improvement efforts, and share concrete outcomes with current and future funders and supporters.

Strong Foundation for Positive Youth Development

To balance the qualitative data describing perceptions and beliefs about the program, other measures were used to quantify the coaching strategies and youth behaviors observed in the lessons as well as the youths’ self-reported experiences. Systematic observations demonstrate that coaches consistently model respectful behavior, set clear expectations and structure lessons in ways that allow youth to experience personal success and interact socially. Correspondingly, youth are consistently participating with high levels of engagement, demonstrating respectful behavior and cooperating with others. While these observations and survey results indicate that the program (across sites) creates a positive learning environment and a strong foundation for youth development, there are opportunities to strengthen this mission. For example, with intentional activities and strategies, coaches could integrate more empowerment based experiences such as leadership, self-evaluation, responsible roles in the program, and more explicit treatment of the program values and how they can be applied “outside the gym”.

Fidelity of the Cannons Creek and Ashburton Academies to the Naenae Model

The data identified that while there were some variations in practice, the Cannons Creek and Ashburton Academies showed a high level of fidelity to the Naenae model. The Passport to Success was central to all three academies and the TARE analysis identified little difference in the way in which the three programmes were implemented in practice. There were some areas in the surveys which showed statistically significant differences, however, in practical terms there were far greater similarities than differences.

Recommendations

At present, the boxing program is clearly being delivered with a strong commitment to its core values and is creating a much needed space for positive youth development to occur. Across sites and a broad range of stakeholders, the program is perceived as having a positive impact on individual youth that also have implications for families and communities. These accomplishments are impressive given the organic development and evolution of the program. However, if the BGYF aspires to grow and sustain programming, it must continue refining its mission, formalizing the program, and providing systematic resources and support to maintain fidelity to the original program. It is with these opportunities in mind that the evaluation team provides a number of recommendations to build on the current foundation by integrating best practices from relevant fields and formalizing the program in ways that will facilitate expansion and sustainability efforts. Overarching recommendations presented address the following topics: diversity & inclusiveness; curriculum development; continuing professional development; communities of practice; community engagement; bringing core values to life; and ongoing evaluation.