August 6, 2018
Early Psychosis Intervention: Fidelity Assessments in Ontario
By Gord Langill
Members of EPION’s Standards Implementation Steering Committee recently presented their work on fidelity to early psychosis standards at the 2018 Canadian Association for Health Services and Policy Research (CAHSPR) Conference. Gord Langill describes the committee’s approach to measuring fidelity to early psychosis standards in Ontario, Canada.
Ontario, Canada, is a province of 13 million people served by about 50 early psychosis intervention (EPI) programs located across a large geographic area. Programs vary widely in size and delivery model to meet the needs of their local populations- from large urban programs with over 20 clinical staff, to rural programs with one or two staff serving vast sparsely populated areas. The growth of EPI here has been advanced by an active volunteer network named the Early Psychosis Intervention Ontario Network (EPION). EPION has wide EPI program participation as well as representation from consumers, family members, decision makers and researchers.
In 2011, the Ontario government published the Early Psychosis Intervention (EPI) Program Standards and, shortly after, in collaboration with EPION, formed a Standards Implementation Steering Committee (SISC).
A key driver for standards implementation here is the strong partnership between EPION and the unique Provincial System Support Program (PSSP) operated by the Centre for Addiction and Mental Health (CAMH). CAMH is the largest mental health facility in Canada, with a mandate that includes clinical care, education, research, and via PSSP, system development. The PSSP is committed to assisting the Standards Committee with planning, monitoring, evaluation and knowledge translation in order to support implementation of the Ontario Standards.
As an initial step to learn about current practice in relation to the Standards, the Standards Committee conducted two province-wide EPI program key informant surveys during 2012 and 2014.
The survey findings helped EPION to identify areas for further exploration and improvement work through subsequent Think Tanks and our bi-annual EPION conferences. The long-term aim of the Standards Committee is to develop systematic objective processes for monitoring delivery and supporting quality improvement work in Ontario’s EPI programs.
In fall 2016, after a visit from Dr Don Addington, EPION decided to partner with PSSP to pilot a novel peer fidelity review model using the First-Episode Psychosis Services Fidelity Scale. Our goal was to learn more about current practice in Ontario EPI programs and also to test the feasibility and value of the peer fidelity model. It is a challenge to develop fidelity review processes that are valid, sustainable and feasible to spread. Options being explored in other jurisdictions include program self-assessments and remote ‘telephone’ assessments.
Our model used volunteer assessor teams, each team consisting of two EPI staff from EPION programs, and one PSSP evaluator. Fidelity ratings were based on a two-day site visit, where assessors review client charts and program administrative data, as well as interview program staff, clients and family members. Assessors were supported through training, tailored data collection tools, interview guides and post-visit rating consensus meetings with an expert.
Ten programs participated in this initial pilot. Each program received a standardized final fidelity report including quality improvement suggestions as well as item ratings and related explanations. In parallel, we gathered and analyzed quantitative and qualitative data from assessor team members and the participating program sites to assess the feasibility and value of our approach.
Overall feedback from assessors and programs was positive. The assessor teams brought complementary skills to the process, and they appreciated learning about and from peer programs. Programs found the fidelity reports valuable as a mechanism to highlight areas of strength, and identify opportunities for improvement. Programs also emphasized the value of front-line assessors who understood the clinical experience.
The assessors noted a steep learning curve. The materials and training were helpful and we also held consensus meetings after each site visit to help standardize the rating process. Programs varied widely in their practice documentation and that posed a rating challenge. Additionally, the time commitment for assessors was considerable and there was attrition within the assessor team, so maintaining an ongoing pool of trained, available assessors could be a challenge.
Participants felt that, despite the burden, this is an important and feasible process to continue. A second wave of assessments is planned for this year and refinements to the tools and process based on feedback from this pilot are underway. More rigorous evaluation of reliability may be warranted in the future depending on how the results will be used. But for now, we are cautiously optimistic of the feasibility and value of this process, buoyed by the success of the pilot, while cognizant that this was a self-selected sample of early adopters.
Gord Langill is chair of the EPION Standards Implementation Steering Committee and Director of Programs and Services at the Lynx EPI program in Canada.
Other members of the project team include:
- Janet Durbin PhD is the lead scientist on our project team.
- Avra Selick MA is our research coord
- Janet and Avra work at the PSSP (Provincial System and Support Program), Centre for Addiction and Mental Health (CAMH).
- Shannel Butt MSW, from the PEPP (Prevention and Early Intervention in Psychosis Program) at CMHA Elgin, has just joined the project team as our KT lead.
You can hear more from Gord and EPION’s work as well as the application of fidelity measures in the United States, Canada, Italy, England, Australia and Denmark at a symposium session chaired by Don Addington, Universty of Calgary at the upcoming IEPA 11th International Conference on Early Intervention in Mental Health, held in Boston in October 2018. Register for IEPA 11 now.
Follow EPION on Twitter: @help4psychosis