IPRA Executive Summary

  • In the spring of 2013, PMC Management Coaching engaged the services of the Institute for Psychological Research and Application (IPRA) at Bowling Green State University to validate its Individual Coaching Assessment (IA). The goal was to ensure that its measure was psychometrically sound and provided valid assessments to its clients.

    IPRA examined the psychometric properties of the performance indicators (Climate and Image) and coaching factors (Desire, Credibility, Skill, and Capacity) measured by the Individual Coaching Assessment (IA) created by Personalized Management Coaching (PMC). The data were collected over a 10 year period from managers, their direct reports, and senior managers at three time points with an initial sample size of 423 managers and a final sample size of 63 Managers.

    The results for the six coaching subscales showed that they were highly reliable with reliability coefficients ranging from .71 to .96 across all three raters. In addition, the IA scales were highly correlated with each other (meaning a high correlation between the factors and the performance indicators) and with the PMC Coaching patterns (e.g., Engagement, Employee Development, Management Competencies).

    Most importantly, the subscales for the direct reports were correlated with promotion and turnover. Specifically, the Climate and Desire subscales were correlated with promotion, while the Credibility, Desire, and Skills subscales were correlated with turnover. There was also a moderate level of consistency among raters, indicating that ratings were similar across sources but that each score also provided a unique perspective. In contrast, the self-reported coaching scores as well as the scores provided by senior managers did not correlate with any of the outcomes.

    Therefore, our results clearly indicated that direct reports on the IA scales were better predictors of important work outcomes than either self-reports or scores provided by senior managers. In other words, direct reports of coaching effectiveness may be more useful for identifying successful managers. This is consistent with the research literature which shows that direct reports have a unique perspective on managers’ performance and may be in a better position to evaluate their potential for future managerial/leadership positions than senior managers (London & Beatty, 1993; Moses, Hollenbeck, & Sorcher, 1993). Consequently, these findings may be particularly important when considering the use of IA scores for making promotion decisions or for succession planning.

    In addition to the validity of the IA scales for predicting important work outcomes, the subscales also showed high test-retest reliabilities, suggesting that the responses were consistent over time. Despite this consistency, we also found that the self-reported scores and the scores provided by senior managers tended to increase significantly over time, suggesting that  managers can become more effective coaches. No significant gender differences were observed, confirming that the six IA subscales were not related to the sex of the individual.

    Our analyses also suggested that the IA scales were measuring a single underlying psychological characteristic. In other words, each of the IA items seems to assess a broader dimension of coaching effectiveness. Therefore, we examined the psychometric properties of a single overall composite score that combined the 45 items in the IA scale. The reliability coefficients for the composite scores were slightly higher than the subscales and ranged from .92 to .98 across all raters. In addition, the composite score provided by direct reports predicted both turnover and promotion, whereas the combined score based on self-reports predicted turnover. There were no significant results for the combined scores provided by senior managers. The implications of these results and recommendations based on these analyses are also discussed.

 

About admin