Federal Court of Appeal Decisions

     A-440-95

CORAM:      HUGESSEN J.A.

     STONE J.A.

     LINDEN J.A.

B E T W E E N :

     THE ATTORNEY GENERAL OF CANADA

     Appellant

     - and -

     VALERIE CLARK

     Respondent

     Heard at Ottawa, Ontario, Wednesday, May 14, 1997.

     Judgment rendered at Ottawa, Friday, May 16, 1997.

REASONS FOR JUDGMENT BY:      HUGESSEN J.A.

CONCURRED IN BY:      STONE J.A.

     LINDEN J.A.

     A-440-95

CORAM:      HUGESSEN J.A.

     STONE J.A.

     LINDEN J.A.

B E T W E E N :

     THE ATTORNEY GENERAL OF CANADA

     Appellant

     - and -

     VALERIE CLARK

     Respondent

     REASONS FOR JUDGMENT

HUGESSEN J.A.

     This is an appeal from a judgment of the Trial Division dismissing an application by the Attorney General for judicial review of a decision of an Appeal Board appointed pursuant to section 21 of the Public Service Employment Act¹ which allowed the respondent's appeal. The single question arising for decision is whether this Court's decision in O'Brien et al v. Canada (A.G.)² stands for the proposition that, as a matter of law, verification of the answers to behaviour-based questions is always required in competitions for positions in the Public Service. Both the Appeal Board and the judge of the Trial Division held that it did. In my view, they erred in law in doing so.

     Behaviour-based questioning is, according to the evidence, a widely used selection tool in both the public and private sectors. It consists of asking a candidate, either in a written test or a in oral interview, to describe a situation (usually problematic) from his or her own past work experience and to indicate in what manner he or she responded to it. There is expert evidence, supported by reference to and extracts from published materials, to the effect that it is a valuable tool in the selection process. An example of a behaviour-based question may be found in the present case where the following question was put to the candidates:

Effective Interpersonal Relations

Give me an example when communicating with a client or co-worker was very difficult. Tell me how you handled it. Think of a specific incident. Take a few moments to think of an example.

(Appeal Book, page 9)

     There can be no doubt that in O'Brien, supra, this Court held that the failure by the Selection Board to verify the behaviour-based questions put to the candidates in the competition which was there in issue had the effect of vitiating the entire competition. Décary J.A., who wrote for a unanimous Court of which Stone J.A. and myself were also members, concluded his reasons as follows:

I would allow the application, set aside the decision of the Appeal Board, dated March 20, 1991, and remit the matter to the Appeal Board with directions to allow the appeals on the additional ground that the failure to verify the information given in the answers to the situational question violated the merit principle.

[at page 317]

[Emphasis added]

     O'Brien was, however, very dependent upon its factual matrix. At the very beginning of his reasons, Décary J.A. set out the problem with regard to behaviour-based questions in the following terms:

One of the questions put by the selection board to each of the candidates, for the purpose of assessing their personal suitability qualifications, was a behaviour-based question ("the situational question") which asked them to described how they had handled a sensitive situation. In order to remove any unfairness resulting from untruthful or exaggerated answers which candidates might give to the question, it was intended that the information given by each candidate be verified by the selection board by drawing from other information available on candidates. The selection board, however, never completed such confirmation and made no effort to verify any of the information or analyses provided by each of the candidates in their respective answers. Instead it proceeded to mark the situational question on the basis of whose answer sounded the most impressive, regardless of whether such answer was, in fact, accurate or even truthful.

[at page 314]

[Emphasis added]

     A few paragraphs further on, Décary J.A. explained how the problem had been dealt with in the Appeal Board and how it was presented before us:

A Public Service Commission Appeal Board ("the Appeal Board") was established to hear the appeals. In his decision, rendered March 20, 1991, chairperson Carbonneau made a finding of fact that the situational question could only constitute a valid method of testing personal suitability if the accuracy of each of the candidates' answers thereto were verified by the selection board. Notwithstanding clear evidence that the selection board had not done the verification, chairperson Carbonneau proceeded to conclude that it had and found that the situational question did not, after all, contravene the merit principle. He dismissed that ground of appeal.

There is absolutely no doubt, and counsel for the respondent did not argue otherwise, that that finding of the Appeal Board was made without regard and contrary to the evidence presented at the hearing and would, in ordinary circumstances, warrant the intervention of this court.

[at page 315]

[Emphasis added]

     The balance of Décary J.A.'s reasons in O'Brien go on to deal with an entirely different problem, that of issue estoppel, which is of no relevance for our purposes.

     In the present case, the appellant before the Appeal Board (the respondent before us) took the position that our decision in O'Brien made verification of the answers to behaviour-based questions mandatory in all cases. The employer contested that position as a matter of law, and also led evidence for the purpose of showing that, contrary to what had been found in O'Brien, verification was not as a general rule necessary for the validation of behaviour-based tests and in particular was not necessary in this case. That evidence included much of what had been said on the subject in O'Brien but also comprised other materials, including evidence tending to show that in many cases verification might be very difficult, extremely expensive and time consuming, or simply impossible.

     The Appeal Board dealt with the question of behaviour-based testing in the following terms:

I do share some of the appellant's representative's reservations about behaviour-based questions. If a certain degree of skill is not used in formulating the questions and assessing the replies it may very well be that the Selection Board will not obtain accurate information about candidates' qualifications. However, the same can be said of other selection tools commonly used in the competitive process. Moreover, the appellant's representative has not presented any evidence to refute the department's statement that behaviour-based questions have been found by experts to be a reliable selection tool. Consequently, I am not prepared to conclude that the Selection Board erred simply because it used behaviour-based questions to assess candidates' personal suitability.

In order to intervene on the issue of behaviour-based questions, I would have to be shown that the particular questions or the method of marking the answers to these questions were unfair. I do not find that this is the case here. In my view, the department has given a satisfactory explanation of the marks assigned the appellant's answers to the behaviour-based questions and I am not convinced that the marking scheme used was inadequate. It may very well be that the mark candidates were allotted was dependent on the amount and type of information they gave in reply to the questions but I am not persuaded that this was unreasonable. For instance, when candidates are asked what decisions they refer to their supervisor it does not appear to be unreasonable to expect candidates to indicate that they looked at the consequences of the decisions and also first consulted manuals and colleagues.

I do not consider that there is cause for concern because the appellant obtained higher scores on the reference check than on the behaviour-based questions. The reason more than one selection tool is used is to obtain additional information about candidates' qualifications and all of the information gathered is not likely to be exactly the same.

As for the issue of the verification of candidates' replies to the behaviour-based questions, in my view, the department has made a persuasive argument as to why such verification should not be mandatory. If this was the only consideration, I would decide that the Selection Board was not compelled to verify the information in candidates' replies. However, I am bound by the decision of the Federal Court in O'Brien et al (supra).

[Emphasis added]

(Appeal Book, pages 18-20)

     In my view everything but the last sentence of this extract constitutes findings of fact by the Appeal Board. The last sentence introduces a short discussion of the law in which the Appeal Board concludes that it is bound to reach the same conclusion as in O'Brien.

     That view was confirmed by the judge of the Trial Division who, in his Reasons for Judgment, said:

I am of the view that even though the issue in O'Brien was narrow in that the Court was asked if the Appeal Board had made an erroneous finding of fact, the Federal Court of Appeal went on to categorically state that the failure to verify information given in answers to behaviour-based questions violated the merit principle.

(Appeal Book, page 202)

     With respect, I think that both the Appeal Board and the Trial Division judge have fallen into error. While it is quite true that the conclusion reached by the Court in O'Brien is expressed in categorical terms, that decision must be read in its factual context. As appears clearly from the reasons of Décary J.A. quoted above, the Appeal Board, in that case "made a finding of fact that the situational question could only constitute a valid method of testing personal suitability if the accuracy of each of the candidates' answers thereto were verified by the Selection Board".

     That finding of fact was never put in issue on the application for judicial review to this Court. Indeed, looking at the material which was before the Appeal Board in O'Brien, it is difficult to see how a successful attack could have been launched on the finding. While the evidence before the Board certainly did not compel such a conclusion, it is equally clear to me that is was capable of supporting it.

     What was in issue in O'Brien was a second finding of fact by the Appeal Board to the effect that the Selection Board had done the necessary verification. That finding was manifestly perverse and contrary to the evidence and, as appears from the judgment, the employer conceded as much. It follows, in my view, quite clearly that the Court's conclusion that "the failure to verify the information given in the answers to the situational question violated the merit principle" was not a statement of law but simply the repetition of an uncontested finding of fact made by the Appeal Board.

     It would indeed be strange if this Court were found to have held that behaviour-based questions must, as a matter of law, always be verified in order to satisfy the merit principle. As the evidence makes clear, behaviour-based questions are simply one of a number of tools available in the process of personnel selection. Like all such tools, they are not infallible and have their strengths and weaknesses. Like many such tools, they are also subject to manipulation and even cheating by crafty candidates. The mere possibility that someone may cheat or get the better of the system, however, is surely not a basis in law for declaring the whole system to be defective.

     The adequacy of any particular test in any particular competition will, at bottom, always come down to a question of the appreciation of the facts of the case. In some cases, of which O'Brien is an example, the evidence may allow the Appeal Board to conclude that verification should have taken place. In others, the Appeal Board may find, as it did in this case, that the employer "has made a persuasive argument as to why such verification should not be mandatory". As in O'Brien, the evidence here was adequate to support the finding made.

     Before leaving this matter, I should add that I find particularly compelling the evidence in the record as to the difficulty, and sometimes even impossibility, of verifying all answers to every candidate's behaviour-based questions. This seems to me to argue conclusively against any mandatory requirement of verification as a condition of the validity of the test: if all candidates' answers cannot always be verified then behaviour-based testing can never be acceptable as a selection tool since it will not be possible to know whether or not the answers can be verified until after they are given. One unverifiable answer out of perhaps hundreds would make the whole test invalid. This would seem to me to be a very undesirable result.

     I would allow the appeal, set aside the judgment of the Trial Division, allow the application for judicial review, set aside the decision of the Appeal Board and return the matter to the Appeal Board for re-determination on the basis that the appeal must be dismissed.

     "James K. Hugessen"

     J.A.

"I agree,

     A.J. Stone, J.A."

"I agree,

     A.M. Linden, J.A."

     FEDERAL COURT OF APPEAL

     A-440-95

B E T W E E N :

     THE ATTORNEY GENERAL OF CANADA

     Appellant

     - and -

     VALERIE CLARK

     Respondent

     REASONS FOR JUDGMENT

__________________

¹      R.S.C. 1985, c. P-33

²      (1993), 153 N.R. 313