Methodology

Identifying risks to lawfulness and privacy

A key ingredient in developing a sound review selection process is the identification of activities, practices or procedures that may pose a risk to CSEC's compliance with the law. For example, these can be potential risks identified by my staff from previous or current reviews of CSEC activities, or from briefing sessions given to my staff by CSEC. CSEC may itself also identify potential risks.

In assessing topics for possible review, I instruct my staff to consider questions such as: to what extent is CSEC exposed to risk of unlawful activity in this area, and what is the likelihood that this could occur?; and if it occurs, what is the potential adverse impact?

In addition, my staff developed more detailed criteria in 2008–2009 to help determine the priority in which the identified areas of potential risk will be reviewed. These criteria, which continue to be refined, include: significant changes to authorities; changes to technology; any area that has never been reviewed in-depth, or has not been reviewed in the past four years; a follow-up to a particular recommendation I made previously; and issues arising in the public domain.

Attributes of a good review

In conducting a review, my staff examine all relevant written and electronic records, files, correspondence and other documentation. My staff conduct interviews with CSEC managers and staff involved in the activities being reviewed and visit CSEC facilities to conduct checks, including CSEC databases. The results of reviews are shared with CSEC and, in most instances, CSEC takes action to strengthen compliance with the law or policy.

One of my primary concerns in the review of CSEC activities is ensuring that each review is based upon appropriate evidence to support all findings, conclusions and recommendations. This means that all evidence gathered must be directly relevant, replicable and valid.

Relevant: refers to the extent to which the information bears a clear and logical relationship to the review objective(s) and criteria. If information is not relevant, it cannot be evidence. Replicable: concerns the likelihood of coming up with the same findings if all steps of the review were reproduced. Valid: refers to whether the information actually is what it purports to be in relation to the content, origin and timing. As a general principle, the quantity of evidence is sufficient when there is enough to persuade a reasonable person that the review findings and conclusions are valid and the recommendations are appropriate. In order to decide if the collective weight of the evidence is sufficient, I must consider the quality of the evidence gathered, and the cost of obtaining more evidence relative to its likely benefits.

Developing review findings and recommendations

The comparison of evidence gathered against previously established review criteria results in the development of usable findings and recommendations.

Review findings confirm whether criteria have been satisfactorily met, or disclose the level, nature and significance of deviations from them. The process of assessing the evidence gathered against criteria is focussed on questions such as: does a deficiency exist between findings and expectations and as established by the review criteria? what is the cause of the deficiency? what are its likely impacts? and can the deficiency be corrected?