EVALUATION AND RECOMMENDATION

Selecting a data security software package should be a thoughtful and orderly decision process. Although the size and nature of an organization may dictate how formalized this evaluation process will be, all organizations should perform the steps discussed in the following paragraphs.

Selecting the Evaluation Team

The personnel who will screen and evaluate the software packages should be carefully selected. No matter what the size of the organization, executive DP management should be actively involved. Among those who should be on the evaluation and selection team are the:

• DP Manager

• Programming Manager

• Operations Manager

• Data Security Manager/Security Officer

• Auditor

• General Counsel

• Contracting/Purchasing Manager

Soliciting and Evaluating Vendor Responses

The potential user should request that vendor responses to inquiries and solicitations take the form of written technical proposals or proposed state-ments of work. These docustate-ments can range from simple descriptions of the vendor's products, services, and support personnel to in-depth presentations of the vendor's corporate structure and overall management philosophy.

These responses may include detailed tutorials to inform (or snow) the

poten-136 DATA CENTER OPERATIONS MANAGEMENT

tial user regarding the extent of the vendor's knowledge in a particular techni-cal area. Therefore, vendors should be encouraged to keep their responses short and to the point. Much bitter experience has shown that vendors who produce the most elaborate proposals are sometimes marketing oriented rather than engineering oriented-to the ultimate disadvantage of the user. The evaluation team should concentrate on several key portions of the vendor's response.

Statement of Capabilities. This should demonstrate the vendor's ability to deliver the needed software security system. Specifically of concern to the potential customer is evidence of:

• Adequate financial resources, as demonstrated in the vendor's audited 'financial statement

• Ability to comply with the required delivery or performance schedule, taking into account all existing business commitments

• Satisfactory record of performance

• Satisfactory record of integrity and professional ethics (particularly important when dealing with security)

• Necessary organization, experience, operational controls, and technical skills

• Necessary equipment and facilities

• Qualified personnel

• Identification and description of resources needed for proposed work, including vendor- and customer-furnished equipment and facilities Technical Approach. This should demonstrate the vendor's understand-ing of the customer's requirements by:

• Formal technical definitions of requirements

• Identification of technical problems involved in meeting any require-ments

• Alternatives or options, with a discussion of feasibility studies or risk analyses

• Technical descriptions of recommended solutions or alternatives Management Approach. This part of the vendor's response should in-clude an overall project management plan reflecting the organization of the project and related schedules. The formal steps required to complete each of the tasks described in the technical approach section should be specified, and plans for quality assurance and cost controls should be included.

Evaluation Findings. All proposals should be evaluated individually against the organization's functional and technical criteria. The results of the evaluations should then be compared to select the most acceptable proposals.

The :final evaluation of a proposal may be in narrative form or may use a numerical rating scale. A narrative rating should incorporate a summary of the proposal and a description of all advantages, disadvantages, and risks (both technical and other) associated with it. Using a numeric rating scheme, how-ever, is a better approach. Generally, the technical and managerial portions of

SELECTING SECURITY SOFTWARE 137

the proposal are considered to be most significant and are assigned weighting factors accordingly. In order to prevent bias in scoring, the evaluation team should not be informed of the weighting factors beforehand. Those proposals found to be technically acceptable should then be evaluated against the estab-lished cost criteria.

Test and Demonstration

Once the screening process has produced several candidate packages, ar-rangements should be made to demonstrate the capabilities of each through an operational test. A representative subset of the organization's data and/or processes should be used to test the adequacy of each system's controls. It is preferable to do this testing at the user's site, using the equipment and person-nel that will be involved in the operation of the system. If this is impractical, however, and testing is to be performed at the vendor's site, special care must be exercised to ensure that test conditions duplicate the user's operational environment as closely as possible. The user should carefully plan and closely monitor the testing. The test plan should exercise the software according to the evaluation criteria. Samples of all reports (e.g., violations, transactions, audit trails) should be generated. Attempts to circumvent the system should be made as well as attempts to "hang" the system by pwposely creating error conditions. These tests should be carefully documented. The system should be crashed to simulate recovery procedures used in case of system failure. A live test with actual system users should be conducted to evaluate user accept-ance of the package. Resource consumption and response-time degradation should be carefully monitored and recorded for· each system tested.

Final Selection and Recommendation

Based on the results of the evaluation and testing, the evaluation team should be in a position to select a package that meets the needs of the organization. The paramount factors in the final selection are:

• Ability of the software to satisfy specific functional and technical crite-ria

• Performance in operational testing

• User reaction and acceptance

• Overall capability of the vendor to supply and support the package

• Cost to implement, operate, and maintain

• Time, personnel, and system resources required for implementation The evaluation team should then compile these findings and the team's rec-ommendations into a final report and forward it to executive management for review and action.

CONCLUSION

Data security software can provide significant enhancement to the security of any computer system; however, a careful evaluation and selection process

138 DATA CENTER OPERATIONS MANAGEMENT

is needed to ensure that the package chosen will fulfill the organization's needs. First, the organization must have a set of well-defined system security requirements on which to base the selection. Then, functional and technical evaluation criteria must be established. An evaluation team should examine the available packages, solicit and evaluate vendor responses, and test the packages chosen. This evaluation process can help ensure that the software security package will provide full security benefits and will function properly in the organization's environment.