ETV TECHNICAL PROTOCOL

Objective
The objective of the ETV Technical Protocol is to establish a process for developing, conducting, and reporting scientifically sound evaluations of environmental products and technologies/ that will be sufficiently determinative of their functional performance under stated conditions as to provide a reasonable basis for future decisions by potential users.

Eligibility Criteria
Two of the three threshold requirements that must be satisfied by candidates for the ETV program product ownership/control by the applicant and market-readiness are self-explanatory by their terms and can generally be resolved rather simply and directly. The third prerequisite, the need for individualized criteria or standards to evaluate the innovative aspects of the candidate product or technology, is more subtle and will require an inquiry in each case. The focus of that inquiry, like the ETV Technical Protocol in general, will be on functional performance. Simply, while a standard test is often available to identify or define some distinguishing performance or characteristics of most products, the ultimate test will be whether any such available standards are determinative of the overall performance of the product at hand.

Application Review
The review of completed ETV applications will encompass administrative issues (e.g., eligibility and fees) as well as substantive content. The initial review by ETV staff (and, if necessary, by an independent technical consultant) is to determine whether it satisfies the Eligibility Criteria and general operational objectives for ETV. Following this review, the Technical Evaluation Panel formed to guide the product through the ETV evaluation process shall review any application selected by ETV.

Clearly, the major focus of these successive reviews will be on information bearing on the product’s expected functional performance, i.e., whether the product performs as claimed or intended. In order to best serve the applicant both with respect to the quality of the developed Evaluation Plan and the relative priority assigned to the particular evaluation it is essential that reviewers have a clear and comprehensive understanding of the nature of the product.

To that end, the applicant should ensure that the completed application provides as much specific detail as possible, particularly for the following core issues:

• What particular technological problem or need within the environmental technology market is the product intended to address?
• What innovative feature of the product is intended to satisfy that problem/need?
• What technical criteria or other measurement can be applied to determine how successful the product is in fulfilling its intended function?
• Are there any issues other than performance that might be of significant interest or concern to a potential user (e.g., implementability, health and safety requirements, regulatory issues)?

Panel Formation

General. A distinguishing feature and key to the ultimate success of the ETV evaluation process is the assembly of a multidisciplinary expert panel to guide the evaluation from its inception to final reporting. Members of Technical Evaluation Panels represent the potential users of the products at hand, and serve as volunteers. Panelists are chosen on the basis of their recognized professional expertise and experience. Each Panel will consist to five to six members and will be guided by a chairperson appointed by ETV. There will be a liaison representative to the Panel from ETV (typically the Project Manager) and, at the applicant’s option, a liaison representative to the Panel from the applicant. All actions taken by the Panel, and all advice and counsel furnished, shall be on the basis of a majority consensus of the Panelists. Accordingly, for purposes of group action, each of the Panelist- but not either of the liaison representatives - shall have voting privileges.

The ETV Project Manager is primarily responsible for ensuring the progress and direction of ETV activities. Among his duties include conducting the preliminary review, formation of the review panel, liaison among the members of the panel and the client, documentation of the work plan, and most importantly, the preparation of the final ETV Report.

The role of the applicant’s liaison representative will be to maintain communication between the applicant and Panel to ensure concerted action in the planning and conduct of the evaluation. Although the liaison representative may provide input on technical issues (e.g., background information on the product, the technical objectives sought from the evaluation), the role will be limited to that of a communicator, not a decision-maker. It is expected that the bywords for the continuing relationship between the applicant’s liaison representative and the Panel will be cooperation and professionalism.

Composition. The size and composition of Technical Evaluation Panels will vary, depending on the nature of the product to be evaluated. In assessing the number and qualifications of Panelists required for a particular investigation, ETV staff will apply the following basic tests:

• Does the Panel have a sufficient cross-section of subject matter experts who–as potential users, permitters, or regulators of the product-will be able to successfully formulate all of the basic technical and regulatory questions that the investigation will need to answer to authoritatively evaluate the merits of the product at hand?
• Is there a sufficient number of technologists and research experts with demonstrated expertise in experimental design and related disciplinary fields who will be able to devise an evaluation work plan to effectively address the basic technical questions and provide advice and counsel throughout the course of the evaluation?

While each Technical Evaluation Panel will be uniquely constituted, the core group of Panelists will generally include:

1. Representative from regulatory agencies

2. Private-sector representatives, with relevant experience in consulting, design, laboratory testing, field demonstrations, or other technical and managerial subjects.

3. Researcher from science and technology (S&T) community or university

4. Local government representative (if applicable)

5. Public interests group, as appropriate

6. Other representatives as needed

ETV staff will complete the appointment of the Panel by expanding this cadre to include specialists from other disciplines (e.g., law, insurance, marketing) whose input might be useful for the successful entry and acceptance of the product in the marketplace. Furthermore, additional Panelists may be added upon request of the applicant or the Panel chairperson.

If deemed necessary by ETV staff or the Panel chairperson, an independent technical consultant will be engaged to assist the panel in carrying out its responsibilities.

Responsibilities. The duties and responsibilities of a Technical Evaluation Panel include:

1. Independently reviewing and evaluating the completed application form and all supporting technical questions to be answered.

2. Developing a statement of the objectives of the evaluation, including a statement of the basic technical questions to be answered.

3. Developing a draft evaluation work plan for attainment of the objectives, including estimates of total cost and time and, if necessary, soliciting bids for the work.

4. Recommending a short list of candidate consultants and/or research facilities to carry out the work.

5. Reviewing and ranking the competing proposals subsequently submitted by consultants and/or research facilities.

6. Where appropriate, assisting in arranging an installation of the product on an operating facility, site, or construction project.

7. Monitoring the progress of the evaluation.

8. Providing advice and counsel regarding the conduct of the evaluation work.

9. Reviewing and evaluating reports (including the final Evaluation Report).

Meetings. The number of meetings necessary to carry out the responsibilities of the Technical Evaluation Panel will vary with circumstances. In most cases, three meetings will be required. The first (and most essential) meeting, held over one to two days, will be devoted to developing a scope of work for the evaluation and a list of potential consultants and/or research organizations to carry out the work (i.e., tasks (1) through (4) listed above). The second meeting would be directed at selecting the most qualified consultant and/or research facility to conduct evaluation testing. Finally, the third meeting would be devoted to the review and approval of the data collected during the evaluation and the final report.

Evaluation Factors
The evaluation criteria used to assess a particular product or technology will be specifically tailored such that, taken as a whole, they will provide a sufficiently complete and comprehensive picture of the functional performance and elements of life cycle cost (such as maintenance requirements) of the product, as to provide a reasonable basis for future purchasing decisions by potential users.

In general, developing the evaluation criteria will require assessment of the following five major categories of variables.

1.  FUNCTIONAL PERFORMANCE
Does the product or technology perform as claimed or intended?

• Achievable contaminant removal rates
• Achievable treatment targets
• Achievable net energy savings
• Achievable reduction in waste generation
• Achievable reduction in resource use
  • Water
  • Fuels and energy resources
  • Raw materials
• Durability and operability

2.  ENVIRONMENTAL BENEFITS
How does the technology perform in relation to key environmental parameters?

• Substitution of less-toxic materials
• Beneficial reduction, reuse, or recycling of waste materials
• Degree of permanence of environmental benefits, potential long-term impacts
• Environmental impacts of manufacture, use, or disposal
  • watershed impacts
  • atmospheric impacts (e.g., greenhouse gas emissions)
  • effect on use of energy and natural resources
  • process effluents

3.  IMPLEMENTABILITY
Is the product or technology easy and safe to use?

• Ease of construction, operation, and use
• Safety of construction, operation, and use
• Availability of equipment and materials
• Availability of specialized labor
• Availability of training and support
• Compatibility with existing infrastructure or system
• Sensitivity to external or operating conditions
  • temperature
  • humidity
  • production rates
• Limitations or special requirements of use
  • limited geographic regions
  • specified operating conditions

4.  REGULATORY ISSUES
Will the product or technology be subjected to regulatory or other approvals?

• National agencies
• Local agencies
• Owners/operators
• Community or public interest issues

5.  MAINTENANCE REQUIREMENTS
Does the product or technology require frequent or significant upkeep?

• Frequency of repair
• Ease of repair
• Cost of repair
• Availability of spare parts
• Wastes or residuals generated as a result of repair

In practice, resolving these issues will often require assembling a testing and/or demonstration program using criteria based on an amalgam of standards, specifications, recommended practices and the like, which, in the whole, provide a comprehensive picture of the product’s attributes and performance. To assist Technical Evaluation Panel members in applying this process to particular products, ETV shall compile a library of reference standards and testing guidelines.

Evaluation Plan

The Evaluation Plan furnished to the applicant is basically intended to answer the “Five W’s” of the evaluation: who will do what work, at what cost, when, where and why. The elements of the evaluation plan are:

1. Statement of Technical Objectives

The basic technical questions sought to be answered by the evaluation.

2. Work Plan

Scope. A statement of the degree of comprehensiveness to be adopted for the evaluation – what is included, what is excluded.

Experimental design. A statement of how the evaluation will be structured to answer the technical questions, including:

• the proposed installations or trial uses of the product to be made;
• the variables to be examined;
• planned observations, both qualitative and quantitative; and
• evaluation criteria.

If the evaluation is to be made as part of a larger environmental or construction project, the entire project will be described in general detail, and the proposed installation of the product described in specific detail within the framework of the overall project.

Data collection plan. A scientifically designed, statistically sound plan describing:

• the type and location of data to be collected;
• procedures and instrumentation to be used;
• the frequency, timing and quantity of data to be obtained; and
• data quality assurance/quality control (QA/QC) measures.

3. Personnel Available

Key personnel who will be involved, their qualifications, and their level of effort.

4. Facilities Available

A detailed description of the available physical facilities and/or test sites, and instrumentation.

5. Cost Estimate

An estimate of cost for task and the evaluation as a whole.

6. Reports

A schedule of planned interim and final reports.

7. Time Schedule

A time-scaled listing of all planned evaluation tasks and milestones.

Technical Consultants

To provide a resource which can be drawn on to support Technical Evaluation Panels in the planning and conduct of their work, ETV shall establish a roster of consulting engineers, university research centers and private and public testing facilities (collectively, “consultants”) with recognized expertise in applicable specialties.

The roster is being developed by advertising in a variety of trade and professional publications and from solicitations to professional societies and organizations in order to attract a pool of qualified candidates. The final roster will list the most highly qualified and cost-competitive respondents.

In all cases, the principle to be followed in selecting particular consultants is that, in all but the most exceptional cases, ETV will be guided by the recommendations of the Technical Evaluation Panel. Simply, if the Panel expresses a strong preference for use of a particular consultant or testing facility on account of its premier reputation or clear superiority for the particular product evaluation at hand, ETV will endeavor to secure the services of that individual or firm. This approach is clearly predicated on furthering ETV’s objectives of providing the best possible evaluations of candidate products, not just the least expensive bid.

Reports

Each ETV Evaluation Plan will contain a schedule of planned interim and final reports. The number and type of reports will vary with circumstances, the reporting plan adopted being designed to permit good record-keeping, effective monitoring of financial and technical progress, early disclosure of significant findings, as well as final documentation of technical findings. However, while periodic reports may be required to provide for proper management and control of the evaluation, the only reports that will be published and released to the public are the final report and a technical summary.

The Technical Summary or Verification Statement will contain a short introduction of the study background, objectives and research approach, together with a concise summary of the evaluation results. A clearly written, easy-to-read technical summary can be particularly useful in explaining and promoting the finds, uses and benefits from the products evaluated. Consequently, this brief report (three pages, maximum) will generally be more widely distributed than the final report itself, and will be available for free through the ITDI web site. The final ETV Report shall be available for purchase from ITDI’s ETV Program.

In preparing evaluation reports (other than technical summaries) shall contain the following features:

1. Title Page. This page includes the type of report, title, evaluation number, author, date and disclaimer. The disclaimer shall state: “This report is the result of an impartial, consensus-based approach to evaluating innovative environmental technology in accordance with the ETV Technical Protocol. The data presented are believed accurate and the analyses credible. The statements made and conclusions drawn regarding the product evaluated do not, however, amount to an endorsement or approval of the product in general or for any particular application.”

2. Acknowledgements. Significant contributions by personnel not directly responsible for the study may be acknowledged. The support and direction of the Technical Evaluation Panel must be acknowledged and the names of panelists listed. Any agencies providing financial assistance to the applicant to support the evaluation must be acknowledged.

3. Abstract. The abstract must be as brief as possible and include the following: the principal objectives and scope of the evaluation, the methodology employed, a summary of the results or findings, and the principal conclusions.

4. Table of Contents. A listing of the major sections of the report with corresponding page numbers.

5. List of Tables. This may be omitted when there are fewer than five tables.

6. List of Figures. This may be omitted when there are fewer than five figures.

7. Introduction. The introduction is necessary to provide sufficient background information to allow readers to understand and evaluate the results of the evaluation without referring to other publications on the subject. It will describe the nature and scope of the problem sought to be addressed, provide identification of the product at hand, review the literature and any earlier work that has a bearing on the evaluation, and present any other information which must be understood before the questions under study can be addressed.

8. Objectives. This report section presents a clear and complete description of what was to be accomplished by or discovered by the evaluation.

9. Scope. This report section describes the degree of comprehensiveness adopted for the evaluation and the consequent general applicability of the results.

10. Methodology. This section describes the nature of the evaluation and how each objective was investigated. It will tell what, how and where the study was performed, including construction, sampling, testing, modeling, and other evaluation methods. The underlying evaluation criteria shall be fully described.

11. Discussion of Results or Analysis of Data. This is the core of the report. It compiles and presents the evidence that forms the basis of the conclusions in a logical, comprehensive manner so that the reader can follow the analysis step by step.

The effectiveness of the product as measured by each of the predetermined evaluation criteria will be discussed, together with an assessment of its overall functional performance, under the selected test conditions. A discussion of the product’s cost-effectiveness will be included where possible.

12. Conclusion. This section restates the results of the study more concisely than the Discussion/Analysis section. The findings are presented in the order of their importance, with the most important first. Any limitations on the applicability of the conclusions must be stated.

13. References. A listing of the sources of information cited in the text.

14. Appendices. These sections may be needed for data which is too voluminous, overly technical, or which otherwise cannot be conveniently worked into the body of the report.

15. Disclaimer. Also, while ETV is operating under the cooperative agreement, all ETV technical reports will contain the following acknowledgement and disclaimer:

“This report is based upon work supported by the __________________________”.

“Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the Author(s) and do not necessarily reflect the view of the Department of Science and Technology”.

ETV Main | Governing Principles | Operating Policies | Download Printer-Friendly Version