Appendix B – New Starts Evaluation and Rating Process

[Appendix B: New Starts Evaluation and Rating Process (PDF)]

This paper describes the basic methodology used by the Federal Transit Administration (FTA) to evaluate, rate and recommend funding for projects included in the FY 2000 Annual Report on New Starts. The paper is in the process of being updated to reflect recent modifications to the evaluation and rating process, including the introduction of the transportation system user benefits measure.

FTA reminds the audience of this paper that project evaluation is an on-going process. It is based on an analysis of the Section 5309 New Starts Criteria and documentation submitted to FTA by local agencies. As New Starts projects proceed through project development, the estimates of costs, benefits, and impacts are refined. The FTA ratings and recommendations will be updated annually to reflect new information, changing conditions, and refined financing plans.

I. Legislative Background

On June 9, 1998, the Transportation Equity Act for the 21st Century (TEA-21) was enacted. It requires the U.S. Department of Transportation to submit an annual report to Congress that includes a proposal on the allocation of amounts to be made available to finance grants and loans for capital projects for new fixed guideway systems and extensions to fixed guideway systems among applicants for those amounts. It also requires that the annual report include the Secretary’s evaluations and ratings of the capital projects seeking grants or loans for new or extended fixed guideway systems.

TEA-21 also mandates that proposed New Starts projects must receive FTA approval to advance from alternatives analysis to preliminary engineering, and from preliminary engineering to final design and construction. This approval will be based, in large part, on an evaluation of the proposed project’s New Starts criteria.

FTA’s evaluations includes a review of each project’s New Starts criteria and the assignment of a rating to each criteria. Based on these criteria-specific ratings, candidate New Starts projects may be rated as "highly recommended", "recommended" or "not recommended". FTA’s proposed approach to developing these ratings is described in new regulations.

I.A Notice of Proposed Rulemaking

U.S. Department of Transportation regulations currently under development will define the summary project ratings of "recommended", "highly recommended" and "not recommended" as required by TEA-21, document the measures for project justification, determine how FTA will use the summary ratings to approve entry into preliminary engineering and final design and discuss the relationship of the project evaluation process to the planning and project development process. The draft regulation is expected to be released for public comment in a Notice of Proposed Rulemaking (NPRM) in Spring 1999.

I.B Interim Approach Applied in the Annual New Starts Report for FY 2000

Since the Final Rule has not been published, FTA has completed the project evaluations and funding recommendations for FY 2000 based on FTA’s existing New Starts criteria and evaluation process as published in the Federal Register on December 19, 1996 (61 FR 67093-106) and amended on November 12, 1997 (62 FR 60756-58) and modified to account for the changes made by TEA-21. For the Annual New Starts Report for FY 2000, FTA applied the rating and evaluation process to forty-two (42) projects in Final Design and Preliminary Engineering.

As proposed New Starts projects proceed through the stages of the planning and project development process, FTA evaluates and rates projects against the full range of criteria for project justification and local financial commitment contained in §5309(e). To assist local agencies in the application and reporting of the criteria, FTA issued the Technical Guidance on Section 5309 New Starts Criteria in September 1997. An Addendum to the Technical Guidance was issued in October 1998 to further support local agencies. Revised technical guidance to further clarify and reflect TEA-21 New Starts provisions and the final regulations on New Starts evaluation and rating are currently under development.

The following sections identify the specific New Starts project justification and local financial commitment criteria applied and outline the New Starts evaluation and rating process.

I.C Project Justification Criteria

Section 5309(e)(1)(B) requires that projects proposed for New Starts funding be justified based on a comprehensive review of the following criteria:

Section 5309(e)(3)( C) requires FTA to further consider mass transit-supportive land use policies and future patterns; subsequently, FTA added the following criteria:

Finally, FTA also considers "Other Factors," as required by Section 5309(e)(3)(H)

I.D Local Financial Commitment

Section 5309(e)(1)(C) requires that proposed projects also be supported by an acceptable degree of local financial commitment, including evidence of stable and dependable financing sources to construct, maintain and operate the transit system. The criteria for the evaluation of the local financial commitment to a proposed project are:

II. The Evaluation And Rating Process

FTA evaluates proposed new start projects against the full range of criteria for both project justification and local financial commitment, using a multiple measure method illustrated on the following flow chart.

II.A Project Recommendation

Consistent with §5309(e)(6), an overall project rating of "highly recommended", "recommended" or "not recommended" is assigned to each proposed project, based on the results of FTA’s evaluation of each of the criteria for project justification and local financial commitment.

II.B Project Recommendation Decision Rule

To assign overall project ratings ("highly recommended", "recommended" or "not recommended") to each proposed New Starts project, FTA considers the individual ratings for each of the financial rating factors and project justification criteria. FTA combines this information into summary Finance and "project justification" ratings for each project. These summary ratings are in turn used to determine overall project ratings according to the following decision rule:

II.C Ratings: An On-going Process

Again, it is important to emphasize that project evaluation is an on-going process. FTA evaluation and rating occurs annually in support of budget recommendations presented in the annual New Starts report to Congress and when project’s request FTA approval to enter into preliminary engineering and final design. Consequently, as proposed New Starts projects proceed through the project development process, information concerning costs, benefits, and impacts is refined and the ratings updated to reflect new information.

III. Financial Rating

The following provides a summary of FTA’s process for evaluating the local financial commitment of proposed New Starts projects.

III.A Financial Rating

FTA assigns a summary financial rating of "high", "medium high", "medium", "low-medium" or "low" to each project following consideration of individual ratings applied to a) the stability and reliability of the proposed project’s capital finance plan and b) the stability and reliability of the proposed project’s operating finance plan. These ratings are based on an analysis of the Section 5309 New Starts Criteria and documentation submitted to FTA by local agencies. FTA strongly considers the project development stage of the proposed investment in its evaluation. FTA assigns one of five descriptive ratings "high", "medium high", "medium", "low-medium" or "low" to each of these factors. In addition, the overall financial rating considers the non-Section 5309 share of project capital costs as well as the historic support of new start projects by the applicant.

Individual ratings for each project reflecting non-Section 5309 share, the capital finance plan and the operating finance plan are combined by FTA into an overall financial rating of "high", "medium high", "medium", "low-medium" or "low". FTA gives particular attention to the stability and reliability of the capital finance plan as reflected in the decision rule outlined below.

III.B Financial Rating Decision Rule

If a proposed project’s capital finance plan receives a "low-medium" or "low" rating, the overall financial rating for the project cannot be higher than a "low-medium."

FTA’s financial assessment and ratings clearly take into account the stage of project development, particularly when considering stability and reliability of the capital finance and operating finance plans. Expectations for firm commitments of non-Federal funding sources are higher as projects progress further through project development. These issues are taken into consideration and reflected by a rating of "high", "medium high", "medium", "low-medium" or "low." The basis for assignment of ratings by stage of project development is documented in FTA’s Technical Guidance and Addendum.

IV. Project Justification Rating

The following summarizes FTA’s process for evaluating the project justification criteria of proposed New Starts projects.

IV.A Project Justification Rating

FTA assigns a summary project justification rating of "high", "medium-high", "medium", "low-medium" or "low" to each project based on consideration of the ratings applied to the project justification evaluation criteria:

Based on an analysis of the Section 5309 New Starts Criteria and supporting documentation submitted to FTA by local agencies, FTA assigns a descriptive rating ("high", "medium high", "medium", "low-medium" or "low") to each of these criteria. Ratings for each of these criteria are then combined to a summary "high", "medium high", "medium", "low-medium" or "low" project justification rating for each project. FTA gives particular attention to transit supportive land use and cost effectiveness in the determination of the overall project justification rating.

The evaluation and rating of individual project justification criteria is discussed below.

IV.B Mobility Improvements

In its evaluation of the mobility improvements that would be realized by implementation of a proposed project, FTA reviews two measures:

  1. travel time savings;
  2. the number of low income households served.

Based on an analysis of the Section 5309 New Starts Criteria documentation submitted to FTA by local agencies as part of their application and reporting process, FTA assigns one of five descriptive ratings ("high", "medium high", "medium", "low-medium" or "low") to each of the two measures. These ratings are then combined (with greater emphasis on travel time savings) to assign a mobility improvements rating.

Travel Time Savings This measure reflects the aggregate travel time savings in the forecast year anticipated from the proposed project compared to the no-build and the TSM alternatives. Section 3010 of TEA-21 prohibits the consideration of "the dollar value of mobility improvements". Consequently, FTA evaluates the sum total of estimated hours of travel time saved (or increased). In order to rate projects in comparison to other proposed New Starts, this measure is normalized by the annualized capital costs of the proposed project, resulting in a measure of hours saved per dollar of capital cost. Based on the project information submitted, a composite index is developed which reflects comparison of the new start to both the no-build and the TSM comparisons. In instances where project sponsors provide only a single comparison, no-build or TSM, only the single comparison is used. For the FY2000 submissions, the travel time savings indices range from .007 hours per dollar in annual capital costs to 1.352 hours per dollar in annual capital costs, with a median reported of .058 hours per dollar in annual capital costs. Projects are then ranked according to this normalized composite measure and assigned a "high", "medium-high", "medium", "low-medium" or "low" rating based on its relative ranking compared to the other New Starts projects reported.

Number of Low Income Households Served This measure reflects the absolute number of low income households (defined as below the poverty level) located within ½ mile of the "boarding points", or stations, associated with the proposed project. In order to rate projects in comparison to other proposed New Starts, this measure is normalized by the annualized capital cost of the proposed project, resulting in a measure of persons served per dollar of capital cost. Based on the project information submitted to FTA by local agencies for the FY2000 evaluations, the number of low income households served ranges from 0.1 low income households per million dollars in annual capital costs to 1,453 low income households per million dollars in annual capital costs, with a median reported of 64.30 low income households per million dollars in annual capital costs. Projects are then ranked according to this normalized measure and assigned a "high", "medium-high", "medium", "low-medium" or "low" rating based on its relative ranking compared to the other New Starts projects reported.

IV.C Operating Efficiencies

FTA measures this criterion by evaluating the change in systemwide operating costs per passenger mile in the forecast year, comparing the Section 5309 New Start investment to the no-build and TSM alternatives. Based on the project information submitted to FTA by local agencies, the projects are assigned a "high", "medium" or "low" based on the following decision rule.

IV.D Environmental Benefits

In its evaluation of environmental benefits that would be realized through the implementation of a proposed project, FTA considers the following measures:

Current Air Quality Designation by EPA This measure is defined for each of the transportation-related pollutants (ozone, CO, and PM) as the current air quality designation by EPA for the metropolitan region in which the proposed project is located, indicating the severity of the metropolitan area’s noncompliance with the health-based EPA standard (NAAQS) for the pollutant, or its compliance with that standard.

Net Change in Air Pollutant Emissions These measures are defined as the net change in emissions of any of the transportation related pollutants for which the U.S. Environmental Protection Agency (EPA) has established health-based National Ambient Air Quality Standards (NAAQS),or in the emissions of a precursor of such a pollutant. The relevant pollutants and precursors are volatile organic compounds (or hydrocarbons), oxides of nitrogen (NOx), carbon monoxide (CO), and particulate matter (PM). Each of the measures is expressed as the annual emissions for the metropolitan region in the forecast year, comparing conditions under the Section 5309 New Start investment first to the no-build conditions and then to conditions under the TSM alternatives.

Net Change in Greenhouse Gas Emissions This measure is defined as the net change in emissions of the primary transportation-related greenhouse gas (carbon dioxide) in the forecast year. The measure is expressed as the difference (in tons) in the annual emissions of carbon dioxide from transportation sources in the metropolitan region, comparing conditions under the Section 5309 New Start investment first to the no-build conditions and then to conditions under the TSM alternative.

Net Change in Regional Energy Consumption This measure is defined as the change in regional energy consumption for transportation purposes in the forecast year, measured in British Thermal Units (BTU), comparing the Section 5309 New Start investment first to the no-build and then to TSM alternatives.

The environmental benefits of each proposed project are rated high, medium or low based on an analysis of the air quality, emissions and energy consumption documentation submitted to FTA by local sponsoring agencies as part of the application and reporting process. FTA assigns an "environmental benefits" rating based on the following rules:

IV.E Transit-Supportive Existing Land Use and Future Patterns

In its evaluation of the transit supportive land use affecting transit projects, FTA explicitly considers the following transit supportive land use measures:

Based on information submitted to FTA by local agencies, FTA gauges each of these seven measurement factors by a variety of sub-elements or considerations. These illustrate various aspects of existing and planned transit-supportive land use, such as mixed use development, employment and population density, pedestrian and bicycle capability, directed growth mechanisms, parking policies, and public and private involvement. FTA assigns an overall land use rating of "high", "medium-high", "medium", "low-medium" or "low" to each project following consideration of the seven factors listed above.

Existing Conditions and Plans and Policies FTA considers the relationship between existing conditions and plans/policies when evaluating projects and assigning overall land use ratings. FTA places a strong weight on existing conditions, including existing station area development, existing zoning, and major trip generators along the corridor because it is FTA’s experience that a major fixed-guideway investment requires some level of transit-supportive land use to justify and support the project. However, for some proposed projects not necessarily rated highly for existing conditions, the local efforts at encouraging future transit-supportive development were sufficiently strong to improve the overall land use rating.

Stage of Development When evaluating the land use factors, FTA also takes into consideration the stage of development of a proposed project. The planning and policy oriented factors (existing land use, containment of sprawl, and corridor policies) are relevant in evaluating projects in all stages of project development, but particularly useful for projects early in project development. The implementation oriented factors (supportive zoning regulations, implementation tools, and performance of land use policies) are more applicable in evaluating more advanced projects further along in preliminary engineering or final design.

IV.F Cost Effectiveness

In its evaluation of the cost effectiveness of a proposed project, FTA considers the incremental cost per incremental passenger in the forecast year. This measure, expressed in current year dollar value, is based on the annualized total capital investment (Federal and local funds) and annual operating costs divided by the forecast change in annual transit system ridership, comparing the proposed project to the no-build and the TSM alternatives. Based on the project information submitted to FTA by local agencies, a composite index is developed which reflects both the no-build and the TSM comparisons. In instances where project sponsors provide a single comparison, no-build or TSM, only the single comparison is evaluated. For the FY 2000 submissions, the cost-effectiveness indices range from $2.54 per new rider to $48.82 per new rider, with a median reported of $10.39 per new rider. Projects are then ranked according to this normalized measure and assigned a "high", "medium-high", "medium", "low-medium" or "low" rating based on its relative ranking compared to the other New Starts projects reported.

IV.G Other Factors

Consistent with §5309(e)(3)(H), FTA also includes a variety of "other factors" when evaluating project justification, including:

FTA considers other factors in the evaluation of candidate New Starts projects in two ways. For evaluations in support of budget recommendations contained in the annual New Starts report to Congress, other factors are introduced after the assignment of an initial summary project justification rating. FTA then evaluates the project’s other factors. If the other factors are determined to be particularly significant, FTA may increase the project’s initial project justification summary rating by one step (for example, from "low-medium" to "medium") to reflect this significance. Projects with less compelling other factors maintain their initial summary project justification rating.

For preliminary engineering and final design approval, other factors are considered in the same way. In addition, the technical capability of the project sponsor to implement and operate the project are considered within the other factors criteria. This inclusion ensures that project management issues are adequately addressed in FTA’s decision to permit advancement into the next stage of the project development process. FTA is currently developing guidance on the preliminary engineering and final design approval process.

V. Conclusion

TEA-21 requires that FTA evaluate each candidate New Starts project, and to assign overall project ratings of "highly recommended", "recommended" or "not recommended." FTA undertakes this evaluation and rating for all projects in preliminary engineering and final design included in the annual New Starts report to Congress. FTA also evaluates and rates projects at the point that their sponsors request FTA entry into preliminary engineering and final design.

To assign overall project ratings to each proposed New Starts project, FTA considers the individual ratings for each of the financial rating factors and project justification criteria. FTA combines this information into summary Finance and "project justification" ratings for each project. These summary ratings are in turn used to determine overall project ratings according to the following decision rule:

Again, FTA emphasizes that project evaluation and rating is an on-going process; as proposed New Starts projects proceed through the project development process, information concerning costs, benefits, and impacts is refined and the ratings may be updated to reflect new information.