- DRAFT -
Federal Transit Administration
Office of Planning and Environment
US Department of Transportation
Table of Contents
|9.3 Understanding the Problem|
|9.4 Identifying Measures|
|9.4.2 Impact Measures|
|9.4.3 Comparison of Benefits and Costs|
|126.96.36.199 Choice of a Baseline Alternative|
|188.8.131.52 Quantification of Costs and Benefits|
|9.4.4 Summary of the Recommended Approach to Cost-Effectiveness Calculation|
|9.5 Financial Feasibility|
|9.7 Trade-Off Analysis|
|9.8 Documenting the Evaluation Methodology|
1. the lack of some basic structure for the evaluation risks a rambling, unfocused discussion that more often repeats rather than interprets the data; and
2. complex “weighting and rating” schemes tend to confuse rather than illuminate the issues and are often only tenuously related to the realities of decision-making.
As a result, some combination of the of structured analysis and informed judgment of local project staff and Technical Advisory Committees is advantageous to focus the evaluation on the key issues.
One suggested approach is to identify and display the key measures against which each alternative is evaluated in a small, one or two-page table. The evaluation measures should be quantitative rather than qualitative if at all possible. The goals and objectives of most transportation projects typically call for five classes of evaluation measures in a desirable project:
1. Effectiveness – the extent to which the project solves the stated transportation problems in the corridor;
2. Impacts – the extent to which the project supports economic development, environmental or local policy goals;
3. Cost-effectiveness (or cost-benefit analysis) – that the costs of the project, both capital and operating, be commensurate with its benefits;
4. Financial feasibility – that funds for the construction and operation of the alternative be readily available in the sense that they do not place undue burdens on the sources of those funds; and
5. Equity – that the costs and benefits be distributed fairly across different population groups.
The evaluation framework must be focused on the transportation problems identified during system planning, which guide the alternatives analysis. The evaluation method should begin with the statement of goals and objectives for transportation improvements. Where existing statements are available, they should be organized into the structure that will be used for the evaluation. Where new or revised statements of goals and objectives are prepared, the perspectives provide a useful starting point for identifying and organizing local concerns.
It is useful to recognize that the evaluation phase of project planning – and of any assessment of complex options – is not restricted to the final phase of the analysis. Rather, it is a continuous and comprehensive process within which the technical work proceeds. The process is continuous in that there is a series of decisions that must be made through the analysis – alignment variations, design standards, operating policies, etc. – that together shape the nature and performance of each alternative. It is comprehensive in that the final evaluation of an alternative considers a broad range of criteria – transportation, environment, costs, finances, etc. – that require a broad perspective in the assessment of design decisions. Clearly, the ongoing decision-making should be carried out with regard to its ultimate impact on the evaluation of each alternative.
It is also important to reemphasize that the evaluation is primarily focused on local decision-making. While this should be obvious, particularly for projects that are not subject to FTA’s New Starts evaluation and rating process, there have been cases in which the entire evaluation has focused on “qualifying” for Federal funding rather than on identifying transportation needs and solutions. Emphasis on the Federal decision is not consistent with the intent or nature of FTA’s New Starts program. The FTA Final Rule for Major Capital Investments recognizes that legitimate differences often exist between the local and Federal views of major transit projects. It specifically identifies the Federal interest in transit and outlines the standards against which funding proposals will be measured. The intention is that local officials examine the transit alternatives against their own objectives, so that an agreement can be reached on the aspects of a project that are consistent with Federal goals (and attractive for Federal investment) and those that are primarily local objectives that should be funded locally. Therefore, the evaluation process should consider all perspectives from which the alternatives will be examined.
The evaluation measures chosen to evaluate the relative merits of transportation alternatives spring directly from the local problems the alternatives analysis is designed to solve. While many transportation projects have similar objectives, such as improved mobility and accessibility and economic development, local conditions should drive the development and evaluation of alternatives.
Local conditions may focus the evaluation on environmental concerns, capacity constraints, congestion relief, social policy goals, mobility of transit dependent populations, land use impacts, or any other local concern. The decision to select a project as the locally preferred alternative should spring from local needs and concerns rather than the evaluation criteria used for the federal funding decision.
However, the conduct of the alternatives analysis where a fixed guideway transit investment could become the locally preferred alternative should produce the inputs required for the federal rating and evaluation process to avoid the possibility that significant new work would be required before entering PE. The measures used in the federal evaluation for New Starts projects is found in Reporting Instructions for the Section 5309 New Starts Criteria, published every year by the Federal Transit Administration.
The measures selected to guide the evaluation of the alternatives should be focused on solving the specific problems in the corridor. Most of the commonly used measures are discussed in the following sections and fall under the general categories of transportation effectiveness, impacts, cost-effectiveness, financial feasibility and equity.
There are several considerations in the selection of evaluation measures related to the assessment of alternative investments:
1) The measures should be developed early in the analysis with appropriate input from local decision-makers.
The review is an obvious step to ensure the relevance and usefulness of the information. The evaluation methodology should be a high priority item in the early stages of the analysis. Development of a written explanation of the evaluation process is often the catalyst for local officials to come to grips with the specific measures that are of importance for local decision-making.
2) The measures should be comprehensive in that they address all of the stated objectives, but they should be structured to avoid simple restatements of the same benefits.
Many potential effectiveness and impact measures are interrelated. In some cases, there is good reason to include measures of the same impact that portray the impact from different perspectives. For example, the increased development potential of an area may be due primarily to the improvement in transit accessibility to that site. While including both measures of accessibility and measures of development potential double-counts some benefits, both may be of sufficient interest to warrant their use in the analysis. This is in contrast to the subsequent cost-effectiveness analysis where double-counting the same benefits would be an error. In other cases, two candidate measures can be purely redundant. For example, it is unnecessary to include both “total transit trips” and “transit trips diverted from autos” since the second measure is a direct mathematical derivation from the first.
3) To the extent possible, the measures should quantify the impacts rather than express subjective judgments on the nature of the impact.
Many of the important objectives of an improvement can be difficult to quantify and the consequent temptation is to use subjective evaluation measures: significant or not significant, desirable or not desirable, and so forth. However, it is usually more useful to provide measurements rather than judgments to local officials and the public. There is an adage to the effect that the relocation of a single residence for a major project is not “significant” unless it is your residence. Useful quantified measures can usually be identified for most objectives. For example, the impacts of street closings on neighborhoods can be addressed with such measures as the number of local streets closed to traffic and the number of residences and business.
4) The measures should provide the proper perspective on the magnitude of the impacts.
Many of the impacts of a transportation improvement occur in terms of numbers that are large in an absolute sense but are relatively small when placed in perspective. For example, travel time savings of 1,000 hours a day represents 3 minutes per trip when spread over 20,000 transit trips. However, 1,000 hours is only 14 seconds per trip if spread over 250,000 drivers, which is not likely to be noticed. Also, the relocation of one million square feet of new office space to station areas may appear quite significant when presented by itself, but is more meaningful when also shown as the percentage (say three percent) of total development expected in the corridor over the study period. Similarly, pollutant reductions expressed in terms of thousands of pounds per day is misleading in terms of region-wide air quality impacts if the reduction constitutes a tiny fraction of total emissions in the region.
5) Finally, discussion of the measures should reflect the magnitude of differences in the measures compared to the likely error levels they may contain.
Varying degrees of uncertainty exist in all information used in project planning. The presentation of evaluation measures should be accompanied by a well-written discussion that both highlights the major differences between alternatives and indicates where the differences are small given the levels of uncertainty. Minor differences in transit ridership, for example, are usually within the error of the estimates.
Within these general guidelines, the identification of specific measures depends only on the locally identified goals and objectives, together with the judgment of local analysts and officials on the most useful ways of portraying the relative merits and trade-offs involved with each alternative. The following sections describe the range of evaluation measures commonly used to evaluate alternatives.
Goals and objectives related to effectiveness both establish the reasons for which major transit improvements are being considered, and identify ancillary concerns that constrain the options. Transportation concerns – congestion, mobility, etc. – are usually the primary basis for consideration of a major action in the corridor.
Effectiveness measures may include, but are not limited to:
• travel costs/user benefits;
• transportation system capacity;
• accidents and incidents;
• level of service/volumes/trips on key facilities;
• accessibility measures (number of jobs or households within specific travel times to destinations by mode);
• system redundancy (travel reliability measures); and
• any other quantifiable transportation system impact.
These effectiveness measures, with the exception of accidents and incidents, are generally the direct result of the travel demand forecasting process. Most of these measures are an output of the regional travel demand model and their calculation covered in Part I Chapter 6 Interpretation and Use of Travel Forecast Data of this guidance.
Transportation projects create numerous secondary impacts that must typically be evaluated during alternatives analysis. The predominant secondary impacts that are commonly used to evaluate transportation alternatives are environmental considerations and economic development impacts. In some cases, these impacts are the focus of the locally defined evaluation if they respond directly to the primary problem in the corridor.
The menu of impact measures generally includes, but is not limited to:
• Regional economic impacts:
o jobs added;
o tax base;
o redevelopment of distressed areas;
o national competitive standing; and
o distribution of economic impacts across jurisdictions.
• Effects on the human environment:
o residential/business/farm property takings;
o impacts on nearby residences/businesses/farms;
o community impacts of facilities, disruption or barriers;
o parks and recreation areas - number, acreage or proximity effects; and
o historic and archeological sites – number, acreage, or proximity effects.
• Effect on the natural environment:
o streams, wetland, floodplains – number, nature, likely impacts, implications for approvals;
o water quality;
o rare, threatened or endangered species and related habitat;
o forests; and
o air quality.
• Consistency with local or state plans and policies:
o comprehensive plans;
o proximity and impact on priority development areas; and
o land use and zoning policies.
Two common methods are used to evaluate the benefits of transportation improvements in the context of their relative costs. These are cost-effectiveness and cost-benefit analysis. These measures help identify the most efficient use of public resources to achieve the projected transportation benefits or other impacts.
Three primary issues arise in any attempt to fashion cost-benefit measures or measures of cost-effectiveness:
• the overall structure of the analysis and resulting measures;
• the baseline against which the alternatives are compared; and
• the measures used to quantify costs and benefits.
FTA has identified an approach used to support Federal decision-making. Local officials may choose a different approach, so long as it is technically sound and can accurately measure project merit relative to the purpose and need for the project. The results of both approaches may be presented in the environmental document produced by the study.
A major question in evaluation is the way in which the trade-off between costs and benefits is portrayed. One option is the standard cost-effectiveness approach in which a required performance level is stated and alternatives are evaluated for the least cost option that achieves this performance. This approach is very useful where the performance requirements are easily stated and measured. Unfortunately for transportation planning, the objectives for urban transportation investments are usually so many, so varied, and perhaps so unclearly defined that they defy statement in terms of specific performance levels.
The conventional approach to comparing the costs and benefits of transportation investments is to estimate resource cost savings resulting from a proposed project relative to a baseline scenario. For instance, an investment in a new light rail transit system will create benefits for existing transit riders who take advantage of the new system and to those who switch from auto or HOV to the new rail line. These user benefits (costs) are generally:
• out-of-pocket cost savings (parking, tolls, fares);
• vehicle operating cost savings (fuel, oil, tires, insurance, depreciation); and
• safety benefits (reduced accidents, injuries and fatalities).
In addition to user benefits (costs), there are a several categories of benefits (costs) that accrue to society at large rather than to users of the transportation system. These so-called non-user benefits (costs) include, but are not limited to:
• environmental benefits (costs); and
• resource savings for transportation operations and maintenance (infrastructure unit costs).
These benefits and costs are driven primarily by changes in travel demand and the generalized cost of travel caused by the project. Each of the benefit measures can usually be related in some way to changes in travel demand and the relative costs of each unit of that demand.
A cost-benefit analysis requires each of these impacts to be monetized to compare the value of the project to its costs. Monetizing these benefits is very difficult and occasionally controversial since this step requires assigning a value to, for instance, a ton of a particular pollutant or greenhouse gas and valuing a person’s life and time.
A cost-benefit analysis begins with forecasts of total monetized benefits and annual capital and operating and maintenance costs for the evaluation period which is typically 20 years, but can be longer due to the long useful life of most transportation investments. These streams of benefits and costs are discounted to reflect the time value of money and summed to reflect the total present value of the stream of costs and benefits. The discount factor is 1/(1+i)n where i is the discount rate and n is the year of analysis. Traditionally the discount rate is between 7% and 10%.
The most common cost-benefit measures are:
1. Net Present Value (NPV) = [PV of Benefits ($) – PV of Costs ($)]
2. Internal Rate of Return (IRR) = Discount rate at which NPV=0
3. Benefit/Cost ratio (B/C) = PV of Benefits ($)/PV of Costs ($)
1. For existing sources that are dedicated entirely to transit, the surplus or deficit of projected funds compared to projected needs is likely the best indicator of financial capability.
2. For new sources, discussion of the steps necessary to develop the source is a primary concern. This discussion would identify the necessary major actions – referenda, local legislation, State legislation, etc. – and, to the extent possible, the likelihood of success given past experience with similar efforts. Levels of risk can be defined and assigned to each source as an indicator of its feasibility.
3. For new sources or for existing sources that are not dedicated entirely to transit, ratios can be constructed to illustrate the size of the transit requirement in comparison with various measures of financial capability. For example, where transit is currently funded as a budget line item of local government, a useful measure is the current and projected percentage of the total budget necessary for transit. This measure reflects the need for transit assistance, the total resources available to the local government, and the needs of other local governmental functions. A second example would be measures of the financial feasibility of value capture mechanisms that indicate the fractional change in profitability of development within a value capture district.
1. The extent to which the transit investments improve transit service to various population segments, particularly those that tend to be transit-dependent;
2. The distribution of the costs of the project across the population through whatever funding mechanism is used to cover the local contribution to construction and operation; and
3. The incidence of significant environmental impacts.
Each of these classes of impacts should be pursued to the extent that they are identified as areas of concern by study team, local decision-makers or by other groups contacted through the study’s public participation process. Where appropriate, there are analytical techniques available to quantify several measures of the distribution of costs and benefits. For the distribution of service improvements, the demographic data and transit network information developed in the travel forecasting work provide a wealth of data on service changes for individual market segments. Finally, the environmental analysis provides an inventory of likely impacts on neighborhoods, residences, and businesses that can be used to quantify the extent to which specific population groups would be adversely affected by any of the alternatives.
Thus far, the evaluation has proceeded sequentially through five perspectives, examining each alternative in turn. The purpose of the trade-off analysis is to pull together the key differences among the alternatives across all of the perspectives. It is designed to take the broadest view possible, highlighting for decision-makers the advantages and disadvantages of each option and pointing out the key trade-offs of costs and benefits that must be made in choosing a course of action.
As in much of the evaluation, the content and approach to the analysis is dependent upon local goals and objectives and the nature of the alternatives. Perhaps the most important component of a successful trade-off analysis is its assignment to an analyst who is able to take a broad perspective on the purpose of the transportation improvement and the merits of the alternatives, and who has strong writing skills. Together with reviews by the Technical Advisory Committee, the analyst’s insight and reasoning are indispensable to a result that aids local officials in the choice of an alternative.
Several examples can be used to illustrate the kinds of trade-offs that might be found in a set of alternatives. One frequently-found trade-off is that between effectiveness and cost-effectiveness. One alternative may yield a modest level of transit improvement at a highly cost-effective return on the investment, while a second may yield greater improvements at such a high cost that its overall cost-effectiveness is lower. In this case, the trade-off analysis should point out that the second alternative provides a higher level of benefits, but that the marginal benefits are purchased at a diminishing rate of return.
Another frequent example is the trade-off between effectiveness and financial feasibility. Often, the alternative providing the greatest improvements in transit service is also the most costly and would require a significant increase in the annual investment made by the local area. The trade-off analysis should highlight the additional commitment by the local governments – and possibly the equity implications of the means used to finance this commitment – necessary to implement this alternative.
The major task of the trade-off analysis, then, is to reduce (to the extent possible) the vast amount of information developed during the analysis to those essential differences between the alternatives. Its purpose is to frame the decision on a preferred alternative in terms of the advantages of choosing one option compared to the foregone advantages of the other options.
The recommended approach is to display key characteristics of the alternatives and the evaluation measures in clear tables that allow comparisons among alternatives within the context of their characteristics for each evaluation factor. A summary table that presents the highlights of the evaluation results should be prepared with the goal of presenting a one to two-page table highlighting the main trade-offs among the alternatives.
Some trade-off analyses have used purely qualitative judgments regarding the evaluation measures. Sometimes these are rated in terms of high, medium, and low or use “Harvey Balls” (e.g. comparison charts used by Consumer Reports™) to offer a qualitative assessment of each evaluation measure. FTA cautions against using these qualitative measures in the trade-off analysis since the scope, complexity, and the number of evaluation measures can result in trade-off analyses that are unclear, unfocused, and do not easily expose the most promising alternatives. To provide the most useful information to decision-makers, the measures should be quantitative rather than qualitative if at all possible and be expressed within a context that exposes the relative magnitude of the measures. An example trade-off analysis is provided in Table 9-3.
The purpose of documenting the evaluation methodology is simply to outline the measures that will be used to quantify the degree to which each alternative meets the stated goals and objectives. The evaluation measures and methodology must be defined at the beginning of alternatives analysis to respond to the problems in the corridor. If the evaluation measures are created after the technical work has been done, they are prone to being manipulated to support a predetermined conclusion. The alternatives analysis should specify at the outset, each objective, identify the measure(s) proposed for each objective, and describes the source of the measure. This step provides a means for local decision-makers and technical staff to agree on a meaningful set of measures, and alert the responsible technical staff of the evaluation data needed from the analysis.
While the evaluation of alternatives in the alternatives analysis does not need to use the same measures required for FTA’s New Starts rating and evaluation process, the study should be sure to produce the information required for the federal process. Otherwise, significant additional work may be required if a major capital investment becomes the locally preferred alternative and the project sponsor intends to seek federal funding.