Before and After Study Qs and As
What is a Before and After Study?
FTA’s Final Rule on Major Capital Investment Projects (December 2000) includes a provision whereby sponsors seeking a Full Funding Grant Agreement (FFGA) for their New Starts project must submit to FTA a plan for the collection and analysis of information leading to the identification of the impacts of the project and the accuracy of the forecasts which were prepared during project planning and development. As a condition of receiving a FFGA, project sponsors must commit to carrying out the defined elements of the aforementioned plan, resulting in the completion of a Before and After Study.
The Before and After Study has two distinct and important purposes: (1) to expand insights into the costs and impacts of major transit investments; and (2) to improve the technical methods and procedures used in the planning and development of those investments.
To accomplish the first purpose - insights into costs and impacts - the study identifies the actual costs of the new project and its impacts on transit service and ridership. The study isolates these actual costs and impacts by comparing the conditions that prevail after project implementation to the conditions that existed before implementation.
To accomplish the second purpose - improvements to technical methods and procedures- each study examines the accuracy of predicted costs and impacts. The study determines accuracy of the predictions by comparing the conditions that prevail after project implementation to the costs and impacts predicted for the project in each phase of the planning and project development process.
Before and After Studies address both purposes through a careful technical analysis undertaken by sponsoring transit agencies in cooperation with other local planning entities and FTA. This arrangement ensures authorship by local agency staff with first-hand knowledge of the project and its development, buy-in of individual project sponsors and the broader transit industry, and consistency with national standards for the analysis.
Why is FTA requiring the Before and After Study?
The Government Performance and Results Act (GPRA) of 1993 requires that federal agencies improve the effectiveness of their programs and their accountability to the public by focusing on results, service quality, and customer satisfaction. The Act further requires that federal agencies improve program management and congressional decision-making by assembling objective information about program results and achievement of statutory objectives. FTA believes that the Before and After Study will assist the agency in meeting the intent of GPRA and to carry out its responsibility to document the accomplishments of the New Starts program.
As importantly, FTA is requiring the Before and After Study in order to encourage improved practices in data collection, documentation, and analyses of pre- and post- investment transit performance; ensure that the planning and project development process generates reliable information for decision making; and enhance its technical support program. FTA believes the conduct of Before and After Studies on all future FFGAs (and FFGA amendments) will provide for an invaluable repository of information and experience which will benefit the entire transit industry.
What are the benefits of the Before and After Study?
The information derived from Before and After Studies will provide the following benefits:
- strengthen the New Starts program by highlighting the successes of individual transit capital investments and the important role that transit plays in improving mobility and the quality of life in communities throughout the Nation;
- identify and transfer the lessons learned in planning, implementing, and operating transit fixed guideway investments to agencies planning similar projects. Information generated from the Before and After Study will enable the sponsors of future New Starts projects to build upon the experiences of past projects, including design and operational features that have proven successful, while avoiding options that have been less successful;
- generate information that can be used to improve the prediction of costs and impacts of major transit capital investments. Better estimates of costs and benefits will result in improved information to support local decisionmaking and Federal funding recommendations;
- identify the strengths and weaknesses in local travel demand forecasting and capital and O&M cost estimating procedures, and provide insight into how technical methods can be improved to support decisionmaking for future projects;
- "imbed" data assembly and analyses which measures predicted and actualized project costs and impacts into the project planning and development process;
- accumulate a source of technical information on the actual costs and performance of major transit investments.
Furthermore, because the cost of the Before and After Study is included in the baseline cost estimate (BCE) of the FFGA, agencies undertaking the Study will benefit from Federal financial participation (of up to 80 percent of total costs) in a comprehensive data collection effort which will be useful for a wide array of local transit planning and performance monitoring activities.
Who must conduct a Before and After Study?
Before and After Studies must be undertaken by all project sponsors receiving a new, or amending an existing, Section 5309 New Starts FFGA. Local agencies implementing major transit capital investment projects without New Starts funding are encouraged to conduct Before and After Studies or similar analyses, and will receive technical assistance from FTA in such efforts.
What data and other information is being collected and analyzed?
Each Before and After Study is intended to examine five key project characteristics: (1) physical scope; (2) service levels; (3) capital costs; (4) operation and maintenance (O&M) costs; and (5) ridership and revenues. FTA has identified these five characteristics because of their central roles in the performance of New Starts projects. Capital costs and ridership are typically the most visible numbers associated with each project. Capital costs, O&M costs, and farebox revenues are key elements of the financial plan. Changes in the physical scope made during planning and project development are the most frequently cited source of capital cost changes. Overall service levels establish the context for changes in O&M costs and ridership, while service levels on the fixed guideway and feeder buses define the quality of service available to transit riders. Ridership increases reflect the magnitude of impacts on congestion, air quality, and a variety of other benefits.
Individual project sponsors may choose to examine other project impacts as well. Other impacts which might be of local interest include changes in land use policies, land use patterns, transit funding, economic development, accessibility to jobs, and system operating efficiency. The assessment of any additional characteristics is left entirely to the discretion of the project sponsor.
What is the local process for undertaking the Before and After Study?
FTA desires that each Before and After Study include information on the five (or more, at local discretion) project characteristics for three specific project milestones: (1) during planning and project development when the predictions were prepared, (2) immediately before implementation of the project, and (3) two years after the project opens to service. Each of the milestones are described as follows:
- Planning and Project Development Predictions - the forecasts of project characteristics available for decision making at three key points – the conclusion of alternatives analysis, the conclusion of preliminary engineering, and the signing of the FFGA;
- Before - the actual conditions that exist immediately before implementation of the project, either just before opening of the project to revenue service or, if construction is likely to introduce major disruptions of transit service and ridership patterns, just before that disruption occurs; and
- After/actual - the actual conditions that exist two years after the project has opened for revenue operation.
These three milestones support the two aforementioned major elements of the analysis: (1) comparisons of actual conditions before and after project implementation to understand the impacts of the project; and (2) comparisons of predicted and actual conditions to examine the reliability of the predictions.
Individual project sponsors may choose to examine information available at other milestones, as well. An additional milestone during project development might be useful, for example, if a major reassessment occurred after the alternatives analysis and resulted in a significantly revised project. As a second example, an additional milestone well after start-up of operations might be useful in order to capture longer-term impacts on land use that might not have matured at the 2-year milestone. Identification and incorporation into the study design of any additional milestones is left entirely to the discretion of the project sponsor.
In the analyses phase of the Study, the project sponsor compares the actual project characteristics to those that were predicted and to those observed before the project was built. Once such comparisons have been made, the subsequent analyses focus on deviations between predicted and observed project outcomes. This involves a review of the variables used in forecasting procedures, postulating hypothesis about the causes of error, and assembling information to confirm and refute these hypothesis. The project sponsor presents a final report of the results of the Study analysis to FTA, which will concur in the findings. FTA further intends to synthesize the results of multiple local studies in order to document the impacts and accomplishments of the New Starts program and to highlight strengths - and suggest improvements to identified weaknesses - of transportation planning and forecasting tools and methods.
What are the responsibilities of the project sponsor and FTA in the conduct of a Before and After Study?
Sponsors of New Starts projects receiving FFGAs are responsible for the development of a plan for undertaking the Before and After Study and the actual conduct and documentation of the Study. For new FFGAs, the Study "Plan" shall be prepared during final design. It is important to note that the "predicted" element of the study requires that information pertaining to each of the five project characteristics be documented at even earlier points in the planning and project development process. Consequently, collection of the data attributes which contribute to the predicted impacts of the New Starts investment must be considered throughout both alternatives analysis and preliminary engineering, with appropriate coordination with other local participants in the planning and project development process.
FTA will approve the Before and After Study Plan and will provide technical assistance throughout the Study process. FTA’s project management oversight (PMO) and financial management oversight (FMO) resources will be engaged in verifying information and results of analyses through their ongoing participation in the project. Data assembled by PMO contractors may be helpful to the Study, and will be shared with its sponsor.
How is the Before And After Study different for new FFGAs vs. amended FFGAs?
The two goals of the Before and After Study – (1) understanding the impacts of major transit capital investment projects and (2) measuring the accuracy of forecasts made during development as a means of improving technical planning methods and tools – remain the same for both new and amended FFGAs.
For new FFGAs, the Before and After Study Plan is prepared during final design, and approved by FTA in advance of entering into the funding agreement. The scope of data collection activities is well-defined and the costs of undertaking the Study are reflected in the BCE of the FFGA. Moreover, because such projects have recently completed planning and subsequent preliminary engineering and design activities, information relating to project forecasts developed during those phases should be readily obtainable. It is expected that projects pursuing FFGAs in the very near future will be fully documenting all relevant information and data pertaining to predicted costs and benefits throughout the planning and development process, in anticipation of the Before and After Study requirement.
For project sponsors seeking to amend an existing FFGA, FTA suggests that development of a Before and After Study Plan begins as soon as possible so it does not delay execution of the amendment. The plan should consider the availability of information developed as part of planning and project development, which may have been completed several years ago. As with new FFGAs, study costs may be included in the redefined BCE. It is important to note that the Before and After Study for FFGA amendments must be undertaken for the entire project under FFGA, and not just for the portion of the scope which is being amended.
What will compliance with Before and After Study requirements cost?
Costs for undertaking the Before and After Study will depend on a number of factors, including the complexity of the project’s planning and development and the availability of data and other information necessary to carry out the Study (much of what is required for the Before and After analyses may already be collected as part of local transit planning, monitoring, and evaluation activities). A great deal of information required for the Before and After Study involves documenting all project-related activities, predictions (specifically related to the five characteristics: project scope, service levels, capital costs, operating costs, and ridership), assumptions, decisions, and outside influences that have shaped the project at the time of the FFGA, and then monitoring these variables throughout the construction and eventual operation of the project.
The Before and After Study also requires information on transit service levels and ridership before and after implementation of the project. This will typically require two on-board surveys – one at a point just prior to implementation of the New Starts investment, and the other approximately two years after its opening. Depending on the complexity of the project and the magnitude of transit service provided both before and after its implementation, such surveys could be significant undertaking. Where transit systems have previously implemented an ongoing data collection program which captures trip origins and destinations and socioeconomic characteristics of the transit market, this program may provide the basis for assembling the needed ridership information. In any case, the collection of transit ridership and market data will typically be the most resource- intensive activity of the Before and After Study with costs which may approach 1-2 million dollars, depending on the magnitude of the local data requirements. As noted previously, FTA funding may be used to cover up to 80 percent of the costs of performing the Before and After Study.
What additional assistance is/will be available?
FTA staff and contractors can provide additional technical assistance on the preparation of a Before and After Study Plan and the execution of the study itself. FTA expects to issue technical guidance on the Before and After Study by February 2003. More specific guidance on on-board ridership surveying techniques, capital cost tracking, and other Study-related activities may also be issued in the near future.