Program Design
Begin with a Pilot
Each program will offer new lessons learned based on goals, geography, target population, and administration. Starting with a small and short-term pilot allows administrators to test the UBM concept in an area and refine the program before scaling it up to a larger or longer-term service. Scaling the pilot program will also allow evaluation teams to test the effectiveness of different approaches among different neighborhoods or user groups.
Select Participants Based on Community Priorities
Program administrators should clearly outline the objectives of the program and consider what eligibility requirements and selection process would best align with these goals. UBM goals vary from program to program, with some placing greater emphasis on transit use, others focusing on general transportation equity, and others seeking to use UBM to support new mobility services, but UBMs generally target more vulnerable segments of the population who are likely to benefit from transportation subsidies.
Specific participant selection considerations for these programs include:
Vulnerable populations: People experiencing homelessness and those currently seeking social services for health, employment, and basic goods and services may benefit most from increased affordability and access to transit and other transportation options.
Screening beyond income: While income screenings can be useful in reaching populations that may benefit most from UBMs, income may not accurately reflect individual or household needs in some cases (such as students who have no employment income but have housing or transportation benefits through school or from family). Transit agencies have used tools such as the Elder Index to accurately estimate applicants’ financial vulnerability, and benefits programs like CalFresh incorporate factors such as age, dependent status, and childcare costs into the screening process.
Likely active users: To ensure that UBM funds are spent effectively, including transportation-related characteristics in the participant selection process can be useful. Prioritizing applicants who live in vehicle-deficient households, those who already have transit cards or use transit, and those who need to regularly travel to work, school, or other essential destinations can reduce the chance of underutilization of UBM resources.
Consider what services to include:
The UBMs reviewed in this study vary widely in the types of transportation supported, ranging from subsidized services from a single provider (micromobility or public transit) to highly flexible UBMs that provide subsidies for nearly any transportation-related travel. The specific set of modes included in a UBM depends on factors such as program goals (VMT or emissions reduction, transportation equity, access to particular destinations), available services, funding requirements, fund media, and partnerships with operators, and administrators should design a program that aligns with these parameters and objectives.
- Limiting purchases to transportation: Including merchant codes that are overly broad can result in misuse of cards. For example, including grocery stores with the intent of allowing participants to purchase transit cards could result in non-transportation expenses; this may also be true to some extent with bike shops and potentially other merchant codes. A card that uses unique merchant IDs rather than merchant codes allows program administrators to select and test a specific range of expenses but gives less flexibility to participants. Some program discussions have considered using cards with no purchasing restrictions but providing program guidelines informing participants what they should and should not purchase; the viability of this approach is undetermined.
- Choose a variety of modes: The more modes supported by a UBM program, the greater insight the UBM can provide into how increased affordability affects travel decisions and frequency. While a UBM exclusively supporting transit-based modes may significantly benefit its participants, it will not address gaps in existing transit. It may overestimate the appeal of transit when measuring its participants’ travel choices. UBMs that include a suite of modes from both private and public operators that meet different end uses and support different destinations allow participants to effectively express the impacts of having increased financial access to transportation.
- Including car-based mode(s): While the UBMs reviewed as part of this study have focused on alternatives to personal vehicle travel, UBM administrators should acknowledge that private and personal vehicles continue to be the main form of reliable and safe transportation for a large portion of the population. Whether a UBM includes subsidies for car-based modes like ridehailing or carsharing, or directly subsidizes personal vehicle travel through funds for toll lanes, vehicle maintenance, or gas, support for car-based travel is an important component of true universal mobility for households with few equally viable alternatives.
Provide sufficient education and training:
For participants to fully use UBMs as intended, they must be made aware of the range of transportation options available and understand how to access and use each service. Lack of awareness or familiarity with modes can result in the underutilization of UBM funds, reduced program outcomes, or a flawed picture of how transportation funds affect travel choices. If possible, UBM administrators should offer or require training sessions with program participants before funds are distributed and should provide ongoing and attentive customer service to address participant questions and issues. Training is especially important for modes that require the use of specialized vehicles, such as scooters or electric bicycles, and modes that require mobile applications for payment information and reservations. However, training is costly and may not be feasible for larger-scale programs unless offered upon request or to a sample of participants.
Program Administration
UBM administrators face many decisions in planning, launching, and scaling a UBM program, and lessons learned from past UBMs may serve as important considerations for navigating the design options, budgets, and administrative resource requirements:
Marketing
Branding should be consistent and recognizable, such as with a clear program name and logo on UBM websites and cards, and potentially working with merchants to include this logo in certain marketing materials.
Outreach/recruitment should involve community-based organizations connected to the target population segments and who can promote the program, support enrollment, and provide supplemental resources to participants such as directing them to other existing transit discounts or benefits services that complement the UBM.The activation of cards has been a significant problem for UBMs that do not distribute in person, which is feasible for small but not large pilots. Nonactivation rates in larger pilots can range from 30% to 50%. The reasons for nonactivation are not completely known. However, the following have been identified by some pilots: an address change, confusing mailed UBM cards with junk mail, mail theft, a realization that cards do not cover personal vehicle costs when users solely depend on that mode, and not understanding how to activate the card. Some of these problems can be partially addressed, and some cannot. Administrators should consider how long they will wait for participants to activate their cards and communicate this deadline consistently to participants once they are selected. Administrators should keep a waiting list and include the cost of reissuing cards in the pilot budget.
Clear and active communication with participants is essential in ensuring that they understand how to use the UBM, are aware of any program changes or delays, and have access to answers when they have questions or issues with the program. Available resources for participants should include, but are not limited to:
Dedicated program website or web page
Clear list of services included, that is updated as needed
Newsletters or direct updates about program timelines or changes
Dedicated customer service phone number or email address, if budgets permit
Higher concentration of customer service resources during periods of changes in the service, or if a major issue is identified (such as cards being declined for certain modes or merchants)
Costs
Administrative costs can be high for initial UBM pilots due to the pilot design process, troubleshooting card functionality, and participant engagement. Administrators should budget for substantial customer service time to answer participant questions and address issues with merchants and transactions. Administrative costs can potentially be reduced for scaled UBM pilots following initial pilot periods, but smaller agencies seeking to implement UBMs may face prohibitive cost barriers that will need to be addressed through grant funding allocated to administrative activities, or support from community engagement organizations.
Mobility wallet format
There are several options for storing and issuing UBM funds, each with potential advantages and drawbacks:
Vendors: If selecting a vendor to provide prepaid debit cards, it is important to ensure that the vendor can issue cards, load funds, and make necessary changes in a timely manner. Additionally, administrators should establish data-sharing agreements between researchers, administrators, and the vendor prior to the program launch.
Payment media and backend structure: UBMs that use prepaid debit cards offer flexibility in the types of merchants supported, but involve a separate form of fare media than customers currently have in hand. UBMs that use funds stored on integrated transit cards allow participants to use the cards they already have to make mobility wallet purchases, but require additional backend work on the part of transit agencies to enable this functionality.
Integration: There may be opportunities to further integrate UBMs with other benefits media such as EBT, but there are currently technological and cost barriers to creating a fully integrated benefits card that combines transportation and non-transportation benefits. Administrators should continue to monitor the status of these technologies to identify opportunities for integration and program streamlining.
Agreements
Data-sharing agreements, purchase agreements, and other arrangements between organizations can be a time-consuming process, and UBM administrators should allow ample time in the planning process to establish these agreements and avoid disruptive delays in the UBM implementation and evaluation process.
Timeline
UBMs have been offered over varying pilot periods such as six months, one year, or 18 months. The decision of program duration can depend on the available budget, program goals, and sponsor requirements. It can take time for participants to begin to change their travel behavior after gaining access to mobility wallet funds, so very short pilots should be avoided unless this is justified by the available budget or research goal.
Program Evaluation
Program design and administration policies can affect the evaluation of UBMs, and there are several evaluation options depending on the resources and data available.
Research design
Evaluations can range from robust randomized controlled trials to basic participant group surveys or analyses of UBM transaction data. The research design depends heavily on program design and administration, and recent UBM pilot examples highlight the following considerations:
Eligibility: Evaluation will depend on whether eligibility is based on income and geography only, or includes screening for factors associated with the likelihood to use funds (e.g. zero household vehicles, TAP card status, use of toll roads). Screening for the likelihood of UBM use may reduce attrition rates and improve sample sizes to improve the evaluation's ability to show the efficacy of the treatment. However, non-screening can improve the generalizability of the treatment group to the low-income/underserved population for use in an "intent-to-treat" study to assess the effectiveness of a treatment, but may lead to higher attrition rates and require greater follow-up with participants or more oversampling.
Recruitment: Whether participants apply online, attend workshops, or are proactively selected such as through random selection of transit subsidy members can affect options for evaluation. Online applications allow the research team to review the applicant pool and recommend a random or stratified random selection of treatment participants (or assignment of treatment vs. control). However, online applications may be less representative of the population of interest due to technological barriers, barriers to program awareness, etc. In-person workshops allow program administrators to distribute cards directly to attendees once they demonstrate eligibility but create challenges in randomizing recipients or creating a control group. Using workshops to collect applicant information and then assigning treatments at a later date can preserve the integrity of the research design (i.e., randomized control trial), but could lead to attrition due to the extra step involved between workshop attendance and card distribution. In general, workshop recruitment absent a control group may be best suited to a longitudinal observational study rather than a randomized controlled trial unless strategic research-based recruitment can be implemented.
Study groups (treatment or treatment vs. control): It is considered a best practice if the pilot allows for the study of a “treatment only group”, or allows for the selection of a treatment and control group. This can be accomplished by selecting participants and using people who remain on a waitlist as a control group, or randomly assigning individuals to treatment and control. If all that is observed is a treatment-only scenario, this is observational and limits evaluators ability to assert that the treatment resulted in specific outcomes. The treatment and control scenario allows the study to attribute outcomes to the treatment but requires sufficient sample sizes (e.g. large enough applicant pool) and more recruitment effort to match both the treatment and control to the population of interest.
Oversampling: If the pilot oversubscribes participants to account for attrition this will likely result in better outcomes for evaluation. Oversampling can reduce the need to move waitlisted applicants into the treatment group, which creates group comparison challenges if the waitlist is being used as the control. However, oversampling risks overcommitting program resources if the attrition rate is lower than expected. Oversampling does not correct for self-selection bias issues resulting from attrition (the individuals who choose not to use the service may be different than the individuals who choose to use the service, in ways that cannot be measured from the demographics collected).
Geographical analysis: For studies that aim to compare urban vs. suburban or other distinct groups and geographies a large number of participants (and potential controls if included in the study) from each of the areas of interest is necessary. The recruitment strategy should align with the goal of the analysis; for example, if the recruitment includes in-person workshops, these should be geographically distributed according to the targeted sample size for each comparison group (e.g. urban and suburban).
Data sharing: Consider research needs when determining who collects and controls the data, and how it is shared between administrators and evaluators. Access to anonymous but individualized data on participant demographics, transactions, and transit use provides the best opportunity to evaluate program outcomes. Providing the research team with program application data allows researchers to recommend a method for assigning treatments (such as stratified random sampling). Access to applicant contact information allows researchers to recruit individuals directly into the research study, potentially increasing response rates. Detailed access to transaction data, particularly linked to participant demographics, enables researchers to demonstrate how an individual with given characteristics may use the card, and if combined with survey or interview data, what effect it has had on their transportation and lifestyle. Limited data access such as aggregated transaction or transit data limits the study to analyzing averages rather than individual change. Generally, the more complex the program, the more detailed the data needs to be to result in a high-quality research study.
Implementation
In connection with the research design, UBM evaluations have highlighted several evaluation implementation considerations that may affect study response, reliability, and representativeness:
Incentivization: Researchers in the UBMs reviewed for this study have found that providing adequate incentives for participating in UBM research studies (such as providing gift cards to each respondent for each survey completed) supports improved study engagement and sample sizes, potentially reducing attrition over the course of the study. This is particularly true for control group participants, who may be less likely to have an interest in contributing if they will not be benefiting from the UBM funds.
Accessibility: As with any research study, it is important to create a research design that is accessible to its participants. This is especially important for studies of marginalized or disadvantaged groups who may face language, technology, time, or other barriers to participation. Studying the program area to determine how data should be collected (online vs. in-person vs. by telephone), what language expertise is needed within the research team, and what other support participants may need to effectively participate in the study will support an equitable and representative evaluation.
Resampling and recruitment: For prepaid mobility wallets, having access to participant card activation rates and expenditures can allow evaluators to assess when it may be time to issue a second wave of cards to allow administrators to meet UBM participation targets. Tracking expenditures over time can also help to identify participants who are underutilizing funds so that administrators can reach out to identify whether there are barriers to actively using the card.
Consider travel diaries: With a sufficient baseline time period and access to contact information for the treatment group (and control group, if applicable), evaluators should consider using travel diary tools such as OpenPATH to assess how the UBM is affecting trip choices and frequency. Travel diaries require additional participant engagement and time and resources for training and analysis, but are likely more accurate than self-report surveys in measuring travel and frequency by mode.
Metrics
UBM evaluations can focus on a wide variety of participant outcomes, from mode shifts to economic benefits to health and well-being impacts. Common metrics assessed in UBM evaluations thus far include:
Expenditures: How the funds are used by mode, over time, by user, and breakdown of cards with surplus funds vs. cards with exhausted funds
Transportation security: How the funds affect participants’ ability to travel where, how, and when they want to
Transport poverty: How the funds help to overcome the experience of transport poverty, or being unable to access the transportation required to meet daily needs
Mode shift: How the funds contributed to changes in choices about what type of transportation to use, or helped participants discover new forms of transportation
Medical access: How the funds supported increased access to medical appointments, reduced instances of skipping medical care, and contributed to better overall health outcomes for participants
Environmental impact: How the funds contributed to climate-related goals such as reduced VMT and emissions
Economic indicators: How the funds affected economic factors such as employment status, student status, household income, and ability to afford different kinds of expenses
Social and emotional well-being: How the funds helped to reduce feelings of social isolation, stress, and anxiety
Analysis
Collected data can be analyzed in a variety of ways, and existing UBM evaluations point to analysis risks and opportunities that researchers should keep in mind during this phase of the study:
Data cleaning: Surveys administered to UBM participants and control groups can be fairly complex with regard to travel patterns and transportation characteristics. It is important for data analysts to review collected survey data and identify potential errors, contradictions, or inconsistencies within responses to avoid including data that skew results. Researchers can include consistency checks within surveys and create potential error flags during the analysis phase to remove responses or respondents that do not appear to reflect actual travel activity or perspectives.
Consistency across evaluations: While UBMs have been offered as fairly distinct pilots thus far, there will be opportunities moving forward to create uniformity in the types of metrics collected and methods used to evaluate these programs. Researchers should look for opportunities to collect similar data across programs to allow for comparisons between UBMs and support the development of additional best practices in the future.
