When existing sources of data on staff activities are insufficient, researchers can gather data through surveys and personal observation. This is called direct measurement.
Level of Analysis
The costs of an intervention may be analyzed at many different levels: the cost per intervention, per clinic visit or hospital stay, per patient contact, per day, and so on. The choice of analysis level should be guided by the researcher's ability to collect data with accuracy and precision, and by the cost of data collection.
Methods
Time and Motion Study
In this approach, the analyst directly observes the staff members and keeps track of the time spent on each activity throughout the day. Observing staff members can yield very precise results. Variation in staff activities may necessitate the calculation of an average day based on a random sample of many days.This can be costly because observers must be paid for their time both in training and in data collection.
Activity Logs
A second approach is to have employees keep daily activity logs for a sample of survey dates. The staff members record activities during each 30-minute interval of work (or 15-minute, 10-minute, etc.) and characterize whether the activities involved the intervention being studied, or some other activity. As with time and motion studies, accuracy and precision can be high but training is essential. Time logs carry additional administrative burdens as well: developing and pre-testing the survey instrument with allowance for staff members' input, training staff members to use the logs, and following up to ensure that logs are completed and gathered. It may be necessary to survey program managers beforehand to learn which staff members will need to complete logs.
It may not be necessary to use activity logs for every day of an intervention, particularly if it extends for weeks or months. A random sample of days will suffice, but the sampling frame must be designed with care. If an intervention becomes less intensive over time, for instance, basing an estimate on activity logs from the early days of the intervention would lead to an overestimate of total time spent.
Manager Survey
A third method for gathering staff data is to survey managers. The surveys can collect two types of information: the number of full-time-equivalent employees involved in the intervention, and the number of hours spent on the intervention per day or per week. In order to calculate staff compensation costs accurately, separate responses should be obtained for each category of employee involved: registered nurses, physicians, lab technicians, and so on.
Manager surveys are common because they take less time to prepare or complete. A single manager can report on activities of many staff members, and so another advantage is the relatively small number of people who must be surveyed. The primary drawback of manager surveys is a relative lack of accuracy and precision. Managers may have a good sense of the number of days spent on the intervention in a week, for example, but probably will not be accurate at the level of hours or half-hours. The quality of data from manager surveys depends on the effort of the mangers themselves. Manager surveys are not advisable when high precision is needed or when many managers would have to be surveyed to cover the actions of all staff members involved.
Combining Methods
It is often advisable to use two or more methods in the same study to limit resources needed to estimate costs while obtaining an acceptable level of precision and accuracy. For example, an analyst might use staff surveys or study logs to determine the number of times an intervention occurs. Precise time estimates for the intervention would then be obtained by directly observing staff members on a small number of days.
Calculating Employment Costs
Once time spent on an activity has been determined, the next step is to assign a cost to that time. Although hourly or annual earnings may be obtained through surveys, they will not be accurate guides to the total employment cost. Benefits, taxes, and time spent on overhead activities are all parts of the true employment cost; yet employees may have little information on costs incurred by their employer.
It is straightforward to determine hourly employment costs. The first step is to determine annual labor costs, including both wages and benefits, assuming a 40-hour workweek. To find the raw hourly cost, divide the annual cost by 2088, the number of hours in a 52-week work year. The raw figure includes time spent on activities other than patient care such as vacation, sick leave, and administrative work. Because such non-applied time must be spent in support of carrying out an intervention, it is necessary to adjust the hourly cost to reflect this extra cost. For additional information, see the HERC webpage "Cost of Employing Specific VA Staff".
Other Costs
The costs of supplies and equipment may be gathered through manager surveys or by contacting manufacturers. Two factors should be kept in mind when obtaining supply costs: supply and equipment costs may fall if a new intervention is widely adopted, and the list price of a good may greatly overstate the cost of supplies and equipment because large providers like VA frequently negotiate substantial discounts.
VA capital costs through FY2004 were available in the Cost Distribution Report (CDR), which provided the depreciation on VA buildings and equipment, but omitted the cost of financing (Barnett 1999). The CDR has been replaced by the Monthly Program Cost Report (MPCR). It is not reconciled to reliable data sources and thus should be treated with caution. VA employees may access CDR and MPCR data through the Austin Information Technology Center (AITC), formerly known as the Austin Automation Center.
One must also account for the cost of space. The value of the next-best alternative use of a building can be determined by the cost of renting similar facilities or by the replacement cost of the VA facilities used in the study. Estimates of land values and rental rates for medical office space may be obtained from real estate agents or other local sources. The replacement cost of current VA facilities may be estimated through a combination of VA financial data and proprietary commercial data (Rosenheck, Frisman, and Neale 1994).
Accuracy and Precision
Accuracy in gathering data is an important consideration. Even small errors in reporting can accumulate if many separate people take part in the intervention. If an activity log requests staff members to list tasks in 15-minute intervals, how will they record interventions that take 5, 10 or 20 minutes? Total intervention time will be underestimated if they round down to the nearest 15-minute interval or overestimated if they round up. Small individual errors can become large if the same upward or downward bias is repeated many times. Solutions include using a more precise measurement system that collects data in 5-minute intervals, using direct observation by a third party who can note the exact time spent, and asking staff members to tally which actions occurred and then assigning each action an average time based on a few direct observations.
The level of precision needed in data collection will depend on the intervention. Consider an intervention that takes 15 minutes to perform. In an outpatient setting it may be sufficient to use a survey that records time in 15-minute intervals. The overhead cost of surgery suites is typically billed by the minute, however, and so for surgical interventions the instrument would need to record time at the minute level. Billing methods are thus a second guide when designing survey instruments.
The Comparator
In a cost-effectiveness analysis an intervention will be measured against a comparator, whether "usual care" or another new treatment. This highlights the need to clearly define the production process of the intervention, the steps taken by each person involved and the expected amount of time devoted to each. Staff time for the intervention and comparator arms must be measured in similarly accurate and precise ways. Researchers must scrutinize data collection methods to avoid bias that might favor one treatment arm, given possible incentives for patients or providers.
Joint Production
In some instances a single product is produced simultaneously with other products. Consider the time of a nurse involved in a clinical research trial. Suppose that patient care activities unrelated to a research protocol take up 25% of the nurse's time; activities which benefit both research and patient care take 50% time; and activities only needed for the research protocol take the remaining 25%. An analyst could justifiably assign as little as 25% or as much 75% of the costs of this time to research. Which figure is appropriate depends on the question being asked. For example, the percentage of time that would be released if research activities ceased is only 25%.
Additional Information
The HERC web site features several documents with information on direct measurement: the HERC micro-costing webpage, an article by Barnett published in a 2003 supplement to Medical Care Research and Review, and an article by Barnett from a 2009 supplement to Medical Care.
Reference
Rosenheck R, Frisman L, Neale M. Estimating the capital component of mental health care costs in the public sector. Administration and Policy in Mental Health 1994;21(6):493-509.
Last updated: March 20, 2023
