Skip
To Content
|  | |||||||||||
 |
|
|||||||||||
|
While every step was designed to help collect the highest quality data possible, the 2000 NHSDA included several specific quality control programs which are described in this chapter.
Each field interviewer had at least one regularly scheduled weekly telephone conference with his/her Field Supervisor. During this call, the FI reported progress made toward completing the work; reviewed Production, Time, and Expense (PT&E) information for the week; discussed field problems; and asked any questions that had emerged during the week. The FS then provided feedback on the progress and quality of work and offered solutions to problems or questions encountered.
Regular weekly telephone conferences also were held between the Regional Supervisor and each of the FSs in his/her territory. FI production and performance were discussed during these conferences, as were budget considerations and any problems that were occurring.
Beginning at training, FI and FS performance was monitored closely and consistently throughout the field period. Training classrooms were small enough to observe and evaluate each FI's individual performance and comprehension. The classroom trainers worked together to evaluate FIs on a daily basis. Reports of struggling FIs were given to the Site Leader daily to help identify problems and develop resolution plans. These evaluations also ensured that those FIs who were struggling but willing and capable of doing the work would receive the necessary help after training to interview successfully on the NHSDA. FIs needing extra help at the close of the training session were identified, and further training was provided by the FS. These FIs also were monitored more closely as they began their assignments.
At the end of every quarter of data collection, each FS evaluated the FIs in his/her region to decide how to allocate bonus funds and whether to recommend any merit-based pay raises. FSs considered all the facets of being a "good FI," including production, response rates, adherence to procedures, costs, timeliness, attitude, commitment, attention to details, lack of data quality errors, and willingness to take on additional work (particularly to work on hard refusals). To decide how to divide bonus funds, the FS ranked each FI. Additionally, pay raises were not necessarily related to bonus money; an FI might not receive a bonus but could still be eligible for a raise. For both bonuses and pay raises, RSs and RDs reviewed the FS's decisions.
At the end of the calendar year, each FS used a standard RTI multiple-choice form to generate an interim evaluation of FIs who were active on the NHSDA. The FS used this same form to provide a final evaluation of FIs who "attrited." Completed evaluations were added to the interviewer's personal data file at RTI. The FS generally completed this form without RS or RD input.
Every month NHSDA management personnel received a listing of those field interviewers who had voluntarily chosen to leave the project (those terminated did not appear on this list). The listed FIs were contacted and a short questionnaire was administered (see Exhibit 8.1) to determine the main and secondary reasons they left the project. These data were then keyed and used to produce a quarterly report for project management summarizing the reasons. Exhibit 8.2 contains the total results for all FI exit interviews conducted during 2000.
Each FS was equipped with a laptop computer and given access to the NHSDA Web-based Case Management System. FIs transmitted screening data daily from the Newton, including record of calls data, verification information for non-interview cases, added DUs, address updates, and cost data. When the Newton screening data was transmitted to RTI, it was checked by the control system's defined consistency checks, then was posted to the CMS for monitoring purposes. The completed interview data was transmitted to RTI by FIs from their laptop computers and checked against screening data to ensure each completed case was received and that the correct respondent was interviewed.
The FS System on the CMS included the following data quality functions:
The Data Quality Report displayed various data quality issues and allowed the FS to provide specific feedback to FIs who were experiencing problems. The report included missing data items on Verification Forms and procedural errors such as Case ID or Verification ID problems. The report also included a list of cases that could not be used due to the FI interviewing the wrong household member.
The Missing Screening Data Report displayed the screening data, listed by FI, that was missing for specific Case IDs. FSs used this report to monitor the quality of the screening data that each FI collected. The data on this report represented information that the respondent refused to provide or indicated areas where the FI either made errors or may have been taking short-cuts. FSs monitored specific problems and trends and were able to provide immediate feedback and re-train FIs as necessary.
FSs used the Overdue Case Report to account for completed interviews that should have already arrived at RTI. Interviews were considered overdue if not transmitted within three days of the date of interview (as reported by the Newton Record of Calls data).
Cases displayed on this report were investigated to ensure the completed interview was transmitted or that the correct Case ID was used and reported as a completed interview. FSs and programming staff worked to resolve any pending issues with overdue cases.
The Length of Interview Report listed the completed interviews that were either finished in a relatively short or extremely long amount of time. The times were derived from the CAI interview file (total time and timing of specific sections) so that FSs could monitor possible problem situations (such as short-cutting or problems with the laptop that might cause the time-frame to be strange).
The Case Data Information portion of the CMS provided all FI production data and allowed the FS to interact with the data and view it in special ways. The type of cases the FS viewed was determined by the drop-down items selected. Each of the following items were available to select (single or multiple items), after which a data table containing all of these items (for the subset of cases) displayed:
There were special features within this function that displayed additional data:
The data provided in this table allowed the FS to evaluate many aspects of the FI's work.
The Filter Record of Calls allowed the FS to view the FI's record of calls events by filtering on the following items:
The FS could analyze the FI's work pattern and spot instances where an FI might have entered "false" results.
The Data Quality Team was responsible for the identification, resolution, and distribution of information to field staff concerning data quality and verification issues. For the first two quarters, the Data Quality Team was made up of one leader who supervised three Data Quality Specialists as they monitored the data quality of specific regional areas. The team leader also interacted with supervisors in RTI's Telephone Survey Unit (for verification issues), and data receipt and data preparation units to oversee data quality issues.
Beginning in July of 2000, the team shifted to a decentralized staff and expanded from three to six Data Quality Coordinators who reported to the Data Quality Task Leader. Each Coordinator reported directly to one or two RDs the results of the in-house data quality tasks, verification task completion, and interpretation of the results. They also assisted the RDs in planning additional or more complete field verification steps as necessary.
A special verification training session was held June 29-30, 2000 in RTP, NC to prepare the Data Quality Coordinators for their new roles. The topical areas covered were the Verification Web Pages, the Verification Reports, Problem Sheets, and Field Verification procedures and protocol.
In order to verify the quality and accuracy of the FIs' work, a complex verification procedure was implemented. This involved the selection and verification, predominately by telephone but sometimes by mail, of at least 15 percent of final interview cases, and at least 5 percent of final non-interview screening cases. Additional work could also be selected for the verification process—field management staff could elect to have 100 percent of the FI's work verified, or they could select an individual case or a group of specific cases to be verified beyond what was randomly selected. These verification contacts were made with the actual respondent.
Verification information for completed interviews was obtained from the Verification Form completed by each interview respondent (see Exhibit 8.3). For the final non-interview screening codes of 10 (vacant), 13 (not primary residence), 18 (not a dwelling unit), 22 (dwelling unit contains only military personnel), 26 (not eligible for the quarter), and 30 (no one selected for interview), the contact information was recorded immediately in the Newton at the time the case was finalized. For codes 10, 13 and 18, the contact was made with a knowledgeable person, such as a real estate agent, property manager, or neighbor. For codes 22, 26, and 30, the verification was completed most often with the screening respondent.
The telephone verification was conducted by specially trained staff in RTI's Telephone Survey Unit (TSU). Spanish translations of all materials were available for verifications with Spanish-speaking respondents. Again, most of the selected code 70s, and all of the selected codes 10, 13, 18, 22, 26, and 30, were verified by TSU. The NHSDA telephone verification script used depended on the final status code of the case (see Appendix H).
For those selected code 70s that did not have a telephone number on the Verification Form but did have an address, verification by mail was attempted. The mail verification letter (see Exhibit 8.4) was sent to the respondent to complete and return by mail to RTI. The completed verification letters were keyed, and the results were displayed in the CMS and on the Verification Reports.
TSU Verification had two stages. For most problems found during the initial call, a follow-up call was made to confirm the problem. That follow-up call was made the following week by the Verification Team, an elite group of telephone interviewers who were trained on all project procedures and protocols.
During the follow-up call, the Verification Team member conversed with the respondent (as opposed to following a script) in an attempt to confirm or resolve the identified problem(s). The Verification Team member had the problem (or list of problems) identified for that case during the TSU verification interview as well as a bulleted list of items (a subset of items from the TSU questionnaire) that must be covered during their conversation with the respondent. The result of the call was either a confirmation that the problem (or additional procedural problems) occurred during the screening or interview or a resolution of the problem by clarifying the issues with the respondent. If any other important information was obtained during the follow-up call, the Data Quality Team distributed this information to the FS and RS.
Table 8.1 provides a summary of the results of the in-house TSU verifications for CAI interviews and non-interview screening codes 10, 13, 18, 22, 26, and 30.
In addition to the verification procedures conducted on completed work received in-house, additional steps were taken in the field to ensure complete and accurate collection of data. This field verification was generally initiated after one of four circumstances occurred:
The Data Quality Team worked with the FS and RS to select the cases to be field verified. These finalized cases were transmitted to the Field Verifier's Newton (either the FS or another FI conducting the field verification) so that the screening data could be verified. The Field Verifier returned to the SDUs that were assigned and queried the respondent in an effort to determine whether or not proper contact had been made by the FI in question. The Field Verifier also verified the screening information. If an interview had been completed, the Field Verifier confirmed some of the demographic data from the interview with the respondent. The Field Verifier also reviewed some protocol issues with the respondent to ensure the FI had followed protocol and acted in a professional manner. Results of the field verification were reported to the Data Quality Team and the FS, RS, and RD. If the Field Verifier found the work to be invalid, he or she reworked the case.
In general, the need for such in-field verification was limited, but it did occur. In the 2000 NHSDA, a total of 1,159 cases were selected for Field Verification. This process led to the identification and termination of FIs who were determined to have submitted fraudulent work. All their work was verified and reworked as necessary. A total of 80 invalid interviews and 208 invalid screenings involving 20 FIs were identified via in-person field verification. All 20 FIs were terminated.
8.4.3.1 Case Data Information Link
The Verification Status on the Case Data Information link on the CMS allowed project staff to view the verification status of each case and monitor trends across status codes or areas. The following Verification Status codes were used to monitor the verification at the case level:
NF: No Form (Code 70s)
NP: No Phone
RE: Refusal—not selected
NS: Eligible, but not randomly selected for verification
ST: Selected for TSU (Telephone) Verification
SF: Selected for Field Verification
SM: Selected for Mail Verification (Code 70s without phone numbers)
OK: Completed Okay
UC: Finalized—Unable to Contact
UN: Finalized—Unresolveable
SS: Completed—Some shortcuts
IR: Completed—Invalid, then reworked
IW: Completed—Invalid, not reworked
Since verification selection was random, it helped to see which cases had been selected. If project staff wanted additional cases to be selected for verification, they worked with their region's Data Quality Coordinator to select additional cases to be flagged for verification.
8.4.3.2 Verification Short FI Level Report (Pages 1 and 2)
The Verification Short FI Level Report provided a snapshot of the problems identified during TSU Verification and Mail Verification. Page one provided a summary of data for a subset of codes: 10, 13, 18, 26, and 30. Displayed were the number of cases of these status codes that had no form (code 70 only), no phone, refused, percent of cases with no form/phone (once greater than or equal to 30 percent), percent of cases refused (once greater than or equal to 30 percent), count of other ineligibles, count of eligibles, count of selected for TSU, and count of selected for mail. From this data, supervisors could see if an FI had a high percentage of cases with no phones, no forms, refused, and how many have been sent to Mail Verification (which is not as successful as Telephone Verification in obtaining a response).
More specific details of the problems displayed on page one were contained on page two of the report. The second page displayed each problem identified during TSU and Mail Verification. A case could have multiple problems, so all problems for all cases were displayed here to track trends related to possible shortcutting. There were 49 Problem Codes divided into four groups by Screening and Interview Result Code (Exhibit 8.5).
A team of specially trained industry and occupation coders worked to classify each respondent's job as described in the interview. Using the information recorded, a coder assigned a three-digit industry classification code and a three-digit occupation code from the 1990 Census Alphabetical Index of Industries and Occupations. Independently, another coder also worked the case. A computer program compared the assigned codes and forwarded those with differences to "adjudication." During adjudication, a senior coder reviewed all the available information and assigned final codes.
To provide feedback and share information with all coders, bi-weekly quality circle team meetings were held to discuss cases that had gone to adjudication. As the adjudicator led the group through the process of reaching the correct code, coders could increase their knowledge base.
Code 70s |
Non-interview Screening Codes |
|||||
Overall Result |
Verified by TSU |
Verified by Mail |
Code 50 |
Verified by TSU |
Code 50 |
|
Q1 |
5,737 |
5,556 |
181 |
3,803 |
5,756 |
4,371 |
Q2 |
6,156 |
5,921 |
235 |
4,106 |
6,286 |
4,922 |
Q3 |
6,063 |
5,801 |
262 |
4,212 |
6,877 |
5,119 |
Q4 |
4,555 |
4,343 |
212 |
3,196 |
4,512 |
3,551 |
Total |
22,511 |
21,621 |
890 |
15,317 |
234,311 |
17,963 |
2000 National Household Survey on Drug Abuse (NHSDA)
Project 7190
A. Contact Information |
||||||
FI Name: | ||||||
FI ID Number: | ||||||
Hire Date: | ||||||
Termination Date: | ||||||
Home Address: | ||||||
Home Telephone: | ||||||
Work Telephone: | ||||||
Field Supervisor: | ||||||
| B. Record of Calls | ||||||
Time |
||||||
Date |
Day of |
a.m. |
p.m. |
Comments |
Result |
FI ID |
C. Introduction |
||
Hello. My name is ________________ and I work for the Research Triangle Institute which is located in Research Triangle Park, NC. According to our records, you worked for us recently as a field interviewer on the NHSDA Project. (NHSDA is the abbreviation for National Household Survey on Drug Abuse) |
||
This large national study depends on high quality field staff to gather the information. Anytime one of our interviewers elects to leave the project, we are always interested in knowing why. We would like to ask you a few questions about your experience on the NHSDA and to learn why you chose to leave the project. Is now a convenient time for you? This will only take a few minutes. |
||
|
||
|
||
|
Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes |
No No No No No No No No No No No No No No No No |
(READ THE REASONS MARKED IN QUESTION 6 ABOVE, IF NECESSARY.) Enter the "letters" from Question 6 above that the interviewer selects =>____________ I want to thank you for your time. The NHSDA management staff certainly appreciate your willingness to provide answers to these questions. Have a nice day/evening. D. Interviewer Notes |
||
Code 70 Problems |
|
1 |
Incorrect phone number for address |
Code 30 Problems |
|
30 |
R unknown and not correct phone number for the SDU OR incorrect phone number for the SDU |
Code 22 Problems |
|
50 |
No known contact with FI |
Code 10, 13, 18, 26 Problems |
|
60 |
No one familiar with the address |
|
This page was last updated on December 29, 2008. |
|
SAMHSA, an agency in the Department of Health and Human Services, is the Federal Government's lead agency for improving the quality and availability of substance abuse prevention, addiction treatment, and mental health services in the United States. 
* PDF formatted files require that Adobe Acrobat Reader® program is installed on your computer. Click here to download this FREE software now from Adobe. |
