Audit Studies

From WolfWikis

Overview

Audit Studies

Audit studies measure discrimination directly with experimental field-work. Subtle discrimination can be hard to detect. Was a person rejected for a job because of the color of their skin or a poor interview? Audit studies overcome this, by presenting the subjects of their studies with two nearly identical candidates who differ in one only characteristic like, race, age or gender. Thus, the differential treatment of these auditors by the subjects of the study can be directly attributed to discrimination. These studies provide powerful evidence of discrimination, literally catching discriminators in the act ^[1]. By providing a systematic picture of discrimination, audit studies disprove notions that racism is merely a lingering product of a previous era, but rather that it is still an ongoing phenomenon today ^[2]. There are, however, serious ethical concerns and a number of methodological problems. Moreover, while directed at revealing discrimination, Audit studies do little to reveal causes of discrimination ^[3].

There are two main kinds of audit studies, in-person audits and correspondence tests. In-person audits are conducted by recruiting auditors, participants of the study who will take the role of job applicants or housing customers. These auditors are then provided with carefully constructed resumes and histories, coached in interviewing and paired with one another. During the research process, auditors are debriefed and continuously coached on better matching their performances. Discrimination can then be directly experienced and measured by the auditors.

Correspondence tests differ from in person audits in that they seek to eliminate the human element. Resumes are paired together, constructed to contain almost identical information, and then sent to employers separately and at different times. Caution is taken to ensure that resumes which are paired together are nearly identical but not so identical as to alert the employer or other subject. Characteristics, like race, are signaled by indicators on the resumes. For example, Bertrand and Mullainathan ^[4] used applicant names as an indicator of race, specifically names like Lakisha Washington to denote a black applicant and Emily Walsh to denote a white applicant. Discrimination is measured by the number of employer responses.

Advantages and Disadvantages

Auditor Non-Alignment

One of the most significant problems with discrimination is the difficulty of matching auditors. There are subtle differences between any two people, posture, the length of time they make eye contact, the quickness of verbal responses. Such differences can affect results, and thus can be the cause of not receiving a job offer rather than discrimination, which threatens internal validity. To avoid this auditors must always be carefully selected and extensively and continually coached. It can also be useful to measure potential auditor non-alignment by comparing results across pairs.

It is possible in some cases to control for non-alignment by switching roles. Auditors investigating discrimination against people with criminal records, for example, have pairs alternate playing the person with the criminal record ^[2]. Statistical results would then be independent of non-alignment. Obviously, this is less feasible with race or gender. Correspondence tests offer a possible solution. By eliminating the personal role resumes can be more closely matched. Characterizing the complex decisions involved in matching auditors, ^[2]. suggests that the matching process be considered an art as much as a science.

Auditor Bias

In-person audits face the particularly difficult problem of finding good auditors. Auditors must be motivated, skilled at deception, and committed to an objective research process, not to mention closely matched with each other ^[2]. It is possible for auditors to give subtle, even unconscious, clues to subjects, creating self fulfilling prophecies. Historically college students have often been employed as auditors or subjects which leads to potential biases, as college students are not typical applicants for many jobs nor are they typical employers ^[5]. ^[6].

Moreover, Heckman and Siegelman ^[7]. point out that race and ethnicity cannot be randomly assigned and therefore many audit studies are not true matched pair studies. Correspondence studies avoid this problem by randomly assigning race or ethnicity to different resumes. However, race and ethnicity are not simple binary characteristics; there are degrees and subtleties of racial and ethnic identities. Audit studies may miss subtle differing degrees of discrimination. Broadening the number of audit pairs, may reduce auditor bias. Yet, this is not without cost.

Cost / Scope

The in-person audit study is expensive. Numerous auditors must be employed full time, with support and training. Cost may necessitate a limited scope of study, limiting generalizability and threatening statistical validity. Further, budget constraints can worsen the impact of auditor non-alignment. As fewer and fewer auditors are employed the potential for non-alignment grows. Again correspondence studies offer a possible solution, allowing for greater scope at less expense ^[8].

Correspondence studies, however, face their own limitations in scope. Many higher paying jobs require face to face interviews. A simple resume process is not really representative of the average job search. There is the possibility that correspondence studies under-report discrimination, because discrimination is more likely to happen in a face to face situation at a subliminal level. Moreover, correspondence studies are impractical in measuring discrimination in housing practices ^[9]. Perhaps most importantly, while correspondence studies offer broader scopes, they abandon that moment of discrimination captured in the in-person audit. One possible solution is a multi-method approach incorporating both an in person and correspondence study ^[2].

Exaggerated significance

By closely matching auditors to each other and then having them apply for the same job, Audit studies potentially exaggerate the importance of discrimination in hiring practices. Faced with closely matched applicants employers will search for minute criteria on which to decide. Applying to jobs with multiple positions, in which both auditors, could be hired, is one possible solution. Auditors, however, are often presented as very qualified. This could lead to a situation in which both auditors are hired in the majority of the cases, disguising the presence of discrimination.

Ethical Concerns

A major ethical concern surrounding audit studies is the fact that it is impossible to obtain informed consent from the study subjects. Even more, debriefing of the subjects is often avoided after the study has taken place. Potential risks this can pose on the subjects are the loss of time and legal liability, should identifying information on the companies or individuals involved ever be uncovered. For this reason it is important to take every possible step to protect all identifying information on the subjects so that even a mere association with an audit study on discrimination cannot be made ^[2].

References

1. ↑ Yinger, John. 2001. “Measuring Racial Discrimination with Fair Housing Audits: Caught in the Act.” The American Economic Review. 76(5):881-893.
2. ↑ ^{2.0 2.1 2.2 2.3 2.4 2.5} Pager, Devah. 2007. “The Use of Field Experiments for Studies of Employment Discrimination: Contributions, Critiques, and Directions for the Future.” The Annals of the American Academy of Political and Social Science. 609:104-133.
3. ↑ Neumark, David. 1996. “Sex Discrimination in Restaurant Hiring: An Audit Study.” Quarterly Journal of Economics. 111(3):915-942.
4. ↑ Bertrand, Marianne and Sendhil Mullainathan. 2004. “Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination.” The American Economic Review. 94(4):991-1013.
5. ↑ Heckman, James J. and Peter Siegelman. 1993. "The Urban Institute Audit Studies: Their Methods and Findings." Pp. 187-258 in Clear and Convincing Evidence: Measurement of Discrimination in America, edited by M. Fix and R. J. Struyk, Washington, D.C.: Urban Institute Press.
6. ↑ Correll, Shelley J. 2007. “Getting a Job: Is There A Motherhood Penalty?” American Journal of Sociology. 112:1297-1338.
7. ↑ Heckman, James J. and Peter Siegelman. 1993. "The Urban Institute Audit Studies: Their Methods and Findings." Pp. 187-258 in Clear and Convincing Evidence: Measurement of Discrimination in America, edited by M. Fix and R. J. Struyk, Washington, D.C.: Urban Institute Press.
8. ↑ Bertrand, Marianne and Sendhil Mullainathan. 2004. “Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination.” The American Economic Review. 94(4):991-1013.
9. ↑ Yinger, John. 2001. “Measuring Racial Discrimination with Fair Housing Audits: Caught in the Act.” The American Economic Review. 76(5):881-893.