Justifying the adequacy of samples in qualitative interview-based studies: Differences between and within journals

Justifying the adequacy of samples in qualitative interview-based studies: Differences between and within journals Prof Julie Barnett a, Konstantina Vasileiou a, Dr Susan Thorpe b, Prof Terry Young c a University of Bath, Department of Psychology b Newcastle University, School of Psychology c Brunel University London, College of Engineering, Design and Physical Sciences

Symposium: “Quality in qualitative research and enduring problematics” Qualitative Methodology Forum – 27 January 2015 Faculty of Humanities and Social Sciences, University of Bath

Experiential triggers for this project (1) Editor “Third, I don’t have a good understanding of the representativeness of the sample. Only 30% of winning organizations were represented. Of them, the key informants are not at all well-described. How do we know that these 15 people best represent their organizations? Who are they? What are their characteristics. The sampling issue (at both of two levels, organization selection and individual selection within organizations) is a critical issue”

“First of all, most of the articles published in the journal are not qualitative in protocol. So we need to enter this manuscript gently as has been done with some others”

“I thought for a topic like this you may need to increase the number of participants to at least 25 (50%). This was not a random selection. 15 is respectable. However, can you prove they are representative of the 51 winners? Secondly what are these winners of? What are the broader implications of using such subjects? Who can they speak for?”

Experiential triggers for this project (2) Reviewers “There remains an issue of how useful this information would be. The poor response rate is certainly something that gives the reader pause, and the results of the work cannot be seen as generalizable.”

“While no decisions regarding clinical practice should ever be based on such a small sample size, in the end I believe it makes a contribution solely in the way we conceptualize evidence. “

“The generalizability of findings is what makes science different from faith-based study. If the purpose of qualitative research is not to produce findings that are capable of generalizing beyond the data gathered, then why should a reader care about the results? We only care when we think the results may generalize to our own studies, theories, situations, etc.”

“Your sample size of 15 out of 51 organizations is very small….However, the issue of sampling error is just as pertinent to qualitative researchers as it is to quantitative researchers. Three statements are just not enough for me to believe that the result will generalize to the larger population. In other words, how do I know this is a real signal rather than noise? The generalizability of the results, perhaps, might be more believable if a large proportion of the interviewees concurred with the quoted statements.”

How many qualitative interviews are enough? Purposeful sampling → Informationally representative samples (vs statistically representative) Tension between achieving informational redundancy and be able to conduct in-depth, ‘case-oriented’ analysis (Sandelowski, 1995) So how many interviews are enough? Experts in the field tend to concur with the answer that “It depends…” (Baker & Edwards, 2012)  Research objective(s)  Epistemological and theoretical underpinnings  Type of analysis  Epistemic community (i.e. rules, norms of the scientific community one belongs to)  Practical considerations (e.g. hard to access participants; resources; researcher’s career stage; institutional constraints) The criteria of data/empirical saturation or theoretical saturation (i.e. when no new insights about the phenomenon/theoretical category are found by adding new cases) remain useful in determining qualitative sample sizes.

Our Research Questions  To what extent are arguments to justify the sample size of qualitative interview-based research employed?  What are these arguments?  Do the presence and/or the nature of justifications differ across journals from different disciplines?  How might the justifications relate to other characteristics such as the type of analysis?  What justifications, other than sample size, are used to defend the adequacy of the sample?  Do the various justifications change over time?

How do we ‘do’ science: Scrutinising published research A few examples…  How is theory used in qualitative research? (Bradbury-Jones, C., Taylor, J., Oliver Herber, O. (2014). How theory is used and articulated in qualitative research: Development of a new typology. Social Science & Medicine,120, 135-141)

 How are research questions constructed in social scientific work? (Alvesson, M., & Sandberg, J. (2013). Constructing research questions: doing interesting research. London: Sage)

 Are participant recruitment and retention in RCTs adequately reported? (Toerien et al. (2009). A review of reporting of participant recruitment and retention in RCTs in six major journals. Trials, 10, 52.)

Our Methods • Systematic review of qualitative interview-based studies • Published between Jan 2003 and Dec 2013 in high quality healthcare-related journals representing different disciplines

Journals:     

British Medical Journal (BMJ) (Medical focus) British Journal of Health Psychology (Psychology) Sociology of Health & Illness (Sociology) Journal of Healthcare Management (Management Sciences) Social Science & Medicine (Interdisciplinary Social Sciences journal)

Inclusion/exclusion criteria  Cross-sectional study design (i.e. longitudinal studies were excluded)  Individual, qualitative interviews as method of data collection (i.e. group interviews and structured interviews were thus excluded)  Data analysed qualitatively (i.e. studies that quantified their qualitative data were excluded)  Mixed method studies were excluded (e.g. qualitative interviews and structured questionnaires)  Papers reporting more than one qualitative methods of data collection were excluded (e.g. individual interviews and focus groups)

Data Extraction Form

Data analysis We used both qualitative and quantitative analysis of data

Some preliminary results: British Medical Journal (BMJ)  Search keywords: interview* AND qualitative

Identification

132 results were obtained from the search

Screening

All 132 results were screened as to whether they met the criteria

Eligibility

Data were extracted from the 23 articles that met the eligibility criteria

Included

22 eligible articles were included in the review

One paper was excluded due to longitudinal design

BMJ: Some basic stats Table 2 N of interviews conducted and N of participants included N of Interviews

N of participants

Minimum

19

19

Maximum

128

128

31

30.5

44.14 (31)

44.73 (31.05)

Median Mean (SD) Table 3 Study populations Study Populations Table 1 Countries of data collection Countries of data collection UK Netherlands Canada Serbia Australia South Africa & Uganda Total

Frequency of papers 16 2 1 1 1 1

Percentage % 72 9 4 4 4 4

22

100.0

Patients Doctors Nurses Relatives & Significant others Senior Healthcare Managers Healthcare Administrative Staff Children (but not patients) Caregivers Other (e.g. sex workers, exoffenders etc.)

N of papers involved each study population

Percentage (%) of the total number (N =22) of studies*

13 5 3 3 1 1

59 22 14 14 4 4

1 1 4

4 4 18

* Some studies involved more than one study population (e.g. patients and doctors).

BMJ: Justification of sample size  The majority of papers (N = 12; 54.5%) did not justify their sample size in any way Frequency

Percentage % over the total number of justifications provided

Data saturation Theoretical Saturation

7 2

41 12

Previous literature Sample pre-defined requirements (e.g. maximum variation sampling) Pragmatic reasons Researchers’ experience

2 2

12 12

2

12

1

6

Nature of qualitative data

1

6

17

100.00

Types of Justification

Total

For the papers that DID justify their sample size (N = 10; 45.5%)

Number of different justifications provided by ‘justifying’ papers (N = 10)

BMJ: Qualitative analysis of sample adequacy argumentation (1) Diversity/Variation Aspects of the phenomenon of interest “An additional strength is its focus on reactions to intermediate results as well as positive and negative diagnostic results” (BMJ07)

Participant Demographic characteristics “One strength of our study was the diverse range of respondents in terms of age, socioeconomic group, location, and household smoking profile.(BMJ10)

Study Populations “We…included the views not only of patients and GPs but also of practice nurses and receptionists, who have generally been excluded from previous studies” (BMJ02)

“Participants were purposely sampled to represent a wide range in medical specialties, age, and sex to reflect the possible diversity of opinions.” (BMJ19)



The argument of diversity counterbalanced the lack of sample representativeness

“As with any qualitative study aiming for a maximum variation sample, the findings are not intended to be numerically representative— the sampling method is intended to show the diversity in responses, including those that are less usual. (BMJ17)

“These tables were, however, derived from a purposive sample and should not be taken to represent the population; rather, we aimed to capture the range and diversity of experience, beliefs, and opinions instead of providing a quantitative summary of findings.” (BMJ20)

BMJ: Qualitative analysis of sample adequacy argumentation (2) Sample particularity: Constructed negatively as it undermines the potential of representativeness of the sample and thus limits the generalizability of findings “The generalisability of our findings is limited because the sample was drawn from a prison in southwest England that predominantly holds white British offenders with sentences of less than one year; to what extent our findings might relate to long term offenders, those from black and ethnic minorities, or women is therefore unclear.” (BMJ09)

“Finally, within the sub-Saharan African population, the participants in this study are a relatively unusual group in that all were receiving palliative care. Most people dying with advanced illness in South Africa and Uganda, and indeed in the rest of subSaharan Africa, may have even less access to information and care than this sample.” (BMJ16)

“One limitation of our study is that we looked only at cases in which a request for euthanasia had not been granted or granted but not performed (about two thirds of all requests), and the perspectives of patients and physicians with regard to unbearable suffering might be different in cases where euthanasia was performed—for instance, showing more agreement between patients and physicians.” (BMJ15)

But because of the particular nature and characteristics pertinent to older people and patients with chronic pain the results presented may not be generalisable to other drugs or different age groups.” (BMJ11)

"Another limitation of our study is that we focused on “difficult” and “straightforward” cases rather than on the type of cancer; our study may therefore not be representative for the whole cancer population.” (BMJ19)

Some preliminary conclusions…  Just over half of the papers did not justify their sample size in any way  Claim to data saturation was the most common justification for the sufficiency of sample size  Stakeholder, demographic or phenomenal diversity/variation was the strongest line of defence of sample adequacy  Sample particularity was viewed to be problematic as it undermined the scope of sample and thus the generalizability of results  There was an absence of claims around ‘theoretical generalisations’

Quasi-quantitative referential system of evaluation of sample adequacy

Thank you for listening! Any questions?

References Alvesson, M., & Sandberg, J. (2013). Constructing research questions: doing interesting research. London: Sage Baker, S.E., & Edwards, R. (2012). How many qualitative interviews is enough? Expert voices and early career reflections on sampling and cases in qualitative research. National Centre for Research Methods Review Paper. Bradbury-Jones, C., Taylor, J., Oliver Herber, O. (2014). How theory is used and articulated in qualitative research: Development of a new typology. Social Science & Medicine,120, 135-141 Sandelowski, M. (1995). Sample size in qualitative research. Research in Nursing & Health, 18, 179183. Toerien et al. (2009). A review of reporting of participant recruitment and retention in RCTs in six major journals. Trials, 10, 52.