CHAPTER FOUR: DESIGN OF THE STUDY
4.1 Re-stating the Research Problem and Research Questions
This research is a qualitative study of EML. Qualitative research is preferable for the study of “an empirical, socially located phenomenon, defined by its own history” (Kirk and Miller, 1985:10) such as this substantive area. The problem in response to which the research emerged is the dialectic that missionary EFL teachers are seeking to help their students, but might be unknowingly causing harm by being engaged in EML.
With so little previous research on EML, the research questions are exploratory. This is especially true of the first research question: What does EML mean to teachers and learners in Bolivia? How do they experience EML? This question involved asking people what they do, why and how they experience it, and the aim is a rich, detailed description. In addition to describing EML practice, a second question addresses specific dialectics, those classroom tensions of values and ideologies. The study aimed to identify them and elicit information about how they are experienced and dealt with. Thirdly, and crucially, having amply described EML in this context, the study sought to identify critical concerns.
The research problem and questions are the focus of this investigation, and in order to attain high quality answers, a study was designed with them at the centre. They were borne in mind when deciding how and where to find data, collect and analyze it. Their origins, importance and context have been discussed in previous chapters and previous studies pertinent to them have been critically examined. In this chapter, the axis of the investigation, the design of this present study is explained. This includes how data were collected and analysed to answer the research questions, constructs of reliability and validity by which the success or otherwise of the study might be measured and other necessary considerations.
4.2 Data Collection
4.2.1 Interviewing as a Data Collection Method
While research at the macro-social level discussed in the previous chapter relies heavily on documentary analysis, this research is focused at a classroom level and requires use of methods and techniques more akin to those used by Canagarajah (1999). Furthermore, while mission organizations may have documents on ELT as mission, they may not have been written by those most closely involved with teaching EML. It was considered preferable to choose a method closer to those involved at a grassroots level, giving them a voice, which is consistent with critical concerns. After consideration of various options and the limits on time and resources, depth interview was selected.
As a means of data collection, interview means little unless qualified. Along Grebenik and Moser’s (1962) continuum of formality, the interviews performed in this study lean towards greater informality. They are depth interviews (Seidman, :9) in which respondents were invited to re-construct their experiences in a substantive area of inquiry. Pre-conceived categories were not used from the beginning since the interview approach used was both emergent (Rubin and Rubin, 2004:38) and iterative (ibid.:40), with each interview approaching the findings more closely than the last. Such an interview style is suitable for studies like this one, examining a complex, multivariate phenomenon. The aim throughout the interview period was to make meaning of informants’ experience and understanding (Seidman, :3; Gubrium and Holstein, (1995:4), finding connections and patterns among the complex elements mentioned.
Interviews are often used in critical studies, where the approach taken exerts a strong influence on the style and techniques, making such interviews entirely different in nature from the positivist variety which de Laine (2000:118) claims manipulate and dominate the interviewees. Critical studies, aiming to pierce through hegemony and problematize realities that are largely taken for granted, affords great importance and validity to the perceptions and concerns of interviewees (Rubin and Rubin, 2004:21). This cultural relativity required the interviewer to listen more and ask less (ibid.:38). Furthermore, a critical approach to interviewing empowers the respondent, allowing her or him to decide what is of importance and potentially giving her or him a voice to be heard by policy-making authorities (Gubrium and Holstein, 1995:16). In a feminist-critical approach this could taken further, with the researcher acting as collaborator with the respondent. Seidman (:92-93) suggests, however, that real equality between interviewer and interviewee is impossible, and Gubrium and Holstein (1995:16) maintains that the relationship between the two is always asymmetrical. This researcher preferred to err on the side of neutrality rather than assume the role of collaborator, due mainly to the danger of bias. Even so, many participants expressed delight at being asked about their experiences and concerns.
The main advantage of depth interviewing as a data collection method was the human texture of material gathered. Data is rich, deep, detailed and vivid (Bell, 2005:135; Rubin and Rubin, 2004:76). Survey data might have been more easily analysed and presented, but Lee (1993:101) states epistemological concerns about the use of such a researcher-controlled, closed method in areas of human experience. A survey allows very limited contact with respondents, and important issues which might surface, as well as significant contradictions and maverick responses, cannot be pursued and explained. Thus, the interviews were also more flexible and adaptable (Bell, 1999:135), allowing deep probing and investigation outside the potential of a survey. A third advantage was the suitability of interviews across ethnic lines. Interestingly, cross-ethnic interviews are generally more successful than average, with respondents tending to give very full explanations and not assume that details are too mundane for the researcher to be interested in (Rubin and Rubin, 2004:111).
The disadvantages of interviews demand close attention also. At the top of the list is their subjective nature and the potential for bias, which is a particular risk in research produced by a lone researcher. Realistic interviewers accept that they are humans and not machines (Selliz, 1962), and certainly this study was inspired by the researcher’s own biographical experiences and interests. However, the real danger lies not merely in interest, opinions and enthusiasm, but the projection of these onto informants (Rubin and Rubin, 2004:18-19). What de Laine calls the tension of involvement (2000:175) can bias the study in two ways. This researcher found initially that it took the form of the response effect (Rubin and Rubin, 2004:18-19), with respondents supplying the answer they felt the interviewer wanted to hear. Moreover, the interviewer might simply have heard what he wanted to and heeded only the data which supported pre-conceived findings. These two possible threats to objectivity were minimized, though not eliminated, by the preparation in advance of questions, piloting and conscious awareness during interviews.
A further problem was the difficulty in obtaining the rich, detailed and personal answers desired. Respondents often withheld information until the relationship with the interviewer became established. Textbook answers were supplied rather than real experience. The interviewer tried to be sensitive and patient, waiting for the inner voice (Seidman, :63) which is used to reveal real, meaningful data. This, of course, took time, which was a further and considerable disadvantage to the individual researcher. Time was required not only for a sufficient number of interviews, but also travel time amidst road blockades and to remote areas, and not least transcription (ibid.:16; Bell, 1999:135). Yet another challenge to the interviewer was encouraging encultured interviewees to overcome their reticence to explain what may seem to them to be banal. This excitement of the ordinary (Rubin and Rubin, 2004:169) had to be achieved in order to obtain full description and explanation of the reality faced by the respondent.
4.2.2 The Research Instrument
Depth interviews are vastly more complex than repeated question-answer turn couplets. Indeed, an array of instruments involving algorithmic sequences are available, but the scale of this present study did not merit anything so complicated. Nevertheless, a sturdy instrument, even if a lightly designed one, is needed if quality data are to be yielded. Bearing in mind the concerns above regarding interviewing, an instrument was designed, relying heavily on Wengraf’s (2001:111) Biographic-Narrative-Interpretive Method (BNIM). It can be seen in Appendix A. Of course, due to the iterative design of the instrument, the researcher added further questions in the probing and follow-up phases as themes reoccurred and categories emerged.
Using this instrument, the grand tour and probing phases occurred in one interview session lasting one hour fifteen minutes. The grand tour was initiated by a Single Question aimed at Introducing Narrative (SQUIN) in order to obtain a useful chunk of the respondent’s stream of consciousness, or spontaneous gestalt (ibid.:). Questions often occurred during this phase to the interviewer, but had to be held back. The researcher’s role here was to listen actively. The SQUIN chosen from a list of three was non-directive and avoided opinion at this early stage. Of course, many respondents felt uneasy at being asked such an open question and requested help in speaking at such length. To this ends, non-verbal prompts were employed and if necessary a few pre-formulated probe questions were asked to advance the interviewee turn with minimal interviewer input.
Following the grand tour, the researcher initiated the second phase of the interview, relying here on notes taken during the previous phase. Now was the time to focus on recurring themes and draw them out. The probe questions here were different in character to those used in phase one. They were designed to elicit from the respondent more detailed, deeper answers than previously rather than supply additional but perhaps unrelated data. This was the point where examples were collected and definitions were provisionally crafted.
The follow-up phase, lasting one hour, took place on a subsequent day in order for the interviewer to think about the data and identify opportunities to carefully and efficiently move towards fuller understanding. In particular, apparent contradictions and under-exemplified claims were more closely examined. At this point also relationships between themes and emerging categories were explored.
After each interview, field notes were written up, including summaries, verbatim quotes and memos. Only notes for immediate purposes were taken during interviews. At the end of the data collection period, there were over forty hours of recordings and several hundred pages of field notes. The next task was to analyze these data in a meaningful, presentable form.
4.3 Grounded Methods of Data Analysis
A strictly grounded approach could have been taken throughout the study instead of a critical approach. However, there is a strong precedent for using grounded methods of data analysis within a critical study. The critical approach already having been justified, here follows an overview of grounded data analysis, including its characteristics and suitability.
Emergence is the main characteristic of grounded methods of data analysis compared to most other methods. The problem, according to Glaser (1992; Glaser and Strauss, 1967), with analysing data using pre-conceived categories is that data are easily forced into categories which may not accurately reflect the substantive field of study. The grounded alternative allowed categories to emerge from the data themselves. Thus, role of the grounded analyst was to inductively shape a theory or model from the data rather than to select data according to how they fit existing categories.
Since pre-existing categories were not necessary for the grounded analysis, the method lent itself well the under-researched field of TEML. Rather than verify previous theories, this complex, multivariate area was researched with respectful, deep listening to real responses as valuable in themselves and not simply of use in as far as they fit analytical categories. There was perhaps one conflict between grounded and critical principles. Critical studies problematize and are often political in nature, whereas grounded studies should have no agenda. However, genuine critical studies are not normatively leftist as is often claimed (Pennycook, ***). The aim is not to impose left-wing political theory onto the area studied, but to problematize it. Similarly, in fact, grounded researchers look for the problem central to a socially constructed or experienced phenomenon.
The procedures of grounded data analysis began with the constant comparative technique (Glaser, 1994; 1998). Each utterance or event was contemplated and the analyst asked what was meant by it. As this process continued, some themes reoccurred and could be grouped together. Thus categories emerged from the data. Later, it arose that some categories were in fact core concepts on a higher conceptual level, and that other categories contained various properties on a lower conceptual level. In this way, categories arose both horizontally and hierarchically to build a picture of the substantive field.
A potential problem was that categories could snowball, with the picture of the area of study growing uncontrollably without any conceptual centre or unity. Alternatively, central concepts might have fractured. This danger of full conceptual description, for which Glaser (ibid.:1978) eventually criticizes Strauss’s interpretation of grounded theory, was avoided by rigorous coding. Data was used to generate categories which reoccurred and saturated, thereby justifying their inclusion in the presentation of findings. Otherwise, they were not included. Coding continued, matching data to categories, until each one saturated and further coding ceased to shed new light on the core category. As coding came to an end, there was a high level of abstraction and conceptualization. Thus the steps involved in grounded data analysis followed were: constant comparison, coding, saturation and conceptualization.
4.4 Validity and Reliability
Kirk and Miller (1985) argue for more attention to reliability and validity in qualitative studies. Concerning reliability, Wragg (1980) asks:
Would two interviewers using the same schedule or procedure get a similar
result? Would an interviewer obtain a similar picture using the procedures on
different occasions?
Kirk and Miller stress that this must not amount to a “vulgar positivism” (1985:20) seeking to imitate quantitative studies, but rather the investigation must be rigorously constructed in order to promote objectivity. How, then, can research based on data involving opinion and life experiences be deemed reliable?
The answer is not, in fact, to narrow the study so much as to make it useless but highly reliable as the term is often understood. In fact, if answers supplied by interviewees are almost identical, it may be a quixotic relationship where data are merely a rehearsed reciting of official policy, what might be called a public as opposed to a private voice (ibid.:41). Even the test-retest model used to prove a diachronic relationship, stable over a period of time, denies the historical context which is so important for a critical study with social, cultural and political threads.
The reliability required for this type of study was one of synchronic relationships where respondents within the same timeframe provided consistent, but not identical, data. It may well be that a replication of this study in ten years would reflect changes in historically bound variables. This kind of reliability was tested by the saturation with data of clear emergent categories. It was different in character from the snowball effect where unrelated data amass with no clear pattern emerging, a type of full description very different from Glaser’s version of grounded theory. If, as Bell (1999:103) suggests, one respondent had just seen a television programme which influenced her or his answer, that influence was unlikely to impact the findings as it would not saturate. Saturation, then, is the indicator of reliability in this study. This means that even with contradictory data, reliability is not necessarily wanting so long as the co-existing multiple categories all saturated during data analysis.
Another important criterion for evaluating this study is its validity, the extent to which it describes what it sets out to describe (ibid:104). Perfect validity is unattainable (Kirk and Miller, 1985:21), but nevertheless a study is hardly useful if questions asked meant different things to different respondents. The researcher checked what respondents meant or understood by terms used during interviews in order to avoid the incorporation of invalid inferences (ibid.:28). A useful technique was simply to ask interviewees what a word or concept meant to them. With experience, as more and more interviews are carried out, the interviewer learned to be alert to discrepancies between the meaning of researcher and participant (ibid.:30).
Without using observation or other data to triangulate interview data and check that interviewees have described real experience and not simply policy, reliability and validity are difficult constructs to prove in a qualitative study such as this. This difficulty may be a criticism of positivist researchers who prefer more closed tests or verification of scales. However, qualitative research occupies such a vital place in re-constructing human experience that such a risk had to be taken and minimized as far as possible, remembering all the while that any research approach has its own advantages and weaknesses.
4.5 Further Considerations
4.5.1 Pre-Tests
Subsequent to planning the data collection and analysis methods, three participants were interviewed prior to the fieldwork with the purpose of troubleshooting weaknesses in the design. These pre-test participants were selected because of their similarity to those main respondents on the field.
One result of piloting was that the time required for each interview was reduced slightly to avoid respondent fatigue. It became apparent that to foster a good relationship with participants, a wider stock of prompt and probe questions was needed, in order to avoid obvious and unnatural repetition.
Another important result of piloting was that the researcher attuned his ear to people’s voices, something which took no little adjustment after lengthy periods absorbed in reading previous studies and theories. After the third set of interviews, the interviewer was no longer trying to relate every response to existing theory but to listen to the response for what it was and generate grounded theory.
4.5.2 Access and Sampling
Access was negotiated in advance of a two-month field trip to Bolivia. Initial contact was made in writing following an Internet search of mission agencies in the country. On the field, however, access was more difficult than anticipated and it was exceedingly difficult to interview all respondents twice in such a short timeframe.
Random sampling was infeasible (Seidman, :44-47) since there is no known list of missionary teachers and EML is generally taught informally. However, purposeful sampling was carried out in order for the study to be as representative as possible on a small scale. Participants were drawn from TEML activities associated with six similar medium to large evangelical mission agencies active in Bolivia across six cities. A total of thirty respondents were interviewed, of whom fourteen were learners and sixteen were teachers. Furthermore, in the early planning stages of the project, a list of possible participants was drawn up in conjunction with interested senior missionaries to include typical cases as well as a few extreme, critical and sensitive cases using Patton’s (1989:100) typology. It emerged that nearly all teachers included were native English speakers and that all participants suggested were urban residents. An attempt was made to include others in the sample, but no non-native English speaking teachers or rural-resident learners were found.
Provenance of Respondents Number
Australia 2
Bolivia 14
Canada 4
Great Britain 4
Ireland 2
United States of America 4
______________________________
TOTAL 30
______________________________
Table 4.1 Provenance of Respondents
Location of TEML Activity Number
Cochabamba 6
La Paz – El Alto 11
Oruro 2
Santa Cruz de la Sierra 6
Sucre 3
Trinidad 4
______________________________
TOTAL 30
______________________________
Table 4.2 Location of TEML Activity
4.5.3 Some Practical Considerations
Interviews must be professional and planned. A number of practical considerations were necessary before the realization of the interviews, namely language, recording, transcription and timing. Language was an especially important issue to be considered. Some learner respondents saw their participation as an opportunity to practise English, while others used Spanish, the usual language in most cases of their personal thoughts and socio-affective responses. All of the teachers interviewed were native speakers of English, not by design of the study but the reality on the field. As each interview started, Bolivians were asked if they would prefer to use Spanish or English, and they were encouraged to code-switch if they felt it appropriate.
All interviews were recorded using a small, unobtrusive device. Although this involved more organization and might have inhibited the openness of some respondents, the act of recording showed interviewees that their contributions were treated respectfully and reported accurately. This also enabled the researcher to return to interviews during the analysis of data and writing of this dissertation and check the exact wording used. This was vital as almost half the interviews were conducted in Spanish, which is not the researcher’s first language. Only quoted and paraphrased material was fully transcribed in order to make the time requirements manageable. Transcribed text was left as it was recorded, rather than tidied up. Despite the risk of presenting respondents as verbally awkward, once again this researcher preferred to close yet another door to bias.
One final practicality which caused serious problems was timing. Although an interview schedule was devised, frequent last-minute changes, cancellations and road blocks caused havoc with the schedule. To interview thirty respondents twice required a longer time than anticipated. In light of cultural and practical factors like punctuality, road and climatic conditions and political demonstrations, any future interview schedule should be carefully piloted and extra time allowed in case of disruption.
4.5.3 Ethical Considerations
Many people were involved in this study to whom the researcher has a great deal of responsibility. All research involves some ethical concerns, such as the cardinal principle that:
[t]he field should not be left more difficult for subsequent investigators to
explore by disenchanting respondents with the whole notion of research
participation.
(Johnson, 1984:14-15)
To this end, it was paramount that the researcher did not presume the cooperation of missionary teachers and others, their time or their candour. Permission was therefore sought in writing where feasible and with as much courtesy as possible. It is hardly a lofty academic idea, but nevertheless an important one that respondents were thanked for their contributions (Bell, 1999:145).
This study touched on sensitive issues of power and motives, and therefore more rigorous attention had to be paid to ethical considerations than leaving the way clear for future research in the same field. Especially in interviews, the researcher had a power over the respondents: his interpretations and choice of citations would ultimately be selected for the written report (Lee, 107). Since this research deals with a dialectic concerning EML, the researcher is a tertius gaudens who in a way benefits from intra-personal if not also inter-personal conflict. Respondents must not be exploited (Seidman, :7). In particular, Kvale (1996) reminds interviewers that despite some similarities in technique and environment, they are not therapists.
In order to minimize the exploitation of respondents and those observed, a statement of informed consent was shown in English and Spanish to interviewees and teachers whose lessons were observed (see Appendix A). This document informed respondents of their rights and expectations as well as about the aims of the research and the identity of the researcher. Crucially, the statement made clear that participation was voluntary and that those willing to be interviewed or observed could withdraw or withhold information (Seidman, :7) at any time until a given date at least one month after data collection ceased.
The researcher undertook also to conceal the identities of those who supplied data in order to minimize risk of harm, sanctions or stigma (de Laine, 2000:214; Lee, 1993:102). Confidentiality was ensured as far as possible (ibid:171), with measures taken including the use of pseudonyms and alteration of some biographical details, thus preventing even deductive disclosure (ibid:175). Data including cassette and digital recordings and field notes were stored anonymously.
Ownership of the data is another important ethical concern (Gubrium and Holstein, 1995:20; Seidman, :53). Because of the degree of confidentiality and practical difficulties in sharing ownership, the informed consent statement also made clear that, unless withdrawn by a date one month after the end of the data collection schedule, the researcher would wholly own the data. With the number of confidential respondents involved, it would have been unwise to make promises about contacting each one about the selection and reporting of data. Therefore, each participant was also informed clearly of the dissemination of the data collected and presented in the dissertation.
4.6 Summary
This chapter was earlier referred to as the hub of the research project. The research questions and the limitations of time and resources were considered before choosing a depth interview study. A great many decisions were taken along the way to report data as accurately as possible, but even so there was always a high risk of bias and several other problems were confronted. Bias was also a concern related to the grounded methods adopted for analysis. This chapter also describes realistic criteria for reliability and validity in judging the success or otherwise of the study, as well as other considerations useful to future researchers who wish to replicate the study.
Just as previous chapters fed the context and previous underpinning knowledge into this design chapter, subsequent chapters flow out of it. Chapter Five now displays the findings of the study as it was carried out from February to April 2006.
No comments:
Post a Comment