Tuesday, March 17, 2009

Qualitative Critique

Article link: http://llt.msu.edu/vol10num3/shin/default.html

Nowadays, computer-mediated communication (CMC) use has prevalently stepped onto the stage of language education and the related researches have addressed overall aspects of its context in language learning and teaching, however, previous studies of CMC context do not comprehensively focus on the issue that one needs to see the configured context co-constructed by language learners to fully capture the complexity of CMC practices, since the context for any learning activity is a complex interconnected relationship among contextual elements of the learning environment that learners configure for learning tasks. Therefore, with the purpose of examining “how a group of ESL students co-constructed online interactions of synchronous CMC practices within the dynamics of their group, while engaging with contextual elements of their CMC activities”, the researcher explores the following questions: 1) What kinds of interactional patterns are a group of ESL students jointly constructing? 2) What kinds of interactional norms are the ESL students establishing within computer-mediated social interaction? 3) How do the ESL students utilize CMC activities for their linguistic, social and academic goals? [pg.66] The three questions are designed according to ecological perspectives of language learning, the theoretical frameworks of this research, to broadly capture the study aims. Consequently, the study helps readers bring about an understanding of the complexity of CMC context in language learning and teaching, and awaken to enlightenment that “language learning is not only an issue of acquiring linguistic forms and functions, but also of developing a new self”. [Pg.66] Besides, the findings encourage language teachers to effectively draw on CMC in their teaching practices by raising their awareness of taking such elements into consideration as learners’ proficiency in using CMC tools, class size and teacher’s role in conducting CMC activities.

Critique of the Study

The study was conducted in an intermediate adult ESL class with 16 students, including international graduate students, visiting scholars and their spouses, at a university in the northeastern United States. In order to provide a detailed day-to-day picture of the culture-sharing group developing their shared patterns of behavior, norms and utilization of the CMC activities, the researcher designed the ethnographic case study to explore an in-depth understanding of context configuration in CMC environment by ESL learners; and writing in first-person point of view helps readers feel getting close to and think in the authentic research environment. On the whole, the study is well organized with arguments logically and persuasively presented, which makes it understandable and convincing. Apart from the writing style, its validity and the report style are also examined, as follows:

(1) Validity: The researcher “was informed of the CMC activity of this ESL class after the class designed their CMC activity”, which indicated that she did not intervene in conducting such CMC activities from the beginning of the research. Then, “she was introduced to the class and got permission for her study from the students to observe all of their CMC meetings and all FtF class meetings.” (See NOTE 5, pg.80) During the process of data collection, the researcher, serving as an observer, attended both CMC and FtF class meetings without being involved in any of their activities. For example, in CMC meetings, the researcher observed the teacher managing chat sessions without logging onto the chat program herself; and in classroom meetings, she sat outside the circle observing and taking notes of their FtF class activities. [pg.69] It implied that the researcher tried to be “invisible” in order not to impose any sense of uncomfortability or any psychological burden on participants to affect their natural interaction and performance. Although the researcher does not explicitly state her assumption, beliefs or biases, the ecological perspectives of language learning, as a core construct of this study, unconsciously influenced her data collection. For instance, even though the study only focused on the participants’ online CMC activities, the researcher also collected data from their FtF meetings because “ecological perspectives are not only concerned with participants’ online lives, but also their offline lives”. [pg.70, pg.78] Therefore, the researcher’s research design and analysis was implicitly affected by the theoretical framework in this study---ecological perspectives of second language learning.
Besides, to help readers better understand the phenomenon under the study, the researcher gave a relatively comprehensive description of the research context, such as the introduction to the university program, the necessity for CMC meetings, participants’ profiles, CMC tools and CMC activities environment, classroom setting as well as online & offline meeting schedules. Meanwhile, from the data analysis, it could be seen that the researcher demonstrated openness to the possibility of multivocality by presenting diverse points of view and interests drawn from participants of different professional roles and purposes such as the teacher, graduate students, visiting scholars and their spouses. Moreover, the researcher showed sensitivity to implicit meanings (tacit knowledge) gaining from observations, interviews or surveys, such as the learners’ age and silence in communications, which have much to do with finding analysis; for instance, participants’ age characteristics (as adults) could explain the configuration of face-work norms and their silence in the online meetings could highlight the importance of the teacher’s role in CMC context.
In addition, the researcher cultivated multiple sources of data through a semester’s long-term observation from both FtF meetings and Web-based chat meetings, including electronically saved chat meeting transcripts, field notes, recorded class interactions from the FtF class meetings, formal and informal interviews with participants, surveys, e-mail exchanges between the teacher and the ESL participants. And in order to increase the credibility and validity of the research analysis, such data as field notes, transcript of recorded FtF meetings, interview data, and electronically saved chat data were triangulated to cross examine and explore the patterns, norms as well as utilization of CMC activities.
Furthermore, the researcher analyzed the data iteratively to seek inner-connections between elements to strengthen the understanding and validity of the results. For instance, after noticing some participant’s (such as spouse learners) certain behavior, the researcher came back to check their profiles such as typing skills or professional roles, and then held interviews with them for logical analysis. Besides, the researcher represented the emic perspectives by demonstrating participants’ online dialogues or quotations to increase the reliability of data interpretation; and the study was then reviewed by anonymous reviewers to ensure the credible results. (See Acknowledgements, pg.80)
However, there is no indication for outlier analysis, representativeness check and coding check, which should be conducted to make the study more valid and universally recognized.

(2) The Research Report: Under the main theme of context configuration, the analytical themes the researcher used included constructed interactional patterns and norms, configured affordances regarding the CMC environment, and utilizations of CMC activities for linguistic, academic and social goals. [pg. 70] And the data were analyzed correspondingly based on these categories with classified data supporting related themes mainly in the form of direct quotations to authentically represent the participants’ voice, so there was no obvious redundancy or overlap in the coding. Meanwhile, though there was no visual representation to demonstrate the findings, it was reasonable in the study in that the description and explanation in the form of three-step finding-evidence-analysis was already easy for readers to understand.
In addition, some findings in the study were supported by the other research studies, which increased its reliability. For instance, at the end of the study, the researcher highlighted “class size” as a critical factor in CMC discussions and the finding reflected on the conclusion of other studies, written by Kitade and Kotter, of synchronous CMC suggesting that no more than five should be in any single synchronous virtual meeting at one time. [pg.78]
Generally speaking, this study is of quality, however, some points still should be reconsidered to make the study more convincing. For example, directly jumping from the introduction to the context and the researcher’s role to the finding analysis, there was no detailed description of how the CMC activities were carried out to create an authentic environment and a holistic picture for readers to better understand the whole situation in the research. Also in the findings section, the researcher uniformly presented the findings firstly and then demonstrated evidence from the collected data to support the findings. While this form of data analysis made the researcher ignore to acknowledge some unexpected or discrepant findings irrelevant to her assertions, but just chose the evidence that could speak for them. Besides, the researcher did not identify her assumption or bias in this study; neither did she explain how her own perspectives might influence interpretation.
Therefore, the comprehensive validity checking and persuasive report styles still deserve the researcher’s attention and endeavor while re-designing this study.

1 comment:

  1. Thank you Carolyn, for providing a link to the study! I need to learn how to do this, because it really helps in this kind of blog. Your review is thorough...I like the way you suggest that the researcher may want to re-design the study and try again! If only that could happen more often!