Monday, March 16, 2009

Summary of Quantitative Critique

Link to the article:


Effectiveness of Interactive Multimedia Environment on Language Acquisition Skills of 6th Grade Students in the United Arab Emirates, written by Almekhlafi, was published in International Journal of Instructional Media. Nowadays, supported by Paivio’s Dual Coding Theory (DCT), the inclusion of Interactive Multimedia (IMM) in education has raised many researchers’ concern, however, from the literature review, it could be concluded that IMM may not always be effective as research results have been inconsistent. Therefore, there is a new tendency to study the effect of IMM in relation to cognitive learning styles. But in the context of UAE, research on the effect of IMM on learning has not gained enough attention yet, particularly when investigating multimedia in relation to cognitive styles. So with the purpose of investigating the effectiveness of IMM on learning ESL language skills and its interaction with cognitive learning styles: field-dependent (FD) and field-independent (FI), one main question “To what extent does IMM environment affect sixth grade students’ acquisition of ESL skills?” needed to be addressed.[pg.430] Results showed that there was no significant difference between IMM users and non-users in the overall ESL skills. However, when the participants were investigated in terms of cognitive learning styles, results showed a significant difference between FD and FI learners in the experimental group in favor of FI learners.

On the whole, this is a well organized article with each part clearly presented. There are some highlights in this study, as follows:

(1) Proper design:

This study investigated two independent variables (teaching method and cognitive learning style) working on the dependent variable (achievement in ESL skills). Because two variations (IMM versus traditional teaching) of teaching method and two variations (field-dependent versus field-independent) of cognitive learning style were manipulated at the same time, a 2×2 factorial design (ANCOVA) fitted the research. Furthermore, the cognitive learning styles (FD and FI) were conducted under each group (experimental and control group). Compared with 4 groups with four combinations, two-group experiment to some extent helped the researcher avoid other administrative factors which may affect the reliability of research results.

(2) Data analysis:

Using SPSS 10.0, two way analysis of covariance (ANCOVA) was used to test the null hypotheses. Pretest was entered as a covariate to control for variability in initial ESL skills level while the posttest was entered as the dependent variable.[pg.434] Besides, the in-depth data analysis was discussed with the hypotheses one by one, so it made the data and explanation more understandable. Furthermore, the statistics of the overall, sub-sections and interaction effects as a whole was analyzed in the factorial experiment, which could increase the understanding of the phenomena being investigated; otherwise, the conclusion may oversimplify the actual situation.

(3) Reliable measurement:

Firstly, GEFT was tested and validated to be applicable to 6th grade students so the classification of cognitive learning styles is reliable; secondly, pretest was developed by course instructor and validated by a jury of university professors, supervisors of English and experienced teachers and the reliability was established by carrying out a pilot study. Meanwhile, posttest was identical with pretest in the form.[pg.432-433] The testing results were therefore correspondingly reliable. Besides, the same instructor with both groups could mirror the same procedure, which made the research results more comparable.

Although the above highlights to some extent ensure the quality of this study, caution still need to be exercised when conducting this study. Apart from some potential measure validity problems listed at the end of the sheet, there are two main shortcomings as follows:

(1) Unrepresentative sample:

Ninety 6th grade students were selected from a private school and all participants were local males. Also, they were non-randomly assigned to experimental (n=46) and control (n=44) groups. So the factors of gender (male), the context of school (private school) and the different number of students (46 versus 44) in two groups may affect the validity and further generalization of the study outcomes.

(2) Vague description of IMM CD-ROM and research procedure:

There was no detailed introduction to IMM CD-ROM such as its interface, how to use it and other characteristics. Besides, the procedure was not clearly described. For example, it was pointed out that participants in the experimental group were asked to enhance their learning by doing some selected homework using CD-ROM at home; however, it did not mention the homework for control group. Moreover, no pretest and posttest question sheets were attached for better understanding. Therefore, the description of materials, measurement and procedure was not sufficient enough for future replication.

(3) Weak conclusion:

After data analysis and discussion, the researcher only presented the study results in the conclusion part without in-depth analysis of the findings and suggestions for practical use. Although the recommendations provided by the researcher may benefit future research, the weak conclusion could not highlight the purpose and significance of the study.

(Photo from Carolyn ^-^)


  1. "Although the recommendations provided by the researcher may benefit future research, the weak conclusion could not highlight the purpose and significance of the study."

    This is a great point that you have highlighted. After having an opportunity to review both Quantative and Qualatative articles for this course, I am truly amazed by the fact that there are so many research papers out their who have this flaw. As a student, who is potentially going to complete my own study, I worry about avoiding these flaws in my own papers.

  2. I agree that one of the major flaws of this study is the lack of description of the interactive media. It is interesting that the goal of the study was to investigate the efficacy of IMM, but really, we have no idea what the components of IMM instruction include. I'm happy that scrapeahappy is thinking about avoiding such flaws in future research, because a cautionary approach is welcomed!!!!