This essay is a final project task given in Research Method Science Education and Communication class. This essay is the following step of writing a critical methodological aspect of research that I have already publish on 13 of August 2012, namely “A critical review of the research: Teacher understanding of the nature of science and classroom practice: Factors that facilitate or impede the relationship”. I hope you enjoy reading it, but unfortunately due to the copyright issue, I cannot share the article with you.
This essay is a critical analysis of an article named, “When Are Graphs Worth Ten Thousand Words? An Expert-Expert Study”. In order to show my critical perspective on this article, to be specified; the methodological aspect of the research I will elaborate the analysis on study conducted by Roth and Bowen (2003) as follows.
Purpose and Research Questions
Since, the researchers did not explicitly state what the purposes and questions of this study, thus it is little bit difficult to find the exact purpose and research question. However we can find it in abstract and introduction part (see in page 429 and 430).
In the last paragraph of the introduction part (page 430) the researchers implicitly stated their purpose in doing this study, i.e. to better understand graphing expertise. The researchers also stated that in particularly, they want to understand the contribution of experience (content represented, laboratory, and understanding of conceptual framework) to the particular readings provided to scientist.
Set in the particular purpose, this study wants to answer an initial question “When are Graphs Worth Ten Thousand Words?”. As a matter of fact, in conducting this study, the researcher focused on answering and describing how the behaviour or action (verbally and pictorially) of scientist when they are dealing with the familiar and unfamiliar graphs. From the question, the answer expected is a clear description about the nature of expert on interpreting the graphs or a qualitative data.
Research Approach, Methods and Theoretical Framework
In this study, the researchers used ethnography as the research approach or research strategy. In my perspective, the researchers had a good argument and good point in choosing the strategy (see page 431), since the result expected is a description about how the question of the study and it is an accepted argument since one of the purpose of ethnography is to describe or interpret actions, experience, and knowledge (Spradley & McCurdy, 2011). In explaining the reason why the researchers choose this approach, they also relate this study with the previous studies framework purposed by Becker (1989) and Becker & Verelas (1993) (see page 431). This decision is also followed by a consistent action in conducting the interview, in which the respondents’ action was not interfered by the researchers while they were elaborating their thinking.
As mentioned before, the method used is interview. In my perspective, since the researcher decided to use ethnography, then interview is a relevant method to gain what is being asked in the research question. Not only interview, this study also employs observation as the method, as the researchers argue that the way scientist interpret the graphs is observable. Unfortunately, there is no certain description that they used observation as the method, and it seems that the one that being observed is only the line of process interpreting and pictorial activity. Furthermore, by face-to-face interview, the researcher can do a depth analysis on the respondents answer and behaviour or action (verbally and pictorially) in explaining the graphs. Thus, the strategy and method chosen is coherent with the research question and purpose. However, in my opinion, if the researcher only do interview, I do not think that it can really measure or get a fuller picture about what should be measured.
The theoretical background of this research is used as the framework in interpreting and choosing the material. The theory is also influence the researchers in deciding the research question and the aim of conducting this study (see page 431 paragraph 2). However, it seems that the theory presented has nothing to do with the result of this study, even though in some extent the researchers compare the result with the other researches’ findings (see page 443).
Population and Sample
The sample chosen must ensure efficient and effective saturation. It means the participant must be people who best represent and have knowledge about the study. Roth and Bowen give description about the samples, which let the reader verify the validity of the sample chosen and also as transparency. As mentioned clearly, all except 3 of the participants are ecologist. Since the graphs, that used as instrument, represent ecology thus in my opinion the choice of sample is appropriate. The number of sample used is representative since the method used is interview and the aim is to get a description how the action of scientist in interpreting familiar and unfamiliar graphs.
In term of experience, the participants are diversity in experience and background of education, however, the diversity did not contribute to the result described in the conclusion. The participants are adequate in categories, verification, completeness, and comprehension that are stated in the research purpose. Talking about the term of expert, it is a good way to limit the definition on this study only on the scientist, even though in my perspective the scientist is less adequate across the board of the definition of expert. Furthermore, the choosing of sample (ecologist) are not best representative for scientist in general, thus I think it will not produce a sufficient data to account for all aspect of problem to be obtained. It also will affect the reliability and the generalisation of the result. I think the researcher should also choose scientist from another field of study such as mathematics, biology, chemistry, etc. Therefore, the sample is in too small scope to formulate the answer for the question.
In this study the researchers used 3 unfamiliar graphs and the other one is the respondents own graphs, as the instrument. There is no statement about how the instrument is being validated or standardized. The researchers were too confident to say that the 3 graphs are unfamiliar to the respondent. Therefore the paper shows as clearly as possible the potential influences on the results, and enables the reader to determine the credibility and validity of these results. Some of these influences are:
- All of the control graphs are common in literature of ecology, since the almost 82% of the respondent are ecologist. If we compare the graphs used in the study conducted by Roth (2009), the graphs used are more variance in term of science.
- It is already published in ecology journal, as a matter of fact one of the respondents bring the journal that presents the graph, even though in general the result shows that the all 3 graphs seem unfamiliar to the respondents.
There are also some goods aspects such as:
- As a familiar graph, the respondents are allowed to use their own graphs. It is an excellent way to get a fuller picture about how the respondents reaction when they are dealing with familiar graphs.
- In some extent, the 3 graphs enable the researchers to make sure that the situation that is being observed is under their control.
The other instrument that also used is the two-stages model. This model is used in interpretation process. There is also no clear statement whether this instrument has been validated or not.
All interview are being videotaped, this increase the internal reliability of the data. During the interview, the researchers did a good effort in order to keep the naturalism of the respondents’ activities in interpreting the graphs, by do not interfere while the respondents interpreting the graphs. However, the researchers did not do member checking after interpretation process that affect the quality of the data. In the other hand, by doing so will help in avoiding a bias result, unfortunately it is not being discussed in this paper. Because interview is the only method used, there is no opportunity for data triangulation. The fact that there is also no discussion about how the videotaped-records from the interview being registered, enable the reader to determine the transparency.
Data Analysis and Interpretation
Roth and Bowen stated that the results got from interview are being transcribed and coded. Nevertheless, there is no discussion about who transcribe the interview, how the data being coded, what codes used, and what of the purpose in doing so. It enables readers to determine the transparency of the data analysis. Roth and Bowen did a good collaborative work and transparent, they describe how they process the data, i.e. the transcripts are independently read and so are the videotapes, before doing a collaborative interpretation. There are some unclear descriptions in data analysis process that also affect the transparency of this study such as:
- The authors said that there are three respondents’ interview results that experience extensive collaborative analysis, but there is no discussion about the purpose doing so, and which respondents experienced it.
- In some extent, Roth and Bowen state that the respondents’ descriptions in the first 3 graphs are not sufficient or satisfactory (see in page 454) or they only describe the graphs on a surface level (see in page 448), but in fact there is no discussion about how the model or instrument measure the validity, and what the respondents’ interpretation that is being expected.
They also have a good diversity in analyses the data, for which the mathematical aspects are being analysed by Roth and the ecological aspects are being analysed by Bowen. The diversity in analysing makes the result more focused on the pictorial and verbally, and avoids an overlapping interpretation. In addition, they also do collaborative interpretation after analysing the data individually. The researchers attached some quotation of the interview transcription; it shows that they want to make the readers see what they did (transparency). Roth and Bowen also attached the graph constructed by the respondents during the interview session (see figure 2 in page 449 and figure 3 in page 454). In page 460, Roth and Bowen said that the process of interpretation did not only just involve inferential processes but also the dialectic process. However, there is no further discussion about how the dialectic processes being conducted.
Roth and Bowen also did not involve other expert in analysing the data, to test the interpretations and conclusion (peer examination). They have not computed the inter-rater reliability.
As mentioned previously that the research question on this study is want to describe how the behaviour or action (verbally and pictorially) of respondents when they are dealing with the familiar and unfamiliar graphs. The conclusion gives the clear answer for this question. The conclusion also gives clear explanation in answering the initial question of this study. However, the result of this study cannot reach the purpose of this study. The result cannot describe how the experiences influence the way scientist interpret the graph or the respondents’ graph expertise that becomes the purpose of this study. The result or the conclusion is also lack of external reliability that means the result cannot be generalised, since the sample itself, ecologist, cannot be generalise as scientist or even for expert. The conclusion did not come back to the result of the previous study that relevant with the findings, as mentioned in Level of Performance paragraph (see in page 443).
Each element of this study is fit together and in some extent it shows a consistency, even though the result cannot reach the research purpose. Roth and Bowen have strong argument in choosing the strategy, the method used is also chosen well. It means that the method is relevant with the data that should be gained; qualitative data that let researchers have depth analysis on scientist graphs expertise. In other word, the method measure what is wanted to measure. In some extent, this study shows inconsistency as follows;
- In the conclusion the author did not come back to background theory.
- In the abstract the authors introduce the term complex iterative process, however there is no explanation about what the term means anywhere in the paper. The use of the term does not increase the credibility of this study.
- The term of scientist is generalised as expert, but in the result the researchers claim that the result cannot be generalised.
Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification Strategies for Establishing Reliability and Validity in Qualitative Research. International Journal of Qualitative Research, 13-22.
Morse, J. M., Swanson, J. M., & Kuzel, A. J. (2001). The Nature of Qualitative Evidence. California: Sage Publication Inc.
Roth, W.-M. (2009). Limits to General Expertise: A Study of In- and Out-of-Field Graph Interpretation. Cognitive Psychology Research Developments.
Roth, W.-M., & Bowen, G. M. (2003). When Are Graphs Worth Ten Thousand Words? An Expert-Expert Study. Cognition and Instruction, 429-473.
Spradley, J. P., & McCurdy, D. W. (2011). Conformity & Conflict: Readings In Cultural Anthropology (14 ed.). Prentice Hall.
Tedlock, B. (2000). Ethnography and EthnograpicRepresentation. In Handbook of Qualitative Reseach; Second Edition (pp. 455-486). California: Sage Publications Inc.
Whittemore, R., Chase, S. K., & Mandle, C. L. (2001). Pearls, Pith, and Provocation: Validity in Qualitative Research. Sage Publication, 522-537.