CRITICAL ASSESSMENT OF AN EMPIRICAL RESEARCH STUDY (DUE TUESDAY, JUNE 12, 2007; 20%)
One of the goals of this course is to enable students to evaluate the results of empirical research of interest to our discipline. This assignment is designed to allow students to identify an appropriate empirical study of interest to them in the open literature of information studies and other disciplines, e.g., psychology, history, fine arts, computer science, sociology, and philosophy; to implement the evaluative skills developed in class and in course readings in the assessment of this study; and to develop a concise, informed written assessment of the study. This assignment is intended to help students import the skills developed in this class to their professional lives and to help prepare them for the formal research proposal and empirical data collection instrument which are the capstone of the class.
As Olson (1996, p. 136) says, good researchers can distinguish “what the author was attempting to get some reader to believe from what they themselves . . . [are] . . . willing to believe.” He further notes that “Critical reading is the recognition that a text could be taken in more than one way and then deriving the implications suitable to each of those ways of taking and testing those implications against available evidence” (p. 281). We must be that informed, critical, evaluative reader, understanding the roles that various kinds of evidence and our criteria for evaluating evidence play in the assignment of illocutionary force to truth claims (p. 280).
It is wise to start this assignment immediately. In order to complete this assignment successfully, the student should:
• Identify appropriate research journals and/or monographs in the subject area(s) of interest; Hernon (1991b), Stenstrom (1994), Creswell (2003, pp. 27-48), and Busha & Harter (Chapter 15) provide some guidance on this score. You may also want to browse in the current serials on the 2nd floor of PCL, in the LIS and other bound serials on the 6th floor of PCL (especially in the T's and Z's), and in other collections in the UT General Libraries. Also browse in the General Libraries OPAC for journal subscriptions; see, e.g., Research by Subject (http://www.lib.utexas.edu/subject/) and Find a Journal (http://www.lib.utexas.edu:9003/sfx_local/a-z/default).
Especially take advantage of the remarkable collection of full-text and other indexing databases available to UT users; see, e.g., http://www.lib.utexas.edu/indexes/. You might find Library Literature & Information Science Full Text especially valuable.
• Scan through a number of empirical research papers in these sources.
• Choose an empirical study of particular interest that addresses the use, nature,
dissemination, or management of information as an object of study. The collection and
analysis of empirical data must be included in the study. The data, however, need not be
quantitative nor be quantitatively analyzed. Please consult the instructor if there is any
doubt of an article's suitability for this assignment.
• After several close and critical readings of the paper, use criteria discussed in class and in
the readings (including, e.g., Katzer et al., Chapters 16-19; Robbins, 1992, especially pp. 85-86; and Busha & Harter, pp. 27-29 and Chapter 15) to evaluate the research report. Also see Babbie on “Reading Social Research” (2004, pp. 473-478), but be wary of his use of terms such as “objectivity.”
The product of this evaluation will be a formal academic paper of no less than five nor more than seven (≥5, ≤7) double-spaced pages in length. Please refer to appropriate style manuals and to the syllabus section on Standards for Written Work while writing.
Your assessment should have the following components:
• An Introduction of 1-2 pages identifying the importance of the phenomenon to the field, stating your overall thesis with regard to the paper (i.e., is the paper good or not?), presenting a brief summary of the paper, and explicitly identifying the major criteria used to assess the paper. Be sure that these are evaluative criteria, not simply a list of topics or sections of the paper.
• An Analysis of 3-4 pages comparing the paper to the evaluation criteria identified in your
Introduction and referring to specific elements in the paper to support your assertions. It
may be helpful to think of organizing the analysis around the Conceptualization,
Operationalization and Methods of Data Collection and Data Analysis, Results,
Conclusions, and Supporting Material, e.g., figures, graphs, charts, notes, tables, and
appendices. This particular format is only suggested, not required.
• A Conclusion of 1-2 pages giving your overall assessment of the research paper and your specific recommendations to improve the study and/or the paper
• An Appendix containing the complete text of the research paper, including appendices and other supporting material. Please submit all material in 8 1/2" x 11" format.
You may find it helpful to review the six model student papers from previous semesters on Reserve at PCL -- the papers are listed alphabetically by title in UTNetCAT: "Analysis of Content Analysis of Research Articles in Library and Information Science," "Analysis of Study of Community Censorship Pressure on Canadian Public Libraries," "Assessment of 'Preservation Analysis and the Brittle Book Problem in Libraries: The Identification of Research-Level Collections,’" "The Eye of the Beholder: Analysis of a Study of the Effect of Subject Matter and Degree of Realism on the Aesthetic Preferences for Paintings," "Library Jargon," and "Public Archives of Canada Collections Survey." Each of the papers is different from the others, but they are all excellent. Do not copy the model papers' approaches; instead, use them to help you understand what I regard as good work and a successful analysis.
If the paper you choose to evaluate uses statistical or other analytic methods with which you are not familiar, do your best to examine their use as carefully as possible given your current state of knowledge. Add a sentence or two to your evaluation that says, in effect, that the author uses some analytic techniques which you are presently unable to evaluate fully, but, e.g., the numbers add up, their use is not clear, their use is clearly explained with a full rationale for use given, the author fails to explain his/her purposes in doing the analysis, and so on. Please be formal in your description of such methods, and remember the strategies for being a skeptical, critical reader of statistics as discussed in Best (2001a) inter alia.
Please hand in two copies of your full paper. I will grade and return one, and I will keep the other for my files. This assignment is worth 20% of your semester grade.
Late assignments will not be accepted.