Sampling Strategy And Codebook For Content Analysis
Directions
ASSIGNMENT
This assignment MUST be typed.
BACKGROUND. Imagine that you work for the United Nations, who issued (through their Intergovernmental Panel on Climate Change) a scientific consensus statement indicating that humans almost certainly cause global warming. At the same time, you are concerned that newspapers who report on global warming (also called climate change) may be presenting the issue with a “false balance” – presenting arguments for and against humans causing climate change, when there is a scientific consensus on one side of the issue. A previous study found evidence for this false balance (Boykoff & Boykoff, 2004); your job is to assess whether patterns of coverage in the last 6 months reflect a similar tendency to report with a false balance.
YOUR TASKS. Your job is to develop a sampling strategy and codebook for a content analysis of New York Times, Washington Post, Los Angeles Times, and Wall Street Journal news stories related to global warming and/or climate change published in the last 6 months. In order to do so, you will need to complete the 6 steps involved in doing a content analysis.
The population of interest for this assignment includes all news stories related to global warming and/or climate change published in the last 6 months. To obtain copies of the text of these news stories, you should use Nexis Uni and enter the following search term for stories that have appeared in the New York Times, Washington Post, Los Angeles Times, & Wall Street Journal abstracts (you may choose whether to focus on print stories only, blogs, or both – see Nexis’ list of sources for these papers):
“global warming” OR “climate change”
AND DATE is within the last 6 months
You may not be able to analyze all of these news stories (e.g., conduct a census), depending on how many newspaper articles are retrieved with this search term, so you may need to figure out a sampling strategy that will allow you to test whether or not these newspapers cover global warming and/or climate change in a falsely balanced way using a sample of news stories.
Task 1. In approximately two single-spaced, typed pages (12-pt font), answer each of the following questions: (4.5 points in total)
1) DEVELOP YOUR MEASURES (step 2 from lecture)
a. Provide a one-sentence conceptual definition of each variable involved in the research question. You’ll need to provide a conceptual definition for (a) news stories related to global warming and/or climate change, and (b) false balance. You will find the following articles helpful to look at as a starting point. You may start from the authors’ conceptualizations of related terms like “information bias,” “balanced reporting,” and “false balance”, but you should come up with your own definitions of what would constitute a “false balance.” (1 point)
Boykoff, M. T., & Boykoff, J. M. (2004). Balance as bias: Global warming and the US prestige press. Global Environmental Change, 14, 125-136. doi: 10.1016/j.gloenvcha.2003.10.001. Available online at:
http://www.eci.ox.ac.uk/publications/downloads/boykoff04-gec.pdf
Dixon, G., & Clarke, C. (2013). The effect of falsely balanced reporting of the autism-vaccine controversy on vaccine safety perceptions and behavioral intentions. Health Education Research, 28, 352-359. doi:10.1177/1075547012458290. Available online at: http://her.oxfordjournals.org/content/28/2/352.long
Grading Criteria:
o Provide a conceptual definition for news stories related to global warming and/or climate change, being sure to (1) address what you mean by news stories and (2) what counts as being related to these topics (.5 pts)
o Provide a conceptual definition for “false balance” including key dimensions or elements. The definition should explain clearly (1) what do you mean by “balance” and (2) why such balance is considered biased or false (.5 pts)
o Note: These definitions should (1) be written in your own words (not copying the authors’ terms but citing the authors if they influence your definition), (2) represent a conceptual, not operational definition (-.25 for each error).
2) DEVELOP A SAMPLING STRATEGY (step 3 from lecture)
Specify how many news stories you’ll need to analyze to test your research question. (1 point)
In other words, calculate your sample size(s). Describe why you chose this particular sample size. To make these decisions, you’ll want to specify the level of confidence and error you are willing to tolerate for this study. You will want to make use of a sample size calculator in making your decisions.
Grading Criteria: [Fulfill all three for full credit; deduct .5 for each error up to 1]
o Specify the number of news stories in your population
o Specify level of confidence
o Specify error willing to tolerate
o Include sample size (Note: Given your level of confidence and standard error, the number of your sample size should look reasonable to us)
Describe your sampling strategy using terms and concepts we’ve learned about in class and/or in the textbook. (1.5 points)
In this description, be sure that you answer each of the following questions: (i) Among all types of news stories (e.g., news stories, editorials, blog posts, opinion pieces, etc.), which ones will you choose as news stories related to global warming and/or climate change? (HINT: this decision should match your conceptual definition of what constitutes a news story related to the topic).
You must start by using the Nexis Uni search terms listed above to search for relevant stories to include in your sample. The set of articles that are returned from your search term will serve as the population for this assignment (HINT: depending on what issues of climate change you are focusing on, your population may be smaller or larger).
You will then need to describe the process of selecting a sample from this population by answering the following questions: (ii) How will you select specific news stories, from the overall population of news stories, to include in your sample? Will this process involve non- random sampling (convenience or snowball) or random sampling (simple or multi-stage cluster)? You can propose any strategy that you consider appropriate, but be sure to justify your decision and identify tradeoffs associated with that decision.
Grading Criteria:
o Describe how you will decide on news stories among all types (with some selection categories reflecting your definition of “news stories”) (.5 pts)
o State which types of news stories are to be included in your population and sample (.25 pts)
o Describe how you will select specific news stories to analyze (e.g., random or non-random sampling) (.25 pts)
o Provide a clear and thoughtful justification of your decisions (e.g., describe trade-offs) (.5 pts)
3) DESCRIBE HOW YOU WILL ASSESS CODER RELIABILITY (step 4 from lecture)
a. Describe the process that you will undertake to assess coder reliability. How will you assess whether coders are reliable in applying the codebook? Will you have multiple people double-code some of the same articles? How many will you double-code? Describe the process that your coders will follow. (1 point)
Grading Criteria:
o Name some procedures that you learned from the lecture about inter-coder reliability (.5 pts)
o Provide a clear and thoughtful description of the process and what it entails (.5 pts)
Task 2. On a separate page (or pages), provide a draft of the codebook you will use to content analyze the news stories (think of these as your operational definitions). The codebook itself should be separate from the 2-page document (4.5 points total)
Your codebook should include no more than 5 coding decisions for your coders, per news story. In other words, you have to decide on just 5 things to assess in the news story you decide to include in your study. The codebook should include (1) the names of (no more than 5) indicators of “false balance”; and (2) specific instructions about how to measure each indicator. You will get to provide your rationale for why you decided to choose these indicators in your 2-page summary document. It is often useful, in a codebook, to illustrate your directions for measuring each item with an example. As part of the grading for this assignment, we are going to take your codebook and see if we could use it to accurately code news stories on the topic.
In addition to providing the codebook itself, provide a brief rationale for each measure (as part of the 2-page summary; the codebook should be separate from this). Why did you choose to measure this as an indicator of false balance? How do these indicators reflect your conceptual definition? It is always a good idea to look at how other people have gone about coding false bias in news stories (and how they sampled newspaper articles to find them!). There is a sizeable literature on this topic.
It is always a good idea to start with questions that other scholars have used before. If you think previous measures are good questions and capture some aspect of the construct perfectly, you can use these measures, but be clear that you are quoting them directly. If you think previous measures are bad questions and you can improve on them, do so. You should begin by looking at Boykoff & Boykoff (2004) (http://www.eci.ox.ac.uk/publications/downloads/boykoff04-gec.pdf) and Dixon and Clarke (2013) (http://her.oxfordjournals.org/content/28/2/352.long), as well as
at least one other reference (each should be cited in your paper in APA format):
If you decide to use other people’s measures, or develop your own based directly on what others have done before, be sure to cite these articles in your assignment using APA format (please include both in-text citations and a reference list at the end of your assignment).
Grading Criteria: [Fulfill all seven for full credit; deduct -1 for each error (except APA style, worth only .5 points – see below), but no more than 4.5 pts]
o Reference at least one outside article in developing the codebook (NOT just the one we have provided a link to)
o Provide (no more than) 5 coding decisions (not more; if a student uses less, then award full credit only if it appears that they have reasonably covered the concept)
o Coding decisions should be directly related to the variables/concepts involved
o Operational definition (coding decisions) should reflect your conceptual definition
o Coding decisions should be specific enough that anyone could reasonably apply them without help (or asking questions)
o Coding decisions should be measureable (and allow us to test the proposed hypothesis)
o In addition, you must provide a correct APA citation; if at least one mistake is noted, deduct .25 pts. Deduct full .5 points if 3 or more mistakes are made
Task 3. Select four news stories from your sample. Code them using your codebook. Enter your codes in the spreadsheet. (step 5 from lecture) (3 points)
These four do not have to be randomly selected or correspond to your proposed sampling strategy – just pick any four to try it out yourself. All of your codes should be quantitative in nature – a number should correspond to each variable in each story. For example, if you were coding for the presence or absence of emotional display, you could assign “1” when it is shown or “0” when it is not. Attach full-text of each news story that you coded to the end of your assignment.
Grading Criteria: [Fulfill all four for full credit]
- Select and list 4 news stories (should be labeled in some way – a brief title is fine) (.5 pts; no points awarded if not 4)
- All codes should be quantitative in nature (.5 pt)
- The codes assigned to each article should appear to the grader to be reasonable applications of the operational definitions described in your codebook (1.5 pts total)
- Should use the attached spreadsheet (.5 pts)
TURN IN THE FOLLOWING. (1) The two-page description which includes the rationale for each measure in your codebook, (2) the codebook itself, (3) your codes written in the spreadsheet below, and (4) a print copy (or a link) of each news story you coded
Coding Spreadsheet:
Write each variable name in the column below
(use an 8 character-or-less abbreviation of this variable name)