Home › 2012 › February

Month: February 2012

By Posted on Posted in Miscellany No Comments

Documents were recently obtained under false pretenses from the Heartland Institute and posted on the DeSmogBlog site. Megan McArdle suspects that the Confidential Memo: 2012 Heartland Climate Strategy file that is purportedly part of the document cache is a fake.

Let's see if the memo provides enough detail to permit identification of an author or at least the drafting of an author profile.

---

Ms. McArdle provides a nice start, observing that the memo author might use high-profile often and write in a run-on style. But let's examine the memo a bit closer:

1. Perhaps the biggest clue is that the memo author did not realize that the possessive form of United Nations is not United Nation's:

The suspect pool is now restricted to persons with a misunderstanding of possessives.

2. The memo author consistently used a comma to set off the final item in a list of more than two items, such as in this sentence:

Another $88,000 is earmarked this year for Heartland staff, incremental expenses, and overhead for editing, expense reimbursement for the authors, and marketing.

The suspect pool is now limited to persons with a misunderstanding of possessives and a preference for the Oxford comma. Let's continue...

---

3. The memo author wrote 20 as a number but two as a word.

4. The memo author did not indent paragraphs.

5. The memo author used ragged-right justification with no hyphenation.

6. The memo author used a dash in K-12.

7. The memo author used periods in most section headings, an unusual choice that might be a modified APA style:

8. The memo author did not mind an orphaned word that appeared at the top of a page:

9. The memo author used periods for U.S. in adjective form.

10. The memo author inconsistently hyphenated the adjective high-profile / high profile.

11. The memo author did not offset such as with a comma.

12. The memo author used focus in where focus on might be more common:

In 2012 our efforts will focus in the following areas...

13. The memo author used parenthetical remarks, especially in the final section that Ms. McArdle suspects is closest to the author's style.

14. The memo author introduced the acronyms IPCC, NIPCC, AGW, and WUWT without explanation, presuming reader familiarity with these acronyms.

15. The memo author indicated a project with quotation marks ("Global Warming Curriculum for K-12 Classrooms") but indicated a written document with italic font (Climate Change Reconsidered). The New York Times was written as NYTimes, without italic font or quotation marks.

16. The memo author used a percent sign (%) instead of writing the word percent.

---

Once this profile was completed, I suspected that the best place to look for documents matching the profile was the Heartland Institute, if the memo was authentic, or the DeSmogBlog site, if the memo was fake; the memo might have been forged by a third party, but I decided to start with the simple scenarios.

Google searches for the exact phrase united nation's coupled with site:heartland.org and with site:desmogblog.com respectively returned 97 results and 617 results. Adding the distinctive focus in phrase decreased the results to the memo itself. I removed the focus in phrase and added the less distinctive K-12 phrase, which returned from DeSmogBlog the memo, this file, and a Heartland budget file, but returned from the Heartland site this transcript of a 2007 speech by Heartland Institute President and CEO Joseph Bast.

The similarities between the memo and the speech transcript were not trivial.

1. The speech transcript misplaced the apostrophe in the possessive form of United Nations:

2. The speech transcript consistently used a comma to set off the final item in a list of more than two items. [see the red boxes below]

3. The speech transcript wrote out three and six as a word, consistent with the use of two in the memo. [see the purple boxes below]

[The green box indicates an unexpected switch from the first person singular to the first person plural.]

4. The speech transcript did not indent paragraphs. [see above]

5. The speech transcript used ragged-right justification with no hyphenation. [see above]

6. The speech transcript used a dash in K-12. [see the purple box below]

7. The speech transcript used periods in some section headings. [see the blue box below]

8. The speech transcript had an orphaned word at the top of a page:

9. The speech transcript used periods for U.S. in adjective form:

10. The speech transcript inconsistently hyphenated man made as a predicate nominative: "Claims of a consensus that global warming is man made" on p. 2, but "claims that global warming is man-made and a crisis" on p. 3.

---

The remaining items provided no evidence of consistency with the memo or were inconsistent with the memo.

11. The speech transcript sometimes used a comma to offset such as, and other times did not.

12. The speech transcript did not contain the word focus.

13. The speech transcript did not appear to overuse parenthetical remarks.

14. The speech transcript defined an unfamiliar acronym before using the acronym. The speech transcript did not contain the acronym AGW, and global warming was modified with man-made and not anthropogenic.

15. The speech transcript used italic font for periodicals and quotation marks for books, reports, and articles. The New York Times was written as the New York Times.

16. The speech transcript did not use the percent sign (%) and instead used the word percent.

---

The memo and the speech transcript appear to be formatted similarly, with similar margins and font, and the reading statistics are similar for the memo and the speech transcript, respectively: 21.2 words per sentence, compared to 25.0 words per sentence; 5.2 characters per word, compared to 5.1 characters per word; 18 percent passive sentences, compared to 16 percent passive sentences; reading ease of 29.8, compared to a reading ease of 26.4; and a Flesch-Kincaid Grade Level of 14.3, compared to a Flesch-Kincaid Grade Level of 14.3.

The memo and the speech transcript are somewhat consistent with each other in multiple elements indicated above and in the sense that both were written by a person who appears to have a strong but imperfect command of the English language: one of the few imperfections in the speech transcript was a passage from page 10 that incorrectly used it to represent groups and that also incorrectly used it's and not its as a possessive:

Environmental advocacy groups raised $6.6 billion in 2006 and it’s take is growing fast...

Based on imperfections that riddle DeSmogBlog, many DeSmog bloggers might not have been able to sustain the level of grammar in the memo throughout the entire memo, and many DeSmog bloggers appear to avoid the Oxford comma. For example, a 17 Feb 2012 post by Richard Littlemore contains a possessive error and overuse of parentheses, but lacks the Oxford comma:

---

The memo also lacks the signature of a Joseph Bast memorandum, the use of large-square bullet lists:

Perhaps a two-page memo was not long enough to warrant a large-square bullet list, or perhaps the lack of a large-square bullet list is evidence of a forgery.

But the memo does look and sound like something that the President and CEO of the Heartland Institute would write and has written, though some passages are inconsistent with that idea, such as: "...two key points that are effective at dissuading teachers from teaching science" and "[t]his influential audience has usually been reliably anti-climate and it is important to keep opposing voices out."

The two most bogus paragraphs appear to be the "Expanded climate communications" paragraph and the paragraph with apparently erroneous information about the Koch Foundation donation (see the update here): these paragraphs just happen to be in the only sections with titles that lack a period.

---

Perhaps the memo is like the Testimonium Flavianum, a core authentic document with inauthentic interpolations inserted by a true believer on the opposite side of a battle.

Perhaps the document cache obtained by DeSmogBlog contained an authentic Heartland memo that served as the basis for the formatting and core text of an interpolated memo; this would explain both the similarities and the differences with the Heartland speech transcript.

The interpolation theory lowers the bar from the highly original idea of generating a bogus confidential memo from scratch to the less original idea of spicing up an existing text.

---

For example, note the parallelism apparently intended between Taylor and Gleick in these two sentences:

The parallelism is broken with variation from a parenthetical e.g. to a parenthetical such as, lack of consistent hyphenation in high profile, and a change in focus from high profile outlets to high profile scientists...and this broken parallelism might signal the presence of two authors.

Or perhaps the entire "expanded climate communications" section is forged, given that the phrases climate communications and climate communication never appear on the Heartland website: 62,600 hits for climate, 17,300 hits for communications, and 0 hits for climate communications:

Oddly enough, though, climate communications is a tag on the DeSmogBlog site:

The evidence is clear: Heartland reserves the phrase climate communications for confidential memos. There might be another explanation for the fact that a phrase absent from the Heartland site but appearing on the DeSmogBlog site also appears in a document hosted by DeSmogBlog that Heartland alleges is forged, but as Tink Thompson reminded us:

If you have any fact which you think is really sinister...is really obviously a fact which can only point to some sinister underpinning...forget it, man, because you can never on your own think up all the non-sinister perfectly valid explanations for that fact.

---

Further notes:

  1. Both the memo and the speech transcript use single spaces between sentences.
  2. Not counting brief indications of payments such as ($11,600 per month), each extended parenthetical remark in the memo appears in one of the two sections without a period in the section title.
  3. The word key appears twice in the memo: once in a section without a period in its title, and another time in the suspect phrase two key points that are effective at dissuading teachers from teaching science.
  4. The phrase such as appears five times in the memo, each time in the "climate communications" paragraph.
  5. The "climate communications" paragraph has multiple errors and odd phrasings, such as especially through our in-house experts (e.g., Taylor) through his Forbes blog and related high profile outlets. Note that his refers to experts.
  6. [Update 19 Feb 2012 at 3:45pm] It is possible and perhaps likely that the Heartland memo was interpolated by someone unaffiliated with DeSmogBlog. Presumably, the person who obtained the document cache under false pretenses and sent the cache to DeSmogBlog is an opponent of climate skeptics, and opponents of climate skeptics appear to use the phrase climate communications more often than climate skeptics themselves; for example, the search site:grist.org "climate communications" -role -gavin returns 3,660 hits from Google. (The -role phrase is to remove hits about the memo itself, which contains the phrase an important role in climate communications; the -gavin phrase is to remove hits regarding the Climate Communications Prize that the American Geophysical Union awarded to Gavin Schmidt in 2011; Grist was chosen merely as an example of a group unaligned with climate skeptics.)
  7. [Update: 19 Feb 2012 at 5:11pm] Jim Lakely of the Heartland Institute explains the release of the document cache: "The stolen documents were obtained by an unknown person who fraudulently assumed the identity of a Heartland board member and persuaded a staff member here to 're-send' board materials to a new email address." The Heartland staffer who emailed the documents presumably has the email address that the documents were sent to, but Heartland does not appear to have released that email address. I presume that Heartland could demonstrate to a neutral third party that the pdf of the confidential memo was not sent from the email address that the Heartland staffer used to send the other documents. I also presume that the person who received the document cache could demonstrate that the Heartland staffer emailed the confidential memo, but I also presume that that person would rather remain anonymous.
Tagged with:

By Posted on Posted in Science knowledge No Comments

The National Science Board might revise two science knowledge questions that are currently phrased as follows:

1.Human beings, as we know them today, developed from earlier species of animals. Is that true or false?
2. The universe began with a huge explosion. Is that true or false?

The Board is considering respectively prefacing the questions with "according to evolutionary theory" and "according to astronomers." This proposal has generated controversy, as Yudhijit Bhattacharjee reported:

The change infuriates Jon Miller, a science literacy expert at the University of Michigan, Ann Arbor, and architect of the original questionnaire, which is now used by several countries. “If you are altering the questions in that way, you are doing it for religious reasons,” he says. “We don’t make statements like, ‘According to some economists, we had a recession’ or ‘According to the weatherman, we had a tsunami” (p. 394).

Miller's examples are not well-chosen analogies to questions about evolution and the big bang: questions about a recession or a tsunami are typically not prefaced with "according to" because these are definitional questions for which there are commonly-accepted definitions; for example, a recession is defined as two consecutive quarters with a decline in gross domestic product.

But questions about evolution and the big bang are inferential questions that involve interpreting a large number of observations to make an incredibly broad summary judgment. Thus, a more suitable analogy to economics might be a question about whether deficit spending necessarily improves an economy in recession. Lack of an "according to John Maynard Keynes" preface to this sort of question would make some Nobel economists look financially illiterate.

This is not to suggest that the uncertainty inherent in economics is commensurate with the uncertainty inherent in science, but it is to suggest that the type of question suitable for measuring knowledge of a definition might not be suitable for measuring knowledge of a disputed inference.

But the dispute over adding an "according to" caveat is misguided because the National Science Board should direct its efforts to developing better measures than the current questions that require mere familiarity with or acceptance of a summary inference.

Perhaps respondents should be asked to cite evidence for the big bang, which would provide a deeper measure of knowledge and would avoid the should-we-measure-belief-or-knowledge issue that the National Science Board is stuck on. Researchers could be much more confident in a person's level of science knowledge if the person mentioned the red shift or cosmic background microwave radiation or the expanding universe, than if all the person did was agree with the true-false statement that the universe began in a big explosion.

See here and here for other thoughts on measuring science knowledge.

Tagged with:

By Posted on Posted in Science knowledge No Comments

Science knowledge has often been measured with a set of questions that include these items about evolution and the big bang.

  1. Human beings, as we know them today, developed from earlier species of animals. Is that true or false?
  2. The universe began with a huge explosion. Is that true or false?

The National Science Board removed from the National Science Foundation's Science and Engineering Indicators 2010 a discussion of public responses to these items about evolution and the big bang, which drew strong criticism from some scientists (see here and here). But National Science Board member Louis Lanzerotti claimed that the evolution and big bang items were "flawed indicators of scientific knowledge because the responses conflated knowledge and beliefs" (Bhattacharjee 2010).

Discussion of responses to the evolution and big bang items were reinstated in the 2012 edition of the Indicators, but disagreement remains about whether questions about evolution and the big bang should be placed in a science knowledge exam alongside questions about non-disputed facts such as whether electrons are smaller than atoms.

To provide more data on this issue, I conducted a factor analysis of sixteen science exam questions that were asked on National Science Foundation surveys from 1988 to 2001 for which full data were available.

Factor analysis

... takes thousands and potentially millions of measurements and qualitative observations and resolves them into distinct patterns of occurrence (Rummel 1967, p.445).

The sixteen questions used in the science exam are listed in the table below.

Blue cells indicate questions measuring an understanding of probability, red cells indicate religiously-neutral factual questions, and white cells indicate factual questions that are, have been, or might be perceived to be in conflict with religious teaching.

The table below presents results of a principal components factor analysis that retained four factors:

Factor analysis reports but does not name or otherwise identify the patterns, so the patterns must be interpreted. The four probability questions unsurprisingly loaded on the same factor, so it appears that an understanding of probability is a separate dimension of science knowledge. Factual recall questions were spread among three factors, but factor 3 contained the questions about evolution, the big bang, and continental drift -- each of which is perceived by some to be in conflict with biblical creation accounts.

This third factor therefore might tap a religious dimension of science knowledge, so I repeated the analysis after adding responses from a question that measured religion-science conflict more directly:

We depend too much on science and not enough on faith. Do you strongly agree, agree, disagree, or strongly disagree?

Disagree responses were collapsed and agree responses were collapsed to form a binary variable.

Results shown below indicate that the religion-science conflict question loaded onto factor 3. Perhaps this third factor is not tapping into science knowledge as much as it is tapping into trust in science or scientists; after all, the facts underlying the factual questions in this factor -- about evolution, the big bang, and continental drift -- are disputed by scientists and religious authorities. So, for instance, the big bang item might be measuring whether a person trusts science or religion to answer questions about the origin of the universe.

This analysis does not resolve the definitional question of whether science literacy requires acceptance of a scientific consensus, but it does suggest science literacy is not a unidimensional concept, and it does suggest that there might be value in separating the dimensions of science knowledge, similar to the way that the SAT separates verbal and math scores.

---

Notes

  1. Non-responses and don't know responses for the science knowledge questions were coded as incorrect responses.
  2. There were 13,503 observations across the survey years of 1988, 1990, 1992, 1995, 1997, 1999, and 2001, and all of these observations were used for the first factor analysis. There were only 12,796 responses to the question about faith and science, so the second factor analysis had this lower number of observations.
  3. Three questions on genes and experiments were not included in the analysis because the questions were not asked in 1988 and 1990. Questions on radiation and the center of the Earth being hot were not included in the analysis because their number of observations was less than the number of observations for other questions in 1990. Inclusion of these five questions does not change the inference that disputed items load onto a separate factor or that the science-religion controversy item loads on the disputed questions factor, but inclusion of these questions does add a fifth and sixth factor in some cases.
  4. The Stata do file for the analysis is located here, and the dataset can be downloaded here. The citation for the data is: Miller, Jon D., Linda Kimmel, ORC Macro and NORC. National Science Foundation Surveys of Public Attitudes Toward And Understanding of Science And Technology, 1979-2006 [Computer file]. 3rd Roper Center version. Tallahassee, FL: Susan Carol Losh, Florida State University, Department of Educational Psychology & Learning Systems/Arlington, VA: National Science Foundation, Division of Science Resources Statistics/Arlington, VA: American Statistical Association [producers], 2009. Storrs, CT: Roper Center for Public Opinion Research [distributor], 2009.
  5. The variables in this analysis are binary (correct or incorrect, faith or science), so a factor analysis with tetrachoric correlations is preferable to the reported regular analysis. M-Plus results that account for the binary nature of the variables is presented below, providing confirmation of the main finding of a separate factor for the religious-tinged items.

---

UPDATE (Mar 11, 2015)

Emil O. W. Kirkegaard has re-analyzed the data and has made the important note that" Just because scree tests etc. tell you to extract >1 factor does not mean that there is no general factor."

Tagged with: