The previous thread
Recently, Mike Bolan’s article, entitled The Truth About Plantations , appeared prominently on this site and attracted 94 comments to date. My particular interest in this article started with Bolan’s claim in the opening paragraph that:
“In this article I use data from a comprehensive pulp mill impacts study (2) to remind us of the impacts of plantations on rural Tasmanians.” Expecting to find a reference to a published report at the end of the article, I was surprised to find no more than “(2) Study took over a year to complete and was carried out to inform the community during the RPDC process (copy available thru TAP).” I was also surprised that the article itself did not refer specifically to the study’s methods, and presented very little sourced data, and seemed really not much different to many overviews of the issue written by single individuals without reference to any impacts study. Hence the comment by reader Dave “The truth about plantations? This is one man’s opinion on plantations”. Whether that was all it was or not, that was certainly what it looked like.
My curiosity (or, if you prefer a more cynical view of my efforts, rampant online contrarian bloodlust) triggered, I enquired (#47):
“Firstly, which of the remarkably few data claimed in this article (most of which concern economics rather than impact) were derived from the “comprehensive pulp mill impacts study”?
Secondly, is this study in question available for scrutiny anywhere other than TAP? One would think that if it was comprehensive enough to be useful and had taken so much work it might be available elsewhere.”
There followed a discussion with Mike Bolan in which it was claimed by him (#50) that:
• The study’s methods “involved visits, interviewing [respondents] and asking for evidence for their comments.”
• Interviewees were selected both at random (many selected this way weren’t available) or random stops during trips, by published interest (e.g. vegetable growers) and by referral.
• The study aimed “to identify patterns of impact (repeated impacts rather than one-offs specific to a particular situation) of any forestry action that was required by the pulp mill proposal, and then to present those impacts graphically on a single sheet of paper.”
• The work was “reviewed” by “Dr [Warwick] Raverty and others who observed that it was probably the only way for most people to understand the total impacts of the mill and all of its various processes on Northern Tasmania.”
I expressed the concern (#53) that “What this sounds like is not a ”comprehensive pulp mill impacts study ” but a database of perceived pulp mill impacts derived from a rather haphazard interviewee selection process.” Mr Bolan then asserted (#55) that he regarded “the community’s perceptions of impacts, given evidence,” as “more relevant than anyone else’s opinion of how those impacts affect communities and individuals.” He also claimed “Determining impacts is not particularly onerous, nor do we need statistical or other methods to establish a comprehensive list.”
I replied (#66) that “Determining impacts of these sorts of things is a big deal whatever you might pretend otherwise, because in many cases a claim of an impact will be made, but it will require expert study in relevant fields and detailed understanding of the mill proposal to determine whether that impact would actually occur. As such it does sound like your study assesses claimed impacts and not necessarily real ones, and is not a substitute for any assessment of real impacts that might or might not be adequate.” I pointed out that “respondents may be too biased to assess “evidence” fairly (for instance they may dismiss it based on its source rather than its evidential basis), or they may lack the expertise to understand the claimed “evidence”. They may also only be looking at a part of the “evidence” picture and not all of it.”
Bolan (#70) pointed out that some impacts could not be verified by others, such as pain or emotional suffering, to which I responded that “then that’s all the more reason to treat them as claimed impacts rather than demonstrated impacts.” He made reference to the study employing “multiple perspectives (see Ian Mitroff’s work for this)” and in #70 engaged in some generalised pseudo-psychoanalytical expert-bashing that seemed to object to an expert-based assessment process on the grounds that it gave experts power and potential ego gratification irrespective of the outcomes. He also justified the work by reference to what he saw as the overly narrow focus of the previous impact assessments:
“The purpose of the diagram was to allow a person to take in the full range of impacts created by the processes involved at one look, and thereby reach conclusions about (for example) how many of those impacts had been considered by various groups (it was about 15%).”
As well as disposing of his comments about experts (see #82) I suggested again that the real issue here was not the potential value of the work, but the trumping up of it as a study of real impacts when it was actually a study of claimed impacts. However, by this stage Bolan had indicated (#79) that he would no longer be responding on the thread.
There is much more to the thread than the above; the purpose of the above is simply to recap my discussion with Mike Bolan concerning the status and validity of his study, which he says he conducted as an independent consultant working for no pay.
The diagram
A copy of a version of the diagram alluded to has come into my hands thanks to the assistance of a helpful Tasmanian Times reader. In an attempt to protect the confidentiality of my source I will not provide the exact version number, but it is version Y.XX where Y is between 5 and 9 and is XX is between 50 and 90, and carries a date about three years ago. It is credited to Paul T. Wilson and Mike Bolan. Comments by Bolan on the previous thread give the impression that the work is still continually updated: “Because the study is presented diagrammatically, it is only available via TAP or myself because we have to prepare each diagram for printing with the latest data, and each is custom printed.” But at the same time he is clear that the work was essentially done “3 years ago”, and that that was the time for input on method design, and the version number suggests it was indeed the product of multiple major redesigns and minor revisions at that stage. Therefore I will assume, until demonstrated otherwise, that it broadly represents subsequent versions.
The diagram is about 2.4 metres wide by nearly one metre high. It is headed “Pulp Mill Proposal – Community’s Own Integrated Assessment of Impacts”. Along the bottom is a pictorial diagram of forestry, mill and claimed plantation-related economic processes, and this feeds up to a 14×16 table (I see no need to call it a “matrix” as I can’t be bothered making any bad movie jokes right now) in which various fields of impact (eg “Plantation Impacts – Fresh Water” or “Conversion of Farms to Plantations”) are assessed against various “types of impact” (eg “Plants, Animals & Ecosystems”, “Tasmania’s Brand Value”, “Local Government”). Impacts are declared to be positive (green), negative (red) or either neutral or unclear (there is actually no key – white). Various side-boxes summarise contentions, go into specialised issues like Atrazine, Managed Investment Schemes, and perceived drift in propaganda claims during the process.
The diagram indicates which impacts are considered to be covered in the RPDC process and its replacements (the former covers five of the 16 fields of impact). It also gives grand tallies of reds and greens for impacts supposedly considered/ignored in both the RPDC process and its “fast track” replacements, and for the overall number of impacts. There is barely a green to be seen. The study effectively claims that it has considered around three times more impacts than the RPDC process (not 6-7 times more as Bolan’s comments quoted above might suggest), which in turn is said to have considered around three times more than its replacements. It also gives a weighting of negative over positive impacts of close to 10-1, compared to about 7-1 for the RPDC process, and around 3-1 for its replacements. So the claim being implicitly made here by Bolan’s presentation is that the more you look at this proposal, the more you find and the worse it gets.
I am not aiming here to do a complete hatchet-job on this much-vaunted diagram, but measured against the ambitious claims made on its behalf by Bolan there is not much positive to be said. Nonetheless in fairness (or the cynics may call it faint praise!) I should say that the diagram is in my view professionally and attractively presented, its structure is logical and it is the sort of thing I would probably look at if it was placed at the right level in a conference posters hall and I had a lot of time on my hands. As a database of claimed impacts – a sort of user’s guide to points on which to agitate about an issue in the media, I imagine that some campaigners against the mill, especially those who were not very well informed across the range of issues, would find such a diagram useful. That said, those who relied on it without investigating the issues for themselves would wind up with egg on their faces, because not all the claims involved stack up. I also think the diagram does make the point that a mill proposal cannot be exhaustively assessed just by considering the more direct impacts of the mill itself, though that point can be just as easily be made in a few punchy paragraphs. Whether the mill proposal can or should be exhaustively assessed by any other process are two other matters.
That is the end of the good news. What follows is my assessment of the most apparent of the document’s shortcomings:
1. Inadequate attribution and biased source selection
Very little on the entire poster is specifically attributed in any way. Some submissions to the RPDC process are noted by number, and a disclaimer attributes information to “ordinary Tasmanians engaged in farming, tourism and other similar activities, or who had special knowledge (eg chemical engineering) and who were highly concerned about the proposal and the impact on their lives and the lives of their loved ones.” There is a call for further information to improve the diagram in future, but the disclaimer implies that aside from the summaries of the RPDC process, expert information has only been included where that expert was opposed to the mill and most likely already active against it. For the vast majority of claimed impacts, the diagram provides no evidence of veracity. They are just ambit claims based on the comments of supposedly “ordinary” Tasmanians. Where expert information is referred to at all (eg nameless “scientific research on Atrazine”) it tends not to reflect a full view of debate on that topic. Also, with the diagram clearly based on a mix of “community” and sympathiser views, it is not clear which claims come from which kinds of sources.
2. No attempt to weight good and bad impacts
The diagram as a whole suggests that bad impacts outnumber good ones by around 10 to 1, and Bolan in comments to the threads states “Positive impacts were massively outweighed (quantitatively) by the negatives, which probably explains why a majority in the area oppose the mill.” But no indication of the magnitude of impacts in either direction is given, so, for instance, all the jobs provided by the forest industries are counted as one of just two claimed benefits of harvesting native forests, but one of the counterbalancing red dots under that item is just “concerns over existing regulation”. The two are hardly comparable in magnitude.
3. Frequent duplication of negative impacts
In far too many cases, what are supposed to be many different negative impacts are the same impact added over and over again. For instance, the lack of tourist appeal of plantations is listed as a negative under “Conversion of Farms to Plantation”, “Plantation Growth” and bewilderingly “Plantation Impacts – Fresh Water”. “1080 killing native wildlife” appears twice in the same row, supposed plantation monocultures as a supposed biodiversity negative three times (more on that below). Indeed the “Decreased Agricultural Production” column duplicates seven claimed impacts in the “Plantation Establishment” column, of which only one (“Threat to natural image”) even might be argued to follow from reduced farmland area per se, and only on the assumption that there are actually people out there who are thick enough to consider intensive farming “natural”.
There is the odd case where a positive impact is duplicated too, but this is so rare that it has probably occurred accidentally. A double standard is frequently applied in that indirect knock-on effects are included as negatives frequently, but if the same standard was applied, positive flowon effects of each job creating process would be included under all of “Individuals and Families”, “Community, Social and Cultural”, “Economy”, “Tourism”, “Agriculture”, “Other Industries” and each level of government – and included for each column in which job creation is a positive – since all of these areas benefit from an increase in jobs in other sectors of the economy. It is completely obvious to this observer that the reason this does not occur is that the compiler is not at all neutral (being already opposed to the mill) and thus positives were giving token representation then downplayed through differential treatment. This need not have been intentional.
The more one looks the more duplication of negatives one finds, sometimes occurring in a less obvious form. For instance the idea of plantations hogging water crops up under one heading as a cause of increased water uptake and in another as a cause of decreased water flow. In the context given, the first is only a problem because it causes the second, and the two are therefore really one impact.
If the standards applies to negative and positive impacts were the same, I estimate that the c.10:1 ratio of negative to positive impacts would be reduced to below 2:1. It is also possible that with input from economists more familiar with positive flow-on effects, the difference would be eliminated entirely.
4. False and dubious claims
Given the lack of non-sympathetic expert impact into the diagram it is bound to contain some scientific howlers. “Monoculture plantation” is listed three times as a “Plants, Animals & Ecosystems” negative, all in contexts linked either explicitly or through the diagram as arising from the conversion of farmland to plantation. But, as I have pointed out many times on these pages, there is no scientific basis for the view that converting a farmland area to a plantation is a biodiversity negative, given that such an area is coming off a low base because farmland is already ecologically trashed. Australian studies have found that eucalypt plantations outperform farmland on biodiversity indicators – the Lyon et al (2007) bird study (Biological Conservation Vol 137 Issue 4 pp. 533-548) being a particularly striking example, but there are others. In one of my own research interest areas (plantation invertebrates) I can say that the conversion of farmland adjacent to bush areas (especially bush remnants) is a biodiversity plus as native species incapable of living in cleared farmland expand into the plantation. For plantations that are entirely surrounded by farms there will probably not be such benefits (depending upon the intensity of past farming), but even then there is no basis for assigning a negative rating.
Other claims are difficult to fathom on surface value. In the “Impacts on Launceston section”, mill jobs and related support jobs are indicated as a negative impact. This makes no sense; if people from Launceston are employed in the mill and related jobs it should be shown as a positive. Another curious item under Launceston refers to “Lost investment in woodheaters and clean air”. Presumably this is meant to refer to lost investment in woodheater reduction programs, but precisely why the mill would have any effect on such programs is completely unclear to me; in any case, if it did lead to reduced investment in such programs by the State then that (however undesirable) is still a saving for the State that should be included as a positive.
Ambit claims without obvious basis in fact (or with apparent basis in severe exaggeration and/or what Geoffrey Hills once called the Doctrine of Tasmanian Exceptionalism) also occur. For instance a reduction in rural communities is supposed to reduce what people can “uniquely see and do in Tasmania”. But rural communities and tourism based around them are hardly unique to this state, nor even a particularly prominent part of its tourist appeal.
5. Inconsistent standards
As well as the issue of differential approaches to duplicating positive and negative impacts, inconsistencies in method abound across the diagram. Impacts that are denoted as questionable are sometimes shown as neutrals and in some cases as negatives, never as positives. Impacts apparently either positive or negative are sometimes shown as neutrals for no readily apparent reason.
Tax incomes are shown as a plus for state, national and local governments under “Mill operation” but they are not shown as plusses for any of the other columns for which jobs or business income are listed as positives. As the “tax income” plus is within the RPDC assessment section, this inconsistency in treatment is alone sufficient to more than explain the apparent difference in the proportion of positives between the RPDC section and the remainder of the diagram.
Double standards (where an impact is shown as a negative but its reverse is relevant and is not shown as a positive) are also fairly common. For instance the impact on social and sporting relationships of people leaving rural areas is assessed as a negative (never mind that they may move to areas where their own such relationships are actually more diverse). But where job creation in areas is cited as a positive, there is no matching positive for improved social and sporting relationships as a result of people coming in to wherever jobs are created.
Miscellaneous comments
In the previous thread, Mike Bolan referred to impacts such as pain and emotional suffering which were supposedly unverifiable. Claims relating to such impacts are almost entirely absent from the diagram, being touched on obliquely and arguably in less than 1% of impact claims.
Mr Bolan also referred to use of “multiple perspectives” and referenced Ian Mitroff. Dr (his tertiary qualifications entirely in engineering of all things) Mitroff is an organisational theorist specialising in crisis management. This is not so different to Bolan’s own specialty but there is nothing about Mitroff’s concepts that is specifically relevant to assessment impact design or survey design. Mitroff’s “multiple perspectives” model essentially involves attempting to solve problems using a range of different approaches simultaneously.
Whatever the merits of this approach (and amid all Mitroff’s rather clichéd science-bashing it would be hazardous to assume too much) there is nothing in this diagram that demonstrates such a method. It is more suggestive of a single rigid approach to the enquiry in question: from the starting point of a previous approach considered inadequate, set out to refute that approach by building (from various sources) a database of potential impacts that falls outside it.
Presentation errors are now and then apparent in the diagram. For instance, one green dot within the fast track process is erroneously included in the total box for impacts outside the fast-track process, rather than the total box for those within. This was the first such case checked by a fellow admirer of the diagram, which is presently taking up a quite large section of my wall.
In summary
The diagram does not present a useful “integrated assessment of impacts” for the following primary reasons:
• It fails to demonstrate that asserted impacts are real or provide evidence relating to them. Some, at least, are not, or have been wrongly classified as negatives when they should be positives or neutrals.
• It presents a massively skewed view of the overall balance of impacts, principally as a result of different standards towards positive and negative impacts, especially indirect impacts, and also as a result of repetition of negative impacts
• Even if the view presented was not skewed, a list of positive and negative impacts would not be a useful assessment of impacts, because assessing impacts takes into account the magnitude of impacts and not just their number
The diagram does not present a representative summary of community views on the impact of the proposal because its respondent base is skewed, and is evidently not sufficiently knowledgeable and/or motivated to identify all positive flow-on aspects or avoid scientific error. At best, it represents a partial view of community perception of impacts, and in this role it is of limited use because it presents a qualitative view only. Its primary use is as a database that may assist campaigners to argue their case if used with caution. As an assessment of actual impacts it is not in any way up to scratch.
Disclaimers
This review is not written with the intention of supporting or opposing the proposed pulp mill, although I am unconcerned about its impact on whether that occurs (and suspect there will be no impact anyway). For the long history of the project I have generally not held an advocative view on whether or not the mill should go ahead, although I have opposed the expenditure of government money on all aspects of its assessment and the clearing of vegetation prior to the provision of evidence that the mill is really on track. In very recent times, in the absence of any firm indication that the venture is on track, and given that Gunns insisted rapid approval was necessary to avoid severe economic losses in calling for the RPDC process to be foreshortened in the first place, it has become my view that the proposal should be axed or moved elsewhere in the interests of public certainty and to ensure that local politics connects better with realities than with far-fetched hypotheticals. Alternatively, if Gunns are really serious about it and able to do it then it should be returned to the RPDC to complete the original assessment at Gunns’ expense. The events of recent years have made it clear there is really no hurry about all this!
The version of the diagram that I refer to here is around three years old. It is possible that significant improvements have occurred since, and if so, I look forward to hearing of them and being supplied with evidence that the diagram has been improved. Whatever the case, at a time when many copies of the poster were apparently in circulation, and the document had been revised several times, the poster was inadequate in the ways noted above, and that alone in my view is a matter of interest to debate given the claims made on the diagram’s behalf by Mr Bolan in the previous thread.
* Dr Bonham is a freelance invertebrate research consultant who has also worked as a scientific referee/editor and student-magazine editor and has had completed tertiary research methods training courses in both the sciences and the social sciences. His other current voluntary refereeing assignment is for the African Journal of Ecology!
