This collective scrutiny has been fun - with some very pertinent issues raised. Let’s have a look at what this report is saying.
The first issue with any survey is the competence of the sample.
In relation to the scope of the survey – Scotland Visitor Survey 2011, the report says: ‘Over this period 1,882 overnight leisure visitors were surveyed including 540 visitors to Argyll & the Isles’.
A sample size of 1,882 for Scotland is perfectly acceptable – if that sample is properly structured. But a Scotland-wide survey of 1,882 with 540 – or just shy of 29% – of that sample from a single area is not.
The dominant impact of that sub-sample accounts for the finding given that: ‘The age profile of visitors to Argyll & the Isles is similar to that of Scotland as a whole [Ed: our emphasis] with 29% aged under 35, compared to 32% of all Scottish visitors.’
Then, as one reader pointed out below, having said that the 1,882 surveyed were ‘overnight leisure visitors’ with 540 of the 1,882, the report that: ’66% of visitors to Argyll and the Isles spend one or more nights in the area’ ought, by the report’s own founding statement, have read: ’100% of visitors to Argyll and the Isles spend one or more nights in the area’.
Since 100% of visitors to Argyll and the Isles will not spend one night in the area, something is clearly astray.
So since we started from here too, raising these questions, yesterday we went poking around the VisitScotland website and through the other area factsheets, like this one, relating to this survey.
Each had the same founding statement, but with the number of visitors specific to the area in question changed as appropriate, each given as out of the 1,882 total surveyed.
So, where we have here: ‘Over this period 1,882 overnight leisure visitors were surveyed including 540 visitors to Argyll & the Isles’, the 540 out of 1,882 for Argyll was replaced by:
- 1,139 for Edinburgh City
- 1,027 for Highlands
- 506 for Cairngorms National Park
- 564 for Fife
- 740 for Glasgow City
- 367 for Borders.
The problem is clear.
There is, on the VisitScotland website, alongside this Scotland Visitor Survey 2011, a Scotland Visitor Survey 2012.
A look at this reveals that neither survey is correctly titled – each ‘surveyed’ some of the regions and together they completed a ‘Scotland’ wide survey.
This has to mean that results published for this 2011 survey, where regional figures are compared with those of ‘Scotland as a whole’ [as they are here] cannot have been so compared. Regional figures in the 2011 survey can only have been compared with total figures for the selection of regions surveyed in that year. Figures for the rest of Scotland did not then exist. When they did, each of the second stage areas surveyed are given as being from a sample of ’1,427 overnight leisure visitors’ and, as in the 2011 survey, the individual numbers surveyed in each region concerned add up to considerably more that the sample total of 1,427.
We looked into the situation from information available here on the visitscotland corporate website and took three screenshots , below, which together show the overall ‘behind the scenes’ approach to the survey, with the ‘short face to face interviews’ conducted seemingly doing little more than harvesting emails for later use.
The information makes no mention of how the people in the sample were selected or, as also asked by readers below, where they were first found for these ‘short face to face interviews’.
There is clearly a gap between what the regional factsheets say was done and what was actually done, with the latter much more extensive – although with its own methodology equally challengeable and unclear – on the basis of the public information given.
While there is a lot more that could pertinently be questioned on the published information, there is the one core issue.
Here we have the national tourism agency disseminating, under its own authoritative imprimatur, public information to the tourism industry and its regional sectors which is obviously misleading as it stands; and whose methodological foundation is inadequately disclosed.
Text of original article of 18th June
A rare weeding out of papers found amongst the delegate pack for the Argyll and the Isles 2013 Tourism Summit held in March, a paper from VisitScotland on the Regional Results for Argyll and the Isles of the 2011 Scotland Visitor Survey.
This was carried out for VisitScotland by TNS Research International.
The paper first clarifies that the fieldwork for this survey was done from July to October 2011 and that: ‘Over this period 1,882 overnight leisure visitors were surveyed including 540 visitors to Argyll & the Isles’.
The front sheet of this 4-page leaflet then cites a couple of facts distilled from the Argyll & the Isles part of this survey. These are:
- ‘The following information relates to summer visitors to Argyll & the Isles staying overnight in Scotland’.
- ‘The age profile of visitors to Argyll & the Isles is similar to that of Scotland as a whole with 29% aged under 35, compared to 32% of all Scottish visitors.’
- 66% of visitors to Argyll and the Isles spend one or more nights in the area. Amongst these visitors the average length of stay is 4.8 nights. Almost half of overnight visitors stay in the area for 1-3 nights [48%], 34% stayed for 4-7 nights and 18% stayed longer.’
There are some serious weaknesses in this report and in the underlying methodology of the survey.
Rather than simply identify them, it seems more fun for you to spot them and comment on what you see.
We’ll publish our own perspectives later today – and have fun in the meantime.