In early 2017, UNESCO contracted the independent Technopolis Group in Paris to conduct an Evaluation of the UNESCO Science Report published in late 2015. The evaluation was presented to UNESCO’s Executive Board in October 2017. Here, we compare the findings of a stakeholder survey conducted as part of this evaluation with those of a reader satisfaction survey available on the UNESCO Science Report portal since 2016.
In the Evaluation’s survey of 99 users of the UNESCO Science Report, almost half of respondents came from academia (21) and National Commissions for UNESCO (24). The latter are government relays in Member States for the implementation of UNESCO’s programme. A further 13 respondents identified themselves as policy-makers in government and four as employees in the private for profit or non-profit sector. Eleven were employed by UNESCO field offices or by the UNESCO Institute for Statistics. Two-thirds of respondents were men.
Respondents said they used the report most for conducting research (56%), for learning purposes (51%), for monitoring and benchmarking (48%) and for policy advocacy (47%). Only 16% of respondents declared using it for fundraising purposes.
Report ‘fully in line’ with mandate and 2030 Agenda
Drawing on the survey’s findings, the Evaluation concluded that ‘the production of the UNESCO Science Report is fully in line with UNESCO’s mandate’… This was confirmed by all of the members of UNESCO staff and Member State Permanent Delegations interviewed as part of this evaluation’. Users considered the UNESCO Science Report ‘to be relevant and coherent vis à vis other UNESCO initiatives and tools in the field of science, technology and innovation (STI)’
In particular, the Evaluation found the UNESCO Science Report to be ‘in line with the ambitions of the 2030 Agenda for Sustainable Development. Specifically, the UNESCO Science Report is compatible with Sustainable Development Goal 9 (SDG 9), the pledge by countries to “build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation”. Target 9.5 calls upon countries to encourage innovation and substantially increase the number of researchers, as well as public and private spending on research and experimental development (R&D).
The evaluators found that ‘the case for continuing to support the UNESCO Science Report is strengthened by its high degree of relevance and the overall positive appreciation expressed by readers, in particular with a view to the significant potential for the UNESCO Science Report in contributing to influencing and monitoring progress towards the SDG target 9.5’. The Evaluation found that ‘the UNESCO Science Report could also potentially contribute to monitoring the contribution of science, technology and innovation to reaching other SDGs’.
Report provides baseline indicators
Despite the fact that OECD members accounted for 66% of global research spending in 2013, compared to 5% for low income and lower middle-income countries, two-thirds of geographical coverage in the UNESCO Science Report concerns the developing world, as information and data for these countries tends to be less readily available.
In order to ensure that as many countries as possible are able to contribute data, the UNESCO Institute for Statistics limits its range to basic R&D indicators, such as researchers per million inhabitants and gross domestic expenditure on R&D as a percentage of GDP, the two baseline indicators for Sustainable Development Goal 9.5.
‘UNESCO, the UNESCO Institute for Statistics and the UNESCO Science Report are seen as legitimate sources of information to monitor this target’, noted the Evaluation. The UNESCO Institute for Statistics ‘is generally considered as a complementary source of “raw data” on STI at the global level, rather than as a competing source of information’.
Report useful ‘bundler’ of information
For the Evaluation, stakeholders and users have clearly identified the uniqueness of the UNESCO Science Report vis à vis external information and data sources. ‘Almost half the evaluation survey respondents (53%) indicate that the information contained in the UNESCO Science Report is not available elsewhere, while a remaining 32% consider it to be only partially available elsewhere. In this sense, the USR appears to be considered by many as a “bundler” of information which may or may not be available elsewhere, which in itself represents an added value to the reader.
One survey respondent stated that ‘I mostly look for country/regional trends. These data are available but in disparate sources. The UNESCO Science Report can act as a one-stop-shop for such information’. A second respondent commented that ‘as it is public information, it can be obtained from other sources too but in the [UNESCO Science Report] it is presented in a convenient form for a researcher”.
The Evaluation observed that, ‘while other international organizations producing similar content (e.g. OECD) are also highly valued and recognized for the quality and value of their work, these tend to focus less on the social and economic implications of STI in countries outside of their constituencies (e.g. OECD member states, EU member states) and are thus to be viewed as less “global” in terms of data availability’.
Geographical balance and thematic relevance ‘good or very good’
‘Over 80% of [survey] respondents considered the geographical balance of the report to be good or very good (see Figure 9 from the Evaluation). Despite this, ‘a number of interviewees indicated that, rather than seeking to achieve a global coverage, the UNESCO Science Report should focus on countries – particularly developing ones – where the lack of reliable and up-to-date data on STI is still considered to represent a major roadblock to the promotion of STI policies and support’.
‘Around 90% of respondents considered the relevance of the themes covered by the report to be good or very good’ (see Figure 9). ‘In addition to this, the level of use and relevance of the different components and sections of the UNESCO Science Report appears to be roughly the same… The frequency of use of the different parts of the UNESCO Science Report appears to be spread evenly, according to survey respondents. The Executive Summary does appear however to be the most frequently consulted section of the report, but only by a slight margin’.
‘In spite of this, the qualitative interviews did reveal that there are specific chapters and sections of the 2015 edition of the UNESCO Science Report which are more frequently cited, when it comes to describing the value and use of the report. For instance, Chapter 3 of the report on the gender gap in science and engineering was frequently cited by interviewees as one of the pieces contained in the report of particular interest. In addition, readers tend to cite the country or regional chapters of their home countries and regions as the sections of the report they most often consult, or have read in detail. Yet, according to the website metrics analysis, the number of visitors to the individual chapters’ webpages differed significantly. Chapter 15 on Iran was visited the most, with 1,479 visitors [by early 2017], followed by Chapter 3 on ‘Tracking trends in innovation and mobility’, with 986 visitors, and by Chapter 1 (Executive Summary) entitled “A world in search of an effective growth strategy”, with 821 visitors.
Survey participants also expressed a high level of satisfaction with regard to the presentation and visual style of the report and its format: over 80% of respondents considered these aspects of the report to be good or very good.
Some respondents were critical of UNESCO’s communications on the report, the quality of the website and the frequency of release of the report. Many felt that a report that had grown to 800 pages, including the Statistical Annex, was now too voluminous a volume, even it did ensure global coverage.
Internaut survey mirrors findings of Evaluation
Readers surveyed on the UNESCO Science Report portal in 2016 in English and French also tended to appreciate the report. Three-quarters of respondents rated the report ‘excellent’ (27%) or ‘very good’ (47%). The remainder rated it ‘good’ (20%) or ‘adequate’ (6%). None considered it ‘poor’.
Thirty respondents to the online survey identified themselves as men and 21 as women. Originating from countries on every continent, they spanned all age groups: 5 were less than 30 years old, 10 were aged 30–39 years, 9 were aged 40–49 years, 7 were aged 50–59 years and 20 were 60 years of age or more. Two had retired from professional life.
The majority of respondents were researchers or policy analysts in the natural or social sciences and engineering. Many indicated more than one affiliation. For instance, some academics also worked for a private research institute, non-governmental organization or academy of science. One respondent described himself as a researcher, policy analyst, entrepreneur and journalist.
Only two respondents indicated that they worked for a government department or ministry. A third was a school teacher. Six stated that they were university students and eight that they worked in the private non-profit or for profit sector.
One-quarter of respondents (13) had heard about the report by word of mouth. Others had discovered it in the workplace (3), traditional press (2) on social media (3) and while surfing internet (3). Seven respondents said they followed the series, two had discovered the report at a library, six at a conference and one at an event presenting the report’s findings. Six had learned about the report through an academic article, one through a research project and another via a university course. (Tallies may not add up to 51, owing to some respondents not answering all questions.)
Almost half of respondents (24) said they had only read selected chapters from the report. A further 14 said they had read most of the report and six claimed to have read the full report. Four had only read the chapter on their country or region, one had searched for keywords and two had read none of the report.
When asked to identify those parts of the report they found most useful (multiple answers were possible), 23 respondents highlighted the socio-economic and geopolitical background to trends in STI, 21 the country profiles, 14 trends in innovation and 13 data trends.
The next most popular categories of information were the policy recommendations in each chapter (12 responses), the development of a national policy framework (12) and national scientific infrastructure (12), science mobility (11) and science diplomacy (10), emerging priority areas for R&D (10), trends in international scientific collaboration (9) and sustainable development (8) gender-related issues (6) the quantifiable targets for trade, R&D and sustainable development (5) business R&D (5) and short essays on emerging issues (4). Six respondents ticked the option ‘all of the above’.
Some 55% of respondents (28) planned to use the report for policy- or other decision-making, 24 for an academic research paper or to prepare a report, 23 for their personal knowledge, 13 to inform project development and 11 to inform international cooperation, 11 for a presentation to a meeting, 11 for advocacy, 4 for market research the same number for a journalistic article or blog, and 3 as input to course teaching. Three respondents were exclusively interested in the data.
One in ten respondents said they found the report hard to find with a search engine and a similar proportion had trouble opening the full report in PDF format (82 MB).
It is still possible for internauts to make their opinion of the report known. All people need to do is answer the multichoice questions in the online reader satisfaction survey.
Source: Reader satisfaction survey and Evaluation of the UNESCO Science Report.
No comments:
Post a Comment