IGL in numbers

Amount invested in trials through IGL

An RCT is an experiment where participants are randomly allocated to receiving an intervention. The randomisation

enables you to compare the effectiveness of the new intervention against what would

have happened if you had changed nothing

Innovation Growth Lab

Working in partnership to make innovation, entrepreneurship and business growth policy more impactful through experimentation and new evidence.

@IGLglobal

There is also too little innovation within policy itself and when there is, be it incremental or radical, we cannot tell whether it is for better or worse. How can an organisation develop and test new ideas systematically?

A really effective way is using a Randomised Controlled Trial (RCT). While currently underused and not always feasible, these are often seen as the most robust approach to demonstrating causality

RCTs can be used as part of a broader experimental approach to test anything from small tweaks to programmes to the overall impact of business or innovation schemes

IGL is a global partnership bringing together governments, foundations and researchers to scope, develop and test different approaches to increase innovation, support high-growth

entrepreneurship and accelerate business growth.

Over the last few years we are seeing a growing number of RCTs. Many of these are (co)funded by the IGL Grants programme, which has supported over 30 trials with close to $3 million from the Kauffman Foundation, Nesta and the Argidius Foundation. We are assisting a number of government agencies in their own journey to experimentation.

We are starting to learn valuable lessons about how we can encourage innovative ideas and support businesses, and many more lessons will emerge as the trials now in the field start to deliver results.

https://www.innovationgrowthlab.org/blog/what-are-we-learning-policy-experiments-increase-innovation-and-entrepreneurship

Visit our website or get in touch with us to find out more

innovationgrowthlab@nesta.org.uk www.innovationgrowthlab.org

€150 billion spent every year in EU supporting business to start, innovate and grow. Yet we know little about what works, and what doesn’t

2. An online database of trials from

around the world, ongoing and completed

3. An online toolkit to help those wanting to know why and how to become experimental

4. A policy brief on why we need more experimental policies and how to become more experimental 5. A regular series of blogs covering

advice and results from trials and new policy ideas

Creating a global community around

experimentation

Our three annual conferences have been attended by >650 senior policymakers, practitioners and researchers from over 40 countries

85% of IGL2018 participants were very likely to recommend the IGL Conference to their colleagues Our five research meetings brought together top researchers to present and discuss new experimental research in innovation, entrepreneurship and growth

Grants Programme

>1,100

attendees at our conferences and workshops

15

agencies we’ve worked with

>85

researchers in our network

Amount invested in trials through IGL

An RCT is an experiment where participants are randomly allocated to receiving an intervention. The randomisation

enables you to compare the effectiveness of the new intervention against what would

have happened if you had changed nothing

PR ELI M IN AR Y RE M AR KS AB OU T SCI EN CE , T ECHN OL OG Y AN D INNO VA TIO N PO LIC Y EV AL UA TIO N IN LA TIN AM ERIC A Ad rian a B in *, R af aela M . d e An dr ad e**, Li ssa Vas co ncello s Pin heir o*, S er gio Lu iz M on teir o Salle s- Filh o**

*School of Applied Sciences **Department of Science and Technology Policy, Geosciences Institute (University of Campinas/UNICAMP) Theresearchfollowsathree-phase methodology:collect,codeandanalyze. Collectionphasereferstoidentifying evaluationstudiesofSTIpoliciesinLA countries.AsdefinedbySIPERproject methodology,qualifiedevaluationstobe includedinthestudyarethose:(i)onscience andinnovationpolicy;(ii)evaluatingaclearly identifiable,specificprogramorgroupof programs;(iii)havingadistinguishable methodology;and(iv)providingsomesortof evidence.

CountryCollectedCharacterized Argentina2624 Brazil3317 Chile3520 Colombia1816 Mexico1818 Uruguay1616 Total146111 WhoTotal External(independent)66,7% Internal24,3% External(independent) andinternal4,5% External(withingovernment)3,6% External(withingovernment) and External(independent)0,9%

TimingTotal Interim94 Ex-post11 Ex-ante6

M ETH OD OL OG Y

Thefocusofthisworkistopresentanon-goingexperienceofcollecting,codingand analyzingScience,TechnologyandInnovation(STI)PolicyEvaluationsinLatinAmerica (LA),withemphasisintheevaluationdesignandmethods.Theresearchispartofabroader initiativenamedScienceandInnovationPolicyEvaluationsRepository(SIPER), coordinatedbyManchesterInstituteofInnovationResearch(MIoIR).SIPERisacentral sourceofknowledgeonscienceandinnovationpolicyevaluations.Itsaimistwofold:(i)to provideon-lineaccesstoauniquecollectionofpolicyevaluations,locatedatasingle location;and(ii)toallowpolicylearningbyprovidinganinformedanalysisofthedatabase contentsthatisbothsearchablebypolicymakersandotherstakeholdersandwhich providesthebasisforadditionalacademicanalysis.

IN TR OD UC TION

Table 1 –Evaluations collected and inserted on SIPER PreliminaryresultsshowthatSTIpolicyevaluationactivityinLAisrecentand heterogeneousacrosscountries.Ingeneral,thepurposesoftheseevaluationsareboth formativeandsummative(62documents),followedbythosethatareonlysummative(36) andonlyformative(13)evaluations.

RE SU LT S (1) Ta bl e 2 – W ho co nd uct ed Ta bl e 3 – Ti m in g

Ch ar t 2 – Asp ec ts of th e pr ogr am e xa m in ed b y th e ev alu at io ns

Regardingdatacollection,theuseofexistingdatabases(101evaluations)standsout;51 evaluationsemploysurveysand44employinterviews;analysisofscientificpublications appearin18documentsandfocusgroup/workshop/meetingsin17.

Ch ar t 3 – Da ta Ana ly sis

RE SU LT S (2)

Quasi-experimentalandnon-experimentaldesignsrepresentthemajorityofevaluation reports,correspondingto52and46evaluations,respectively.84evaluationsmeasurethe long-termimpactsofpolicymeasures,withemphasisontheuseofscientificand technological,socialandeconomicindicators.Amongtheevaluationsthatmeasured economicimpact,26analyzedcost-benefitorreturnsoninvestment.Asetof53 evaluationsmeasuresadditionality,25ofwhichanalyzeonlyadditionalityofoutput,8 additionalityofinputand1behavioraladditionality.Thereisalsoacombinationofinput andoutputadditionality(11documents)andofthethreetypes(5documents).

CON CL USION S

•Theanalyzescarriedoutsofarindicateagrowingmovementtowardsthe institutionalizationofSTIpolicyevaluationpracticesinLatinAmerica,inlinewiththe growingimportanceofthesepoliciesandtheperceptionoftheircontributionto countries'economicandsocialdevelopment. •However,therearefewvariationsonthemethodologicaldesignsandindicatorsused, evidencingtheneedforsubstantiveadvancesinthisfield. •Complementaryanalyzesshouldbeperformedafterthecompletionofthecollection andcharacterizationphasesoftheevaluations,seekingtoidentifytheoccurrenceofa relationshipbetweenthevariablesanalyzed,aswellasthecountries'profileregarding STIpolicyevaluation. •VisittheSIPERrepository:http://si-per.eu

Research funded by: CNPqand FAPESP

Ch ar t 1 – Ty pes o f po licy RE FE RE NC ES

EDLER,J.etal.(2012).ThepracticeofevaluationininnovationpolicyinEurope. ResearchEvaluation,London,v.21,n.3,p.167-182. SCRIVEN,M.(2009).Meta-EvaluationRevisited.JournalofMultiDisciplinary Evaluation,6(11),iii-viii. STUFFLEBEAM,D.L.;CORYN,C.L.S.(2014).Evaluationtheory,models,and applications.Secondedition.SanFrancisco(EUA):Jossey-Bass.

CodingphasewasdedicatedtocharacterizationofcollecteddocumentsfollowingSIPER requirementsbasedonasurvey,whichincludes:(1)Relatedpolicymeasure characteristics;(2)Evaluationcharacteristics:(2.1)Basic;(2.2)Topicscovered;(2.3) Design;(2.4)Datacollectionmethods;(2.5)Dataanalysismethods;(2.6)QualityIssues; and(3)Documentproperties.Finally,thelastphasecomprehendstheuseofcodified informationinordertodiscussstate-of-artofSTIevaluationpracticeinLA. 020406080100120

AltmetricsNetwork AnalysisQualitative/quantitative analysis of textsIntellectual property (IP)Input/output, cost/benefitCase StudyPublicationsEconometricsDescriptive statistics 12

555

9 24

8989 77 1

2 1

4 2

4 12 1

1 1

3 1

4 4

2 11 1

4 1

111

2 1

4 1

12 051015

25 199820032005200620072008200920102011201220132014201520162017 Prizes and AwardsNon-financial support (e.g. trainings, advisory) Infrastructure supportIndirect financial support: tax & fiscal incentives Direct financial support: scholarships, fellowships, etc.Direct financial support: grants, loans, contracts, etc.

24810102630343840415053545758718497 020406080100120Minority/inclusivity issuesGender issuesNetworkingCareerMobilityAppropriateness of goalsValue for money/return on investmentAppropriateness of designCollaboration/partnershipProgramme implementation efficiencyCoherence/complementarityDegree of satisfaction of stakeholdersAdditionalityAppropriateness of the rationale of the measureUptake of programmeGoal attainment/effectivenessPolicy/Strategy DevelopmentOutcomes and ImpactOutputs

PR ELI M IN AR Y RE M AR KS AB OU T SCI EN CE , T ECHN OL OG Y AN D INNO VA TIO N PO LIC Y EV AL UA TIO N IN LA TIN AM ERIC A Ad rian a B in *, R af aela M . d e An dr ad e**, Li ssa Vas co ncello s Pin heir o*, S er gio Lu iz M on teir o Salle s- Filh o**

TimingTotal Interim94 Ex-post11 Ex-ante6

M ETH OD OL OG Y

IN TR OD UC TION

RE SU LT S (1) Ta bl e 2 – W ho co nd uct ed Ta bl e 3 – Ti m in g

Ch ar t 2 – Asp ec ts of th e pr ogr am e xa m in ed b y th e ev alu at io ns

Ch ar t 3 – Da ta Ana ly sis

RE SU LT S (2)

CON CL USION S

Research funded by: CNPqand FAPESP

Ch ar t 1 – Ty pes o f po licy RE FE RE NC ES

AltmetricsNetwork AnalysisQualitative/quantitative analysis of textsIntellectual property (IP)Input/output, cost/benefitCase StudyPublicationsEconometricsDescriptive statistics 12

555

9 24

8989 77 1

2 1

4 2

4 12 1

1 1

3 1

4 4

2 11 1

4 1

111

2 1

4 1

12 051015

Figure 2: Median values of Employment by Year, n = 72.

Th e im pa ct o f t he E XI ST B us in es s St ar t-u p G ra nt on c or po ra te g ro w th : A gr ou p co m pa ris on fo r D re sd en (G ER ) In tr od uc tio n

The EXIST Business Start-up Grant (BSG) is one of the most important govern- mental programs in Germany to support founders by turning their business idea into action. This paper investigates the start-ups’ corporate development using a peer group comparison on longitudinal data. Research Question Do BSG-funded start-ups outperform non-funded industry peers in terms of: (i)Risk of cessation (ii)Survival time (iii)Employment (FTE) development (iv)Revenue development Keywords • corporate growth, governmental start-up assistance, innovation policy, longitudinal analysis, peer group comparison Conference pillars mainly addressed • Leading edge concepts, tools and methods to assess impact of R&I policy • Effects of and policy learning from impact evaluation Acknowledgement • Cornelia Ernst, Dresden University of Technology • Marco Rösler, dresden|exists

M et ho ds

Data set The self-collected dataset was created by conducting desk research and field research (online survey) due to a lack of reliable and publicly accessible longitudinal micro-level data at the time (2017). Cross-sectional • Treatment group (TBSG): limited liability corporations (Ltds.) in Dresden (GER) funded by the BSG (n=21) [provided by dres- den|exists, the local authority responsible for the BSG] • Control group (CBSG): non-funded peers (n=18) which would have been eligible for a BSG funding. The eligibility criteria were assessed on information related to the Ltds.‘ time of incorporation, e.g. its registered object, in retrospect by two inde- pendent experts (four-eyes principle). Two independent datasets were merged, one provided by Dresden Chamber of Com- merce and one retrieved from the database Amadeus. Longitudinal • Period of incorporation:2008 - 2011 • Observation period: First five post-incorporation years [data on corporate development obtained from the two inde- pendent datasets and an online survey]

Co nc lu si on a nd Po lic y re co m m en da tio n

• Funded Ltds. do not outperform industry peers in terms of (i) risk of cessation and (ii) survival time. • Rather non-funded industry peers perform better than funded Ltds in terms of (iii) employment and (iv) reve- nue development. • According to the online survey, for 8 out of 11 respondents it would have been unlikely or even very unlikely having founded without the BSG funding. So, treatment and control groups’ Ltds. might differ in their pre- treatment willingness to incorporate. →The BSG in Dresden might not fund the founders ready to fly high, as intended, but helps start-up seeds to see the light of the day, which tally with the “theory of external assistance as the support option of last re sort” (Juita-Elena (Wie) Yusuf 2017). Policy recommendation 1. The start-up agents responsible for the BSG funding should increase active sourcing in order to not rely on the people who come in and apply for a BSG funding. 2. BSG funding might provide even more guided preparation during the one year funding period with regard to the “theory of outside assistance as a knowledge resource” by Chrisman and McMullan (2004). 3. Funded Ltds. should be encouraged to self-reliance by loosening university-related ties since proximity to uni- versity is not necessarily related to better performance of the start-ups (Doutriaux 1987).

Re su lts

(i) Risk of cessation • Visual inspection of proportional risk of cessation (Figure 1) is inconclusive because the curves cross each oth- er but lie in the same range overall. However, the result of a performed log-rank test (p = .820) provides evi- dence that the survival distributions of the two groups are not statistically different. This is supported by a test based on Schoenfeld residuals (p = .397). (ii) Survival time • Is almost equal (Table 1) between the two groups which blends into the result of almost equal risk of cessa- tion. (iii) Employment development • Related to employment overall, Wilcoxon’s two-sample rank sum test (Table 1) provides evidence to negate an overperformance by funded Ltds., which is displayed in Figure 2. On the contrary, groups‘ means indicate higher employment for the funded Ltds. (Table 1). • For employment development, the results of Analyses of Variance are ambiguous and highly effected by outli- ers which is depicted in Figure 2. Over or underperformance of funded Ltds. depends on the industry sector. (iv) Revenue development • Related to revenue overall, independent t-test (Table 1) provides evidence to negate an overperformance by funded Ltds. Groups‘ medians indicate the same (Figure 3). • Moreover, there are higher probabilities of generating higher revenue for control peers. n MMdnt-test Wilcox. r2M-W stat. (ii) Survival time (5 years)

Total3952.860.901.000.509 Treatment 2152.360 Control1853.360 (iii) Employment Total728.93 .032**.072°.654 Treatment 479.62 Control257.36 (iv) Revenue (e.v.n.a.)

Total764.24 .082*.040° Treatment 463.83.5 Control304.74 n represents the amount of observations for Ltd. i in point of time j. *, **, ***. Denote significance at 10%, 5%, and 1%, respectively (two-tailed test). °, °°, and °°°. Denote effect sizes: small (°): .01≤r2 <.09, medium (°°): .09≤r2 <.25 , and large (°°°): .25≤r2.

Table 1: Results analysis of differences for Survival time, Employment and Revenue by Treatment. Philip Dörr FSU Jena, Chair of Economics/Microeconomics Carl-Zeiss-Straße 3, 07743 Jena, Germany philip.doerr@uni-jena.de

Li m ita tio n an d Fu rt he r re se ar ch

• Group sizes of 21 and 18 companies do not meet the self-proclaimed sample size threshold of n > 30, with regard to Student’s t-distribution and the associated t-test. • The results are not representative for Germany and must be interpreted even for the case of Dresden with caution. Further research 1. Replicating on a larger sample size to meet the threshold of n=30, at least. 2. Detecting peers with another method, e.g. text mining the companies’ object, to certify judges’ matching (investigator triangulation). 3. Verifying the same pre-treatment conditions for the two groups to justify the matching. 4. Conducting qualitative analysis to investigate the reasons for the corporations’ devel- opment on an individual basis. 5. Taking advantage of variables from the self-collected dataset, which are not considered in this paper already, e.g. Year of incorporation. References Chrisman, J. J., & McMullan, W. E. (2004). Outsider Assistance as a Knowledge Re-source for New Venture Survival. Journal of Small Business Management, 42(3), 229–244. Doutriaux, J. (1987). Growth pattern of academic entrepreneurial firms. Journal of busi-ness venturing, 2(4), 285–297. Juita-Elena (Wie) Yusuf. (2017). The effectiveness of entrepreneurial start-up assistance programs: Evidence from the U.S. Panel study of entrepreneurial dynamics. In Preventing Ageing Unequally (pp. 1–9). OECD Publishing.

Measurement (i) Risk of cessation(ii) Survival time(iii) Employment development (iv) Revenue development Method Variables Specification Analysis of Differences Cox proportional ha- zards resgressionSurvival time analysisTwo-way Analysis of varianceCumulative odds ordinal logistic regression Survival time by TreatmentSurvival time by TreatmentTreatment and Year on FTE, Treatment and Industry on FTERevenue by Treatment, Industry, and Year • Log-rank test • Scaled Schoenfeld resi- duals

• Kaplan-Meier survival curves • Type III sums of squares • Bonferroni adjustment

• Proportional odds • Post-estimation: discrete and marginal change Figure 3: Medians values of Revenue by Year, n = 76.

Figure 1: Kaplan-Meier survival curves.

Figure 2: Median values of Employment by Year, n = 72.

Th e im pa ct o f t he E XI ST B us in es s St ar t-u p G ra nt on c or po ra te g ro w th : A gr ou p co m pa ris on fo r D re sd en (G ER ) In tr od uc tio n

M et ho ds

Co nc lu si on a nd Po lic y re co m m en da tio n

• Funded Ltds. do not outperform industry peers in terms of (i) risk of cessation and (ii) survival time. • Rather non-funded industry peers perform better than funded Ltds in terms of (iii) employment and (iv) reve- nue development. • According to the online survey, for 8 out of 11 respondents it would have been unlikely or even very unlikely having founded without the BSG funding. So, treatment and control groups’ Ltds. might differ in their pre- treatment willingness to incorporate. →The BSG in Dresden might not fund the founders ready to fly high, as intended, but helps start-up seeds to see the light of the day, which tally with the “theory of external assistance as the support option of last re sort” (Juita-Elena (Wie) Yusuf 2017). Policy recommendation 1. The start-up agents responsible for the BSG funding should increase active sourcing in order to not rely on the people who come in and apply for a BSG funding. 2. BSG funding might provide even more guided preparation during the one year funding period with regard to the “theory of outside assistance as a knowledge resource” by Chrisman and McMullan (2004). 3. Funded Ltds. should be encouraged to self-reliance by loosening university-related ties since proximity to uni- versity is not necessarily related to better performance of the start-ups (Doutriaux 1987).

Re su lts

Total3952.860.901.000.509 Treatment 2152.360 Control1853.360 (iii) Employment Total728.93 .032**.072°.654 Treatment 479.62 Control257.36 (iv) Revenue (e.v.n.a.)

Li m ita tio n an d Fu rt he r re se ar ch

• Kaplan-Meier survival curves • Type III sums of squares • Bonferroni adjustment

• Proportional odds • Post-estimation: discrete and marginal change Figure 3: Medians values of Revenue by Year, n = 76.

Figure 1: Kaplan-Meier survival curves.

Qualitative analysis of the evidence on characteristics of German cluster policies, their intended effects, and the availableevaluation studies; three components are combined: •Cluster initiatives: Internet-based survey of 600 cluster initiatives from which 370 focus on innovation and knowledge transfer, deeper analysis of30 randomly chosen technology-oriented cluster initiatives, •Cluster policies: Survey of about 25 cluster programmes of the Federal Government and the 16 State Governments, •Evaluationstudies:Analysis of the available cluster literature (formal evaluations, empirical studies) on German cluster policy. 1.How does the diversity of practical cluster policies relate to their intended effects? 2.To what extent and on what methodological basis were the Federal and State cluster programs evaluated? 3.What effects were identified and which questions remain open in respect to (a) the impact of cluster policies and (b) the evaluation methods applied?

1.Cluster policies (and the related concept of network promotion) embrace a large range of different instruments from financial support of common activities to R&D funding of projects related to a common cluster mission. 2.Cluster initiatives are the intermediate organizations needed to promote cluster development; based on different characteristics, four types of cluster initiatives could be identified which relate to different intended policy effects. Types of Cluster Initiatives, Characteristics, and Effects 3.Several evaluation studies have been conducted both at the Länderlevel and for Federal programmes. These evaluations were rather different in respect to which policy effects could be assessed.3Major results are: •Qualitative studiesshow that the policies were successful in shaping supportive environments for cluster actors and in offering assistance for existing and emerging R&D cooperation networks.4 •These activities were successful only in a part of the cases; there is more relevant knowledge on effects on the cluster than on the programme level. •Success on cluster level was determined by multiple factors, including program design, financial endowment, abilities of the cluster management, but also the individual constellation of the cluster/cluster initiative. •Quantitative analysesidentified effects for individual firms’ input and output indicators. However, doubts arise with respect to the causal interpretation of some of the output-related results because of the probable time structure of impact patterns.5 4.While primary effects (induced activities and network development) could be clearly identified, much less is known about secondary policy effects (such as resulting inventions and innovations on the firm level, and impact on cluster population and performance on the cluster level).6 Evidence on Cluster Policy Impact from an Evolutionary Perspective Selected References 1Uyarra, E., Ramlogan R. (2016), see also the methodological discussion in Schmiedeberg, C. (2010). 2 Kiese, M. (2012). 3Wessels, J. (ed.) (2008). 4E.g. Dohse, D. (2007)and Töpfer, S., U. Cantner, H. Graf (2017). 5E.g. Falck, O., S. Heblich, S. Kipar (2010), Crass, D., C. Rammer, B. Aschhoff (2016) and Engel, D., V. Eckl, M. Rothgang (2017). 6 Rothgang, M., J. Dehio, B. Lageman (2017). *We thank Martina Böhmel for her contribution to the research project.

M ic hael Rothg ang (RWI), Ber nh ar d Lag em an, Anne-M ar ie Sc hol z (ISG)

Re se ar ch Que stion s

Availabilityandmethods •Manyofficial evaluation studies, however mostly ex ante or accompanying evaluations (some ex post evaluations mainly of Federal Programmes), •In many cases simple qualitative assessments, based on a limited set of indicators, •Wefind a smallnumberofmorein-depthclusterpolicyimpact analysesin theresearchliterature(somequalitative, some quantitative). Challenges •Most programme effects are the result of complex effect mechanisms with many external influences. •Politicians’ interest in short-term results contradicts basic structural features of these programmes. •Major policy effects can only become visible after longer time- lags and based on a resource-consuming thorough analysis.

St arting poin t

Contact: Michael Rothgang, RWI-Leibniz-Institute forEconomicResearch, Hohenzollernstraße 1–3, 45128 Essen, Germany, michael.rothgang@rwi-essen.de

Finding s Implic ation s

1.To gain more relevant insights into the long-term impact of cluster policy, impact research should reconstruct chains of effects of policy-induced cluster development and confront the findings with the original narratives of policy makers on expected cluster evolution. 2.New developments in the evaluation methodology of cluster policies are needed: improved qualitative approaches to understand the sequential effects of policy impulses (e.g. process tracing), the thoughtful application of advanced quantitative approaches based on theoretical impact modelswhich take account of the complexity of impact patterns,innovative combination of qualitative and quantitative approaches.

Empir ic al Appr oac h

Cluster policies in Germany Cluster policies have changed structural policies in Germany in a substantial and sustainable manner in the past decades: 1.All State Governments use cluster promotion to promote regional growth and innovation, while the Federal Government implemented several own cluster programmes.2 2.Cluster policies are rather popular due to (i) a sound theoretical basis in cluster literature, (ii) flexibility of the instrument with cluster organizations as intermediaries that take up firm needs. 3.There are substantial differencesbetween programs in respect to funding and targets. To improve cluster policies it is important to understand the mechanisms of their policy success and failure. Theory-basedexpectations Cluster policies should •promote existing clusters and function as midwife of emerging clusters, •encourage cooperation between firms, academia and state authorities, •increase the developmental potential of regional economies and promote technology evolution. However... •Evidence on the actual effects of cluster policy is limited,1 •The experiences with practical cluster policy in Germany have not been assessed in a systematic manner yet.

WHA T AR E THE IMP ACT S OF CL US TER PO LIC IE S? THE GER MAN E XPERIENC E Ev alu ation st udie s

M ic hael Rothg ang (RWI), Ber nh ar d Lag em an, Anne-M ar ie Sc hol z (ISG)

Re se ar ch Que stion s

St arting poin t

Contact: Michael Rothgang, RWI-Leibniz-Institute forEconomicResearch, Hohenzollernstraße 1–3, 45128 Essen, Germany, michael.rothgang@rwi-essen.de

Finding s Implic ation s

Empir ic al Appr oac h

WHA T AR E THE IMP ACT S OF CL US TER PO LIC IE S? THE GER MAN E XPERIENC E Ev alu ation st udie s

Making the invisi ble visible To war ds st rat eg ic m easu res o f r esear ch an d t ech no lo gy o rg an isat io ns ( RT Os) Kir si H yyt in en , Ka tri Ka llio , O lli K uu sist o & Ar ho Su omin en

VTT Technical Research Centre of Finland Ltd Contact Kirsi Hyytinen Tel. +358 40 5818495 Kirsi.Hyytinen@vtt.fi

Co nclu sio ns

To make the invisible visible we need more diversified evaluation approach and indicators which enable to analyseRTOs impact in the systemic and complex environment. RTOs’ impact cannot be demonstrated with one –or even a few single indicators: a comprehensive picture of impact requires multi-criteria approach, a variety of indicators and data gathering from multiple sources. Policy measures and evaluation practices could be updated to perceive the significance and ‘hidden performance’ of innovations in systemic context: innovations do not emerge without evaluation mechanisms that support their creation.

R TO s ar e tacklin g wi th syst em ic an d co m plex ch allen ges

There are no simple solutions to complex societal problems. Modern innovation theories emphasizes that tackling system level problems requires considering three perspectives: 1)Solutionsrequiredeepintegrationoftechnologiesand service-basednovelties. 2)Collaborationbetweenmultipleactorsfromdifferent sectorsofsocietyisrequired. 3)Developinganddisseminatingsolutionsrequires understandingofcustomerandcitizenneedsandthe policy-makingcontext.

Cu rr en tin no vat io n m easu res do not m ake the im pact visib le

TargetofRTOsistosolvebroadsocietalchallenges. However,currentinnovationindicatorsdonotcaptureRTOs’ multiplerolesandimpactinsystemicenvironment. Therearetwomainreasonswhytraditionalevaluationand measuresfail.Thefocusison: •techno-economicaspectsofinnovationandimpact,which arenotabletocapturetherealityofinnovations(e.g. system,socialandserviceinnovations)and“hidden performance”. •inputanddirectoutputs.Thereforewelackdataabout impact.

VT T’s st rat eg ic evalu at io n fr ame wor k

VTT’snewstrategicevaluationframeworkandrelated measuresofsuccess(KPIs)integratemulti-criteriaapproach ofinnovationtothetraditionalbalancedscorecard(BSC).The frameworkexpandstraditionalinnovationindicatorstowards broadersocietaltransformationsinaccordancewithVTT’s strategy.TheymakeVTT’slong-termimpactvisibleinfour categories: 1)Benefitforsociety 2)Benefitforcustomers 3)Excellenceineverythingwedo 4)Sufficiencyoffinancialresources Figure1. The focus in the impact analysis is typically in the linkages between the visible innovation and visible performance, which do not capture the non- technological innovation and hidden performance of innovations. (Djellal& Gallouj, 2010).

Figure2. Framework to measuresuccessat VTT. Biasedinformationmaycauseinaccurateanalysisand interpretationsandleadtoinappropriatedecisions.Inorderto paintatruthfulandcomprehensivepictureofRTOs’ impact,andtoprovidebetterinformationformanagingand learning,weneeddiversifiedevaluationapproachand measures.

VTT – be yond the obv ious

www.vttresearch.com Thelong-termsocietalimpactisbasedonVTT’sabilityto createcustomerimpactinshorterterm.Toensurethis,we needtosupportcustomers’successandgrowthandactively createnewpartnernetworks.Aprerequisiteforcreating impactishigh-levelcompetenceinresearchandinnovation. Finally,operativeexcellenceandfinancialprofitabilityshould securethebalanceofVTT’sfinanceinthelongrun.

Making the invisi ble visible To war ds st rat eg ic m easu res o f r esear ch an d t ech no lo gy o rg an isat io ns ( RT Os) Kir si H yyt in en , Ka tri Ka llio , O lli K uu sist o & Ar ho Su omin en

VTT Technical Research Centre of Finland Ltd Contact Kirsi Hyytinen Tel. +358 40 5818495 Kirsi.Hyytinen@vtt.fi