Evaluation of the Effectiveness of the ASTRA Research and Development Program

Background

The Institutional Development Programme for Research and Development Institutions and Universities (ASTRA) is one of the key measures alongside core funding and mobility support in implementing the Research and Development (R&D) and Innovation (RDI) Strategy “Knowledge-Based Estonia 2014-2020” as well as several other national strategies. The objective of the measure is to support structural changes in research and development institutions and universities, as well as the development of institutional responsibility areas and core activities. In total, 17 different universities and research and development institutions have received support from the ASTRA measure for the implementation of 36 projects. The total funding amount is 136 million euros.

Objective

The aim of the study is to assess the effectiveness, impact, and efficiency of the ASTRA measure, and to describe the lessons learned from its implementation.

Methodology

The study primarily employed qualitative methods such as analysis of documented information, interviews, and focus groups with representatives of stakeholders and project implementers.

Benefits

The results, conclusions, and recommendations from the evaluation of the ASTRA measure and the activities implemented within it help prepare new measures for the next structural fund period, including measures that assist Estonia (primarily the Ministry of Education and Research) in planning and supporting the institutional development of research institutions and universities.

Project contracting authorities: Ministry of Education and Research, Estonian Research Council
Project implementers: Baltic Studies Institute (lead partner) and LevelLab
Duration: 04/2021 ‍– 01/2022

View the final report

Analysis of the Implementation of the Qualifications System

Background

The qualifications system is part of the Estonian qualifications system – it connects the education system with the labor market and contributes to the development, assessment, recognition, and comparison of work-related competencies. The implementation of the qualifications system has been regulated by the Vocational Qualifications Act since 2001, and the system consists of four parts: the qualifications framework, vocational standards, awarding of vocational qualifications, and the vocational register.

Objective

The aim of the study is to identify the expectations of the participants in the qualifications system, their readiness for the implementation of the reform, and the transition to a skills-based approach, as well as to identify the potential challenges and risks associated with the planned innovations.

Methodology

During the study, the Estonian qualifications system will be compared with the experience of other countries. To assess the readiness and expectations of the participants in the qualifications system for the system’s reform, interviews and focus groups will be conducted with the direct and related stakeholders of the qualifications system: applicants for vocational qualifications, issuers of vocational qualifications, educational institutions, learners, stakeholders involved in the development of vocational standards (including employers and professional associations), career counselors, qualifications system specialists, and policymakers. During the study, risk assessment will also take place to ensure the realism and feasibility of the innovations.

Benefits

As a result of the study, input will be obtained for the renewal of the legal system underlying the implementation of the qualifications system and for the implementation of a skills-based qualifications system.

Project contracting institution: Ministry of Education and Research
Project implementer: LevelLab
Duration: 03/2023 – 10/2023

Overview of Estonia’s Research and Innovation System

Objective

The purpose of the background report was to provide a comprehensive overview and assessment of Estonia’s research and innovation system to the European Commission. The overview included the structure and organization, management and funding, development, and other aspects of the research and innovation system. An assessment was provided on the management of the Estonian research and innovation system, the funding system for research and higher education, and external financing over a period of 10 years.

Main results

  • The most significant driver of economic development and growth is innovation, particularly innovation based on research and development (R&D).
  • The capabilities necessary for creating innovations are needed for the national innovation system to learn, develop the necessary human capital, stay abreast of international advances in science and technology, and maintain its productivity and competitiveness.
  • Innovators typically do not innovate alone but within national and international innovation systems that involve multiple participants and institutions. Therefore, policymakers must ensure that the combination of policies related to various parts of the system is balanced and that interventions across the entire innovation system are coordinated.

Methodology

Quantitative data and analysis, as well as document analysis, were used in the preparation of the background report.

Project contracting authority: European Commission
Project implementers: Technopolis Group Estonia and LevelLab
Duration: 10/2018 ‍– 02/2019

View the final report

Interim Evaluation of the Implementation Plan for EU Cohesion Policy Funds

Objective

The purpose of the interim evaluation was to assess the relevance, effectiveness, impact, and sustainability of the implementation plan and the activities carried out within it. The evaluation covered all 14 priority areas of the implementation plan.

Results

The interim evaluation concluded that the sectoral objectives and activities of the implementation plan remain relevant today, and there is no need for significant changes to achieve the objectives of the implementation plan.

As of December 31, 2018, 32% of the €4.6 billion from the EU Cohesion Policy implementation plan for 2014–2020 has been disbursed, totaling €1.5 billion. The support received from the EU Structural and Investment Funds has clearly contributed to achieving the objectives of all priority areas: the funding has a direct impact on increasing employment, improving public health, developing the living environment, and achieving other objectives. The impact of Structural Funds is long-term and manifests in changes at various levels. The operational program has had a significant impact on achieving Estonia’s key strategic objectives.

Challenges and Recommendations

Regional development balance remains a significant challenge. Overall, regional disparities remain significant, and there has been no noticeable shift towards more balanced territorial development. Close to 50% of EU funds have been invested in Harju and Tartu counties, leading to a widening of regional disparities. EU funds should be directed towards preventing problems and developing comprehensive solutions in regions, with an increased focus on the role, awareness, and competence of local governments in regional investments.

Additional funds from the performance reserve should be directed towards preventive measures and value-added activities such as research and development, innovation in entrepreneurship, education, and regional development.

Research methodology: focus group interviews, individual interviews, workshops, document analysis (documentation of projects, measures, and implementation plans), and quantitative analysis (financial, output, and outcome indicators of measures).

Project contracting authority: Ministry of Finance
Project implementers: Technopolis Group Estonia (lead partner), Ernst & Young Baltic, Praxis, Baltic Institute of Research and LevelLab
Duration: 11/2018 ‍– 04/2019

View the final report

Estonian Management Field Study 2021

Background

Studies examining the field of management in Estonia have been commissioned by Enterprise Estonia at five-year intervals. Since the completion of the previous study in 2015, several developments have taken place in the economic and business environment, in companies’ overall capabilities and management capacity, and in managers’ preparation for leadership roles. In addition, the field itself has developed rapidly in recent years, with new approaches and trends emerging. Awareness of and consideration for these developments may significantly influence the future success of organizations.

Objective

The aim of the study is to identify the current state of the management field, the prevailing problems (including possible market failures) and shortcomings, and to develop proposals for increasing management capability, including with the support of the state.

Benefits

As a result of the study, we provide an assessment of the activities, scope, and quality of organizations operating in the field of management. The study identifies changes in the market and maps out key areas of action for increasing management awareness and capability in both the near and longer term. A self-diagnostic tool will also be developed to help managers assess the level of their strategic competencies, compare themselves with similar companies, and receive recommendations for possible follow-up actions.

Project client: Enterprise Estonia (EAS)
Project implementers: University of Tartu and LevelLab
Study duration: 08/2021 – 04/2022

View the final report

Analysis of Local Government Social Protection Expenditure and Its Reporting

Objective

The aim of the analysis is to examine how the reporting of local governments’ social protection expenditure differs in public sector financial accounting and reporting, and to identify the reasons for these differences.
The study will develop proposals for classifying social protection expenditures and improving the chart of accounts methodology so that the state and local governments can obtain comparable data for relevant target groups as well as key services and benefits. It will also develop appropriate indicators for comparing and forecasting the volume of social protection expenditure in the future, which could be used in policymaking.

Background

According to §6 of the Local Government Organisation Act, local authorities are responsible for organizing social assistance and social services as well as elderly care within their municipality or city. An analysis of these expenditures helps plan social protection services and the resources required to provide them.
Analysis of the chart of accounts data has revealed a problem: the level of social protection expenditure per capita and the level of municipal contribution to social protection vary significantly between local governments. The existing methodology for allocating expenditures does not provide sufficient comparable information for forecasting across key target groups, social services, and benefits.
One possible reason for this is the differing practices and methodologies used by local governments in reporting social protection expenditure.
As a result of the analysis, appropriate indicators will be developed for comparing and forecasting the volume of social protection expenditure in the future, and proposals will be made for changes to the chart of accounts methodology.

Project client: Ministry of Social Affairs
Project implementers: LevelLab and Geomedia
Study duration: 10/2021 – 06/2022

View the final report

Interim Evaluation of the Priority Area “Administrative Capacity”

Objective

The objective of the interim evaluation was to assess the achievement of the first two objectives of the administrative capacity priority area, as well as the implementation and effectiveness of the activities funded to achieve these objectives, while also considering other activities with similar goals.

Methodology

The evaluation applied an innovative evaluation model in which the activities under the administrative capacity priority area were grouped into three areas: development capacity of the government sector, leadership capacity of the government sector, and the quality of policymaking.
The evaluation model takes into account the key prerequisites of good governance, which emerge through the interaction of these three main components.

Main results

  • Measures implemented to strengthen the development capacity of the government sector have produced the expected results and remain relevant. However, a certain fragmentation and duplication in strategic planning has been identified as a problem, limiting the potential synergy between different measures.
  • Leadership capacity within the government sector has improved as a result of the supported activities, and implementation is progressing according to plan.
  • The quality of policymaking has also improved due to the supported activities: cooperation capacity has increased, inter-institutional collaboration has strengthened, and the substantive quality of impact assessments has improved.

Challenges

Currently, there is no long-term national development plan for strengthening administrative capacity, which means that the continuous development of administrative capacity is not guaranteed. It is recommended to develop a long-term, cross-sectoral development plan within the framework of the Estonia 2035 strategy to support the development of administrative capacity for both the central government and local governments.

Evaluation Methods: The evaluation included an analysis of financial, output, and outcome indicators of the measures and their activities. Focus group discussions and individual interviews were conducted. In addition, information was collected and analyzed through document analysis.

Project clients: Government Office of Estonia and the Ministry of Finance
Project implementers: LevelLab (lead partner), Institute of Baltic Studies, and Technopolis Group Estonia
Study duration: 03/2019 – 12/2019

Business Analysis of AS GoTravel

Background

AS GoTravel is an Estonian-owned travel agency that has been operating since 1992. The agency has offices in six Estonian cities: Tallinn, Tartu, Pärnu, Viljandi, Rakvere, and Jõgeva. The company serves both private and business clients and offers both domestic and international tourism services.
The company’s information system has been developed based on operational needs and investment capacity. The greatest focus has been placed on developing the public web environment as the main sales channel for package tours.

Objective

The objective of the business analysis was to describe the current state of the company’s information system and its connections with the company’s business processes, and on this basis provide development proposals for improving the information system and adapting the related action plan.

Benefits

As a result of the business analysis, GoTravel received development proposals for improving its information systems in order to simplify customer communication, internal information sharing, the signing of various contracts, and the use of the loyalty program, as well as to enhance the security of the public website.
The analysis also resulted in the development and implementation of a payment link solution for PDF invoices, created in cooperation between LevelLab and OÜ WeIT, which significantly simplifies and speeds up invoice payments for customers.

Project client: AS GoTravel
Project implementer: LevelLab
Study duration: 09/2019–11/2019

Evaluation of the Performance-Based Round for Labour Market Services

Background

Population aging is currently the main demographic trend in both Estonia and Europe, which, coupled with deepening labor shortages, requires effective measures to integrate older adults into the workforce. Through European Union support measures, the Results-Based Call Measure 3.2 was implemented, focusing on enhancing the employability of individuals aged 55 and older. Prior to announcing the call for proposals, the funding scheme for supported activities targeting the older age group was also amended, implementing simplified reimbursement methods – fixed-sum payments and standardized unit costs.

Objective

The aim of the study was to assess the relevance, effectiveness, impact, cost-effectiveness, and sustainability of the approach/funding scheme used, and to provide recommendations for organizing call for proposals in the future.

Methodology

The study employed both qualitative and quantitative data collection and analysis methods, including document analysis, quantitative data analysis, interviews, and a validation seminar. The purpose of the interviews was to gather in-depth input for evaluating the relevance and effectiveness of the program. The interviewees included grant recipients, project participants, representatives from the Ministry of Social Affairs, and representatives from the State Support Services Centre.

Benefits

As a result of the study, the justification for the results-based approach in open call tenders for employment policy implementation was identified, and suggestions were made for future improvements in organizing results-based grant schemes in the field of employment policy.

Project ordering institution: Ministry of Economic Affairs and Communications
Project executor: LevelLab
Duration:04/2023 ‍– 12/2023