Digital transformation in the analysis of survey questionnaires and formulation of recommendations: AI solution for CARNET with the support of EDIH Adria

Challenges

The Croatian Academic and Research Network CARNET is continuously working on improving the digital maturity of the education system in the Republic of Croatia. As part of the e-Universities project, a ‘Framework for Digital Maturity of Higher Education Institutions in the Republic of Croatia’ was developed, as well as a model for self-assessment of digital maturity and related questionnaires for several groups of respondents at higher education institutions. However, processing a large amount of data and formulating individualized recommendations for each higher education institution proved to be a challenge given that it is a complex and resource-intensive task. For this reason, the need has been identified to introduce advanced technological solutions based on artificial intelligence, namely large language models (LLMs), to enable scalable and automated support for the digital transformation of higher education institutions.

Solution

As part of the TBI (Test Before Invest) service and with the professional support of the EDIH Adria consortium, CARNET launched the development and testing of a prototype of the LLM system entitled ‘Advisor for Digital Transformation of Higher Education Institutions in the Republic of Croatia’. The system is designed to automatically analyze data collected through survey questionnaires, enrich the context by using publicly available strategic documents from selected higher education institutions, and generate structured reports and concrete recommendations for digital transformation for the respective higher education institution. The prototype was successfully tested through a pilot project where the possibilities of automated processing, document integration into the prompt and interactive generation of recommendations were demonstrated. Although it is not a commercial solution ready for implementation, this proof-of-concept project has clearly demonstrated the technical feasibility and potential of LLM technology in the context of supporting higher education institutions in digital development processes.

As part of the TBI activity, a prototype was developed and implemented as a web application in the Streamlit environment, which enabled an intuitive user interface available through the browser. The system uses prompt engineering techniques and includes entire strategic documents in the context of the model to link quantitative survey data with the textual content of strategic documents. The result is an automatically generated report that clearly shows the state of digital maturity, identifies priority areas for development and offers personalised recommendations based on data and comparisons with the best practices of European universities. The report can be saved locally on the computer in PDF format for further processing and analysis.

Input data obtained from the self-evaluation system, 360 questionnaires conducted on four groups (management, teachers, IT professionals and students) are shown in Figure 1, while the following images show the analysis as a result of user interaction with LLM.

Results and Benefits

Testing and evaluation of the prototype “Digital Transformation Consultant” showed significant potential for the application of large language models in the field of strategic advice and analysis in higher education, bringing a number of concrete benefits:

• Faster and scalable data analysis: Automated synthesis of survey results and strategic documents allows you to generate structured reports within minutes, which significantly speeds up processes.
• Standardised and data-based results: The use of LLM increases the consistency of analysis and reduces subjectivity in the interpretation of data, thereby increasing the quality and comparability of reports among institutions.
• Increased efficiency of advisory services: Experts get an initial insight through automatically generated reports, which allows them to focus their time on deeper interpretation and individual counseling.
• Flexibility and upgradability: The system can be easily extended to other institutions, include additional documents or extend the analysis to new areas, ensuring the long-term usability and adaptability of the tool.

Lessons learned

During the TBI phase, key insights for successful implementation were identified:

• Contextual relevance of the data: The quality and precision of the output directly depends on the input data. The inclusion of relevant, up-to-date and precisely structured strategic documents is essential for the accuracy of the recommendations.
• Critical importance of prompt engineering and contextual enrichment: Including full strategic documents in the prompt allowed the model to provide a deeper, contextually rich analysis.
• Potential for comparative analysis and learning from the practices of others: The inclusion of European Universities documents has demonstrated how the recommendations can be further enriched with examples of good practice from the international environment.
• Necessary adjustment of the user interface: Although the prototype demonstrates technical feasibility, for the production solution it will be necessary to develop an intuitive user interface and integration with existing systems, so that the tool can be used without advanced technical pre-knowledge.

Conclusion

The ‘Test Before Invest’ project carried out with CARNET has successfully demonstrated how the use of artificial intelligence, specifically large language models, can significantly improve the processes of analysis and strategic consulting in the higher education system. The developed prototype, together with the acquired knowledge and identified recommendations, provides a solid basis for making informed decisions about the further digital transformation of higher education institutions. The project has confirmed the technical feasibility, cost-effectiveness and relevance of this technology in a real context, thus opening the space for the development of an advanced, scalable solution that can bring measurable benefits to higher education institutions and educational policy makers. EDIH Adria continues to support CARNET and other public institutions, SMEs as well as local self-government units (LGUs) in their digital transformation efforts, helping them to reap the benefits of advanced technologies for a more efficient, transparent and sustainable management of their resources.