Home / Vol. 38 No. E1 (2026): SPECIAL VOLUME (OCT,2026) - The ICTs and their impact on society. Technical Track TICEC 2026

Articles

An academic chatbot with hybrid retrieval and local large language models

Authors

Abstract

Higher-education institutions devote a substantial share of their staff time to answering queries whose answers are already set out in institutional regulations. This paper presents the design and evaluation of an academic chatbot based on retrieval-augmented generation that lets users query those documents in natural language and receive grounded answers in Spanish. The system combines a structure-aware ingestion pipeline, a hybrid retriever that fuses dense semantic search and lexical search through reciprocal rank fusion, and a locally hosted large language model. A distinguishing goal is data sovereignty: every component runs on institutional hardware, so no data is sent to external services. Quality is evaluated with the RAGAS framework over five regulatory documents: the corpus-wide set attains a global score of 0.90, with faithfulness of 0.97. The acceptance criterion is not met, however, under its conjunctive reading: answer relevancy stays below its threshold across all sets, and p95 latency reaches 6.47 seconds against a 5-second target. Quality further degrades on long, heavily cross-referenced regulations. A contrast questionnaire administered to nineteen users places the perception index at 3.64 out of 5 and exposes a relevant discrepancy: only 57.9% agree that the system does not produce fabricated information, against the faithfulness the automatic judge assigns it.

Keywords

retrieval-augmented generation higher education artificial intelligence natural language processing information privacy automatic evaluation

How to Cite

Guadalupe Andrade, J. P., Topon Visarrea, B., & Iza, C. (2026). An academic chatbot with hybrid retrieval and local large language models. Revista Tecnológica - ESPOL, 38(E1), 50-70. https://doi.org/10.37815/rte.v38nE1.1595
BibTeX

Statistics

0 Views
0 Downloads

References

Antico, L., Giordano, C., Koyuturk, D. B., & Ognibene, D. (2024). Unimib Assistant: Designing a student-friendly RAG-based chatbot for all their needs [Preprint], arXiv. https://doi.org/10.48550/arXiv.2411.19554

Asamblea Nacional del Ecuador. (2021). Ley Orgánica de Protección de Datos Personales [Organic Law on Personal Data Protection]. Registro Oficial, Suplemento 459.

Brooke, J. (1996). SUS: A "quick and dirty" usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester & I. L. McClelland (Eds.), Usability Evaluation in Industry (pp. 189–194). Taylor & Francis.

Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., & Liu, Z. (2024). M3-Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 2318–2335). Association for Computational Linguistics.

Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 758–759). ACM. https://doi.org/10.1145/1571941.1572114

Coronel Crespo, B. P., & Ulloa Bernal, C. F. (2025). Desarrollo de un chatbot multiagente basado en modelos LLM y arquitectura RAG para la interacción en lenguaje natural con bases de datos organizacionales [Undergraduate thesis, Universidad de Cuenca]. Repositorio Institucional de la Universidad de Cuenca.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16 (3), 297–334. https://doi.org/10.1007/BF02310555

Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 150–158). Association for Computational Linguistics.

Espinosa-Luna, G., Castillo-Oliva, R., Montañez-Díaz, M., & Mendoza-De-los-Santos, J. (2023). Implementación de un chatbot basado en modelo de lenguaje de inteligencia artificial para responder preguntas frecuentes de estudiantes universitarios. Revista Científica de Sistemas e Informática, 3 (2), e570. https://doi.org/10.51252/rcsi.v3i2.570

Guerra, M., Monti, C., & Fernández, A. (2025). Sistema de generación aumentada por recuperación (RAG) para el acceso a información académica. In Actas del XXXI Congreso Argentino de Ciencias de la Computación (CACIC 2025) (pp. 1165–1169). Red UNCI.

Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28 (1), 75–105. https://doi.org/10.2307/25148625

Khan, A. U. R., Hasan, M., Kemell, K.-K., Rasku, J., & Abrahamsson, P. (2024). Developing retrieval-augmented generation (RAG) based LLM systems from PDFs: An experience report [Preprint], arXiv. https://doi.org/10.48550/arXiv.2410.15944

Kuratomi, G., Pirozelli, P., Cozman, F. G., & Peres, S. M. (2025). A RAG-based institutional assistant [Preprint], arXiv. https://doi.org/10.48550/arXiv.2501.13880

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474.

Livathinos, N., Auer, C., Lysak, M., Nassar, A., Dolfi, M., & Vagenas, P. (2025). Docling: An efficient open-source toolkit for AI-driven document conversion. In Proceedings of the AAAI Conference on Artificial Intelligence (Demonstration Track). AAAI Press.

Malkov, Y. A., & Yashunin, D. A. (2020). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42 (4), 824–836. https://doi.org/10.1109/TPAMI.2018.2889473

Manjarrés-Betancur, G., & Echeverri-Torres, J. (2020). Asistente virtual académico utilizando tecnologías cognitivas de procesamiento de lenguaje natural. Revista Politécnica, 16 (31), 85–96. https://doi.org/10.33571/rpolitec.v16n31a7

Neupane, S., Bhatt, M., Aryal, S., Lafleur, E., Poudel, S., Fathi, H., Calyam, P., & Gupta, R. (2024). From questions to insightful answers: Building an informed chatbot for university resources [Preprint], arXiv. https://doi.org/10.48550/arXiv.2405.08120

Nguyen, M.-T., & Quan, T.-T. (2025). URAG: Implementing a unified hybrid RAG for precise answers in university admission chatbots — A case study at HCMUT. In Proceedings of the 13th International Symposium on Information and Communication Technology (SOICT 2024) (CCIS 2351, pp. 95–109). Springer.

Odede, I., & Frommholz, I. (2024). JayBot — Aiding university students and admission with an LLM-based chatbot. In Proceedings of the 2024 Conference on Human Information Interaction and Retrieval (CHIIR '24) (pp. 391–395). ACM. https://doi.org/10.1145/3627508.3638293

Qwen Team. (2025). Qwen2.5 technical report (arXiv:2412.15115). arXiv. https://doi.org/10.48550/arXiv.2412.15115

Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3 (4), 333–389. https://doi.org/10.1561/1500000019

Swacha, J., & Gracel, J. (2025). Retrieval-augmented generation (RAG) chatbots for education: A survey of applications. Applied Sciences, 15 (8), 4234. https://doi.org/10.3390/app15084234

Xu, Z., Cruz, M. J. N., Guevara, M., Wang, T., Deng, M., Liu, L., & Pan, S. (2024). Retrieval-augmented generation with knowledge graphs for customer service question answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '24) (pp. 2905–2909). ACM. https://doi.org/10.1145/3626772.3661370