Research Update | A New Literature Retrieval Benchmark that Integrates Semantic Search with Domain-Specific Knowledge Graphs Conducted by Professor He Bingsheng's Team at National University of Singapore-NUS Guangzhou Research Translation and Innovation Institute

The NUS Guangzhou Research Translation and Innovation Institute is committed to building a platform for interdisciplinary and cross-domain scientific research exchange, promoting the effective transformation and application of research achievements. We will regularly share cutting-edge research results and updates in this section. We welcome your engagement and discussions!

This research is led by Professor He Bingsheng, Professor and Vice-Dean (Research) at the School of Computing, National University of Singapore (NUS). Professor He also serves as a Ph.D. supervisor at the NUS Guangzhou Research Translation and Innovation Institute (NUS GRTII). The team member, Junyi Hou, is a recipient of the 2025 Spring NUS GRTII Doctoral Scholarship. The team is committed to advancing innovation and excellence in systems research.

1. Research Background

The exponential growth of academic publications has led to information overload. Existing academic search tools, such as Google Scholar, exhibit limitations when handling rapidly evolving research domains. Traditional keyword-based retrieval methods are particularly inadequate for modern research needs. This study aims to propose a new literature retrieval benchmark and introduce a novel method that integrates semantic search with domain-specific knowledge graphs to enhance retrieval accuracy and recall. By constructing a literature dataset based on the ICLR 2024 (a top-tier conference in artificial intelligence), we provide a systematic evaluation framework to assess the performance of academic literature search tools. Our work not only establishes new methods and benchmarks for academic search but also lays the groundwork for exploring the potential of LLM-driven retrieval systems in academic discovery.

2. Key Findings

Currently, we have constructed a high-quality keyword retrieval dataset using the abstracts and citation relationships of ICLR 2024 papers. Based on this dataset, we have completed an initial evaluation of the retrieval accuracy of Google Scholar and Semantic Scholar, focusing on metrics such as precision and recall for keyword-based searches.

Although the foundational benchmarking has been completed, we are designing additional experiments to further refine the evaluation metrics. Specifically, we are focusing on areas such as semantic search, query context understanding, and modeling relationships between papers. These improvements aim to provide a more comprehensive and accurate evaluation framework. The new experiments will help us gain deeper insights into the performance of different literature retrieval tools in complex search scenarios, contributing to the advancement of literature retrieval technologies.

3. Impact and Applications

For Ph.D. students, improved academic search tools simplify literature searches, helping them quickly gather the information needed to write stronger manuscripts and increase their chances of paper acceptance. For experienced researchers, our context-sensitive retrieval systems offer accurate access to the most relevant literature, even in unfamiliar areas, making it easier to assess the novelty and significance of the latest research.

Future collaboration opportunities include technology transfer and joint research. For example, academic database platforms can collaborate with our team to integrate this retrieval method into existing literature search engines, enhancing both search efficiency and accuracy. In addition, industry pilot projects will advance research in automated academic search, providing more accurate literature data sources for Large Language Models, further improving their performance. We look forward to collaborating with industry partners to drive the application of this technology, fostering interdisciplinary innovation and development.

4. Team Leader Profiles

图片1(3).png

Professor He Bingsheng

A Professor and Vice-Dean (Research) at the School of Computing, National University of Singapore. His research focuses on cloud computing, database systems, and high-performance computing. He has received numerous industry awards and has been an ACM distinguished member and fellow of IEEE.

图片1(4).png

Junyi Hou

Junyi Hou, awarded the 2025 Spring NUS GRTII Scholarship, focused on advancing infrastructure and systems for large language models.

The NUS Guangzhou Research Translation and Innovation Institute primarily focuses on seven key areas: smart city, information and communication, electronic science and technology, advanced manufacturing, artificial intelligence, biological sciences, and financial technology. Relying on NUS’ world-class research capabilities, the Institute has engaged dozens of professors from NUS as academic leaders. These leaders guide local research teams in promoting scientific research, research translation, and postdoctoral training programs.

We look forward to working with global research experts, entrepreneurs, and policymakers to stimulate innovative thinking, explore the infinite possibilities of scientific research, and establish an interdisciplinary and cross-field ecosystem for scientific research and innovative applications.