How to Perform Bibliometric Analysis Using Academic Search Engine Data

Bibliometric analysis—the quantitative study of publication patterns, citation networks, and research output—has become a core methodology for funding decisions, tenure evaluation, and institutional benchmarking. A 2023 report from the National Natural Science Foundation of China found that over 68% of Chinese university research evaluation committees now require bibliometric evidence in grant applications, up from 41% in 2018. Simultaneously, the OECD’s 2024 Science, Technology and Innovation Outlook noted that global academic output surpassed 3.2 million indexed articles in 2023, making manual literature review impractical. The bottleneck is no longer data availability, but the ability to extract and structure that data from academic search engines like Google Scholar, Web of Science, and CNKI. This article provides a practical, four-dimension framework—coverage, search syntax, export formats, and API support—for performing robust bibliometric analysis using these platforms.

Coverage Depth and Discipline Bias

The foundation of any bibliometric study is coverage completeness. Google Scholar indexes an estimated 389 million documents as of 2024, according to a study in Scientometrics, but its coverage is heavily skewed toward English-language journals and open-access repositories. In contrast, CNKI (China National Knowledge Infrastructure) covers over 95% of Chinese-language academic journals published since 1994, as stated by CNKI’s own 2023 platform documentation, making it indispensable for domestic Chinese research evaluation.

Discipline-Specific Gaps

For engineering and natural sciences, Scopus (Elsevier, 2024) claims 93% coverage of peer-reviewed journals in these fields, while Google Scholar excels in capturing conference proceedings and preprints. For social sciences and humanities, Web of Science’s Arts & Humanities Citation Index covers only about 1,800 journals, compared to Google Scholar’s broader but less curated index. Researchers must match the search engine to their discipline: use CNKI for Chinese medical literature, Scopus for global engineering output, and Google Scholar for interdisciplinary or emerging topics.

Search Syntax for Precision Retrieval

Accurate bibliometric analysis depends on constructing reproducible search queries. Each platform employs a distinct syntax that affects recall and precision.

Boolean and Field Operators

Google Scholar supports basic Boolean operators (AND, OR, NOT) but lacks field-specific tags like TI= for title. For example: "machine learning" AND "China" AND 2023 This retrieves any document containing both phrases anywhere in the text, leading to high recall but low precision. In contrast, Web of Science uses field tags: TI=("deep learning") AND PY=(2020-2024) This limits results to title matches only, yielding a more manageable dataset for citation analysis. CNKI uses a similar field syntax: SU='人工智能' AND YE='2023' where SU denotes subject. Researchers should document the exact query string for reproducibility.

Phrase and Proximity Searching

For concept mapping, proximity operators are critical. Web of Science’s NEAR/n finds terms within n words of each other, e.g., "climate" NEAR/5 "policy". Google Scholar and CNKI do not support proximity operators, forcing analysts to rely on broader Boolean combinations. This limitation can inflate false positives in co-word analysis.

Export Formats and Data Structuring

Bibliometric software like VOSviewer, CiteSpace, or Bibliometrix (R package) requires structured input files. The export format determines whether your data is usable.

RIS and BibTeX Support

Web of Science and Scopus export RIS (Research Information Systems) files with standardized fields for author, title, journal, DOI, and cited references. Google Scholar offers only a plain-text citation export (BibTeX or EndNote) per individual record—no batch export. For a study of 500 articles, manually exporting each record is infeasible. CNKI provides batch export to RefWorks and NoteExpress formats, but the exported data often lacks cited references, limiting citation network analysis.

CSV and Excel Limitations

Many researchers default to CSV exports, but these often truncate abstract fields or omit cited references. A 2022 test by the Chinese Academy of Sciences found that 34% of CSV exports from CNKI had incomplete author lists for papers with more than 10 co-authors. For reliable bibliometric analysis, always export in RIS or BibTeX format, and verify a random sample of 20 records for field completeness.

API Access and Automation

For large-scale or longitudinal bibliometric studies, manual data collection is unsustainable. API support enables automated querying and data harvesting.

Scopus and Web of Science APIs

Scopus offers a RESTful API with a rate limit of 20,000 requests per week for institutional subscribers. It returns XML or JSON with full citation metadata, including author affiliations and cited-reference counts. Web of Science’s WoS Starter API provides 500 queries per day for free tier users, but the full Expanded API requires a paid subscription. These APIs allow researchers to programmatically retrieve data for thousands of papers, enabling temporal trend analysis.

Google Scholar’s Restricted Access

Google Scholar does not provide an official API. Unofficial scraping is against its Terms of Service, and Google actively blocks automated requests via CAPTCHA and IP throttling. A 2024 study by University of Amsterdam researchers found that 72% of automated Google Scholar queries were blocked within 10 requests. For bibliometric analysis requiring Google Scholar data, researchers must rely on third-party tools like Publish or Perish, which uses a local database snapshot rather than live API calls.

CNKI’s Closed Ecosystem

CNKI offers no public API for external developers. Data extraction must be performed through its web interface or via institutional subscription tools. This closed system makes automated longitudinal analysis of Chinese literature extremely difficult, often requiring manual data collection over multiple sessions.

Citation Normalization and De-duplication

Raw export data from different search engines contains duplicate records and inconsistent citation formats. A single paper indexed in both Google Scholar and CNKI may appear with different author name orders or journal abbreviations.

De-duplication Strategies

Use DOI (Digital Object Identifier) as the primary deduplication key. In a test of 1,000 papers on “renewable energy” from both Web of Science and Google Scholar, the University of Tsukuba Library (2023) found that 18% of records lacked a DOI, requiring manual title matching. Tools like Zotero can automate this process, but researchers should verify at least 50 random records.

Citation Count Discrepancies

Citation counts vary wildly across platforms. Google Scholar includes citations from books, theses, and non-indexed sources, often inflating counts by 30-50% compared to Web of Science. For example, a 2022 paper on CRISPR in Nature had 4,200 citations on Google Scholar but only 3,100 on Scopus. Bibliometric analyses must state which source was used for citation data and justify the choice.

Visualizing Results with Open-Source Tools

Once data is cleaned and structured, visualization transforms raw numbers into actionable insights.

Co-citation and Co-word Networks

VOSviewer (free, Leiden University) accepts RIS or CSV files and generates network maps of co-citation clusters. For a study of 2,000 papers on “blockchain in healthcare,” VOSviewer can identify four major research fronts: data security, supply chain, clinical trials, and regulatory frameworks. CiteSpace (free, Drexel University) adds temporal slicing to show how research fronts evolve over time.

Temporal Trend Analysis

Bibliometrix (R package, free) provides built-in functions for annual publication counts, average citation growth, and Lotka’s law fitting. A typical workflow: export data from Web of Science as RIS → import into Bibliometrix → generate a biblioAnalysis() object → plot annual growth curves. This process takes under 30 minutes for datasets up to 5,000 records.

FAQ

Q1: Which academic search engine has the best coverage for Chinese-language papers?

CNKI covers over 95% of Chinese academic journals published since 1994, according to its 2023 platform documentation. For comparison, Google Scholar indexes only about 40% of these journals. For any bibliometric study focused on Chinese domestic research, CNKI is mandatory.

Q2: How do I export citation data in bulk from Google Scholar?

Google Scholar does not support batch export. You can use third-party software like Publish or Perish, which retrieves up to 1,000 records per query from a local database snapshot (updated quarterly). The export format is RIS or CSV, but note that citation counts may be 30-50% higher than Web of Science.

Q3: What is the most reproducible search syntax for bibliometric analysis?

Web of Science offers the most reproducible syntax with field tags like TI=, PY=, and SO=. For example: TI=("carbon capture") AND PY=(2019-2024). Document the exact query string and the platform version in your methodology section. Avoid Google Scholar for reproducibility due to its lack of field tags and changing algorithms.

参考资料

• National Natural Science Foundation of China. 2023. Report on Evaluation Criteria for Grant Applications.
• OECD. 2024. Science, Technology and Innovation Outlook 2024.
• Chinese Academy of Sciences. 2022. Data Quality Assessment of CNKI Exports.
• University of Amsterdam. 2024. Google Scholar Scraping Block Rates Study.
• University of Tsukuba Library. 2023. DOI Coverage in Multisource Bibliometric Datasets.