📊AI Search Engine Multilingual Evaluation Report (v1.2)

In previous evaluation, we assessed multilingual and cross-lingual search capabilities of Perplexity Basic, Perplexity Pro, Metaso, iAsk, and you.com. Among these, Perplexity Pro consistently performed the best. This time we added a new ai search player, Felo AI, including both his Basic and Enhanced edition. Although Felo Enhanced requires a browser plugin installed, it remains a free version.

Evaluation Results

After rigorous evaluation, we have reached the following conclusions:

Felo Enhanced leads Perplexity Pro by a slight margin, achieving an accuracy of 86.67% compared to Perplexity Pro's 84.17%.
Felo Basic also surpasses Perplexity Basic with an accuracy of 81.67%, compared to 72.50%.
Metaso performs well in searches within the mainland China Simplified Chinese community but shows average performance in cross-lingual searches, with an accuracy of 66.67%.
iAsk and you.com demonstrate poor performance in searches outside of English-speaking communities, with accuracies of 52.50% and 33.33%, respectively.

Product Name

Average ACC

complex_search

business_consulting

local_search

products_search

real_time_news

technical_consulting

Evaluation Data

To comprehensively assess the performance of the above AI search engines in a multilingual environment, we conducted tests in six languages: Japanese, English, Traditional Chinese, Simplified Chinese, Russian, and Korean. We have open-sourced all test datasets and results, which can be found on GitHub at the following URL:

Dataset：https://github.com/sparticleinc/ASEED

Evaluation Methodology

In this round, we migrated our testing platform from Ragas to Promptfoo. Promptfoo offers multiple answer detection rules and achieves higher overall accuracy in evaluations compared to Ragas. Therefore, we will only update test cases in the Promptfoo format going forward. For this evaluation, we utilized the Promptfoo platform and conducted tests using the GPT-4o large language model. All test results underwent manual verification to ensure accuracy.

PreviousAI Search Engine 多语言评测报告(v1.2)

Last updated 3 months ago

Evaluation Results

After rigorous evaluation, we have reached the following conclusions:

Felo Enhanced leads Perplexity Pro by a slight margin, achieving an accuracy of 86.67% compared to Perplexity Pro's 84.17%.

Felo Basic also surpasses Perplexity Basic with an accuracy of 81.67%, compared to 72.50%.

Metaso performs well in searches within the mainland China Simplified Chinese community but shows average performance in cross-lingual searches, with an accuracy of 66.67%.

iAsk and you.com demonstrate poor performance in searches outside of English-speaking communities, with accuracies of 52.50% and 33.33%, respectively.

Product Name

Average ACC

complex_search

business_consulting

local_search

products_search

real_time_news

technical_consulting

Evaluation Data

Dataset：https://github.com/sparticleinc/ASEED

Evaluation Methodology