Challenging the Machine: Which Translation Engine Has The Best Outputs for Asian Languages?

Locaria has worked tirelessly to upscale, improve and streamline its translation workflows to meet an unparalleled demand for multilingual content. For over a decade, our integrated MTPE (Machine Translation & Post-Editing) workflow has drawn from a pool of carefully selected NMT (Neural Machine Translation) engines that are continuously updated to meet exacting standards. Keeping at the forefront of modern technology enables us to carefully balance accuracy and style; laying the foundations for post-editors to work more efficiently while also ensuring language precision, consistency, and formality across all language groups.

One of the major criticisms for MT engines has been their “western-centric” approach, especially when handling Asian languages. The lack of understanding often leads to inaccuracies and inconsistent formality, syntax, grammar and unit measurements. And Locaria has been busy conducting analyses which reveal that local MT engines – such as Baidu and Tencent – have a better understanding of cultural scenarios, in comparison to their western counterparts. As such, these engines are often better at employing the appropriate level of formality, more adept at utilising natural sentence breaks and layouts, and are also more likely to apply local unit and numbering systems.

Here we summarise our findings on the effectiveness of different translation engines in both Chinese and Korean, across various types of content. The content groups (legal, manuals, surveys and creative) have been tested with different language pairs (English>Simplified Chinese and English>Korean) using popular engines DeepL and Google Translate, Baidu and Tencent from China, and Kakao and Naver from Korea.

Testing & Analysis

Legal Content:

When translating general legal content from English into Simplified Chinese, our study found that Baidu outperformed its counterparts, but was closely followed by DeepL’s NMT engine. Both Tencent and Google Translate lagged behind the competition in terms of quality output. 

The real differentiating factor was Baidu’s sentence structure. The engine produced sentences that were notably clearer and made the content much more understandable. Although to varying degrees, both of the Chinese engines – Baidu and Tencent – were able to utilise the Chinese comma to effectively break up long sentences into more digestible chunks. This is a task that DeepL and GT failed to do, resulting in sentence structures that were difficult to assimilate and closer to the original text in English. 

However, not one of the engines was able to produce a flawless text and the phrasing was unanimously stiff to some degree. 

Contrary to the English>Simplified Chinese tests, Google Translate outperformed the other engines when translating general legal content from English>Korean, and Naver also provided fairly good results. DeepL and Kakao were distant runners-up, and fell behind the competition due to an unnatural tone and the use of incorrect terminology. 

Instruction Manuals:

Conducting the same tests on various types of content, we also used the same six engines to translate instruction manuals from English>Simplified Chinese and English>Korean. In our assessments, DeepL emerged as the most accurate engine in both language outputs. 

There were, however, some noteworthy differences between the competing engines (detailed below).

Simplified Chinese Translations (Instruction Manual):

• Tencent, while not as accurate as DeepL, maintained a consistent and organised style in Chinese outputs

• Google Translate and Baidu were inconsistent in translating technical terms

Korean Translations (Instruction Manual):

• Google Translate managed to keep a natural tone but had major mistranslations

• Naver showed inconsistencies in technical terms

• Kakao contained the most errors

Surveys:

As part of our survey translation assessment, DeepL came out on top once again in both Chinese and Korean. 

Google Translate took the second place for Chinese, whereas Tencent and Baidu struggled to convey the accurate meaning of terminologies and phrases.

In Korean, Google and Naver had several major inaccuracies, and Kakao also had numerous issues including unnatural tone.

Creative Content:

For creative content, it was clear that Machine Translation should be approached with caution. DeepL had the edge over the other three for both language pairs, mainly due to its capacity to keep consistent in translating brand names. Despite this advantage, the translations were generally unidiomatic and awkward.

Baidu & Tencent

We found that when it comes to translating English>Simplified Chinese, Baidu and Tencent outperform Google and DeepL in understanding the logic of Chinese sentences. Baidu and Tencent break down long sentences into more digestible parts which are connected by Chinese commas. This visually enhances readability and comprehension, especially when conveying complex ideas. 

Additionally, Tencent was able to convert several (not all) long-scale numbers from the western ‘power of ten’ system into the Chinese ‘myriad scale’ (万进), which was a surprise finding. None of the other engines addressed the numerical issues, or attempted to make a conversion. 

What’s more is Baidu’s use of verb clauses and Tencent’s incorporation of conjunction words, which inherently contribute to a clearer understanding of the text and make these engines competitive choices for translating content into Chinese.

DeepL & Google

DeepL consistently delivers the best accuracy among all of the MT engines. It also maintains a more precise and contextually accurate translation, which makes it a reliable choice for content that requires high accuracy (eg. technical manuals). 

Both Google and Tencent follow suit and provide relatively accurate translations with occasional errors. Baidu, on the other hand, falls behind in terms of accuracy, occasionally producing mistranslations and leaving sentences unfinished, which makes it less suitable for critical content.

Consistency plays a crucial role in maintaining the integrity of translations and Tencent showcases remarkable consistency throughout the paragraph, using coherent terminology. DeepL and Baidu also exhibit a level of consistency, whereas Google struggles in this aspect. 

All MT engines in the test demonstrate some stiffness in their translations and can still be vastly improved, but Baidu for Chinese stands out as the most idiomatic in its rendering in terms of a more natural flow and word choice when compared to the others. DeepL and Tencent also fare relatively well in terms of idiomatic language, whereas Google Translate struggles in this aspect. 

Overall, for Chinese translations, Baidu was better for general legal content due to its ability to break sentences efficiently and provide clearer translations. For manuals and surveys, DeepL is the preferred choice due to its accuracy. However, for creative content, we advise approaching Machine Translation with caution and rely on human translators for idiomatic texts.

For Korean translations, DeepL emerged as the top performer in terms of overall quality. It produces well-toned translations, and successfully caters to native Korean readers. However, there are still some minor inaccuracies and non-translating issues which require careful post-editing.

Google can be considered for a general translation base, and demonstrated relative strength in the domain of general legal translations. While maintaining a consistent tone and manner, Google failed in delivering full accuracy. The main issues arose in translation completeness, repetitive language use and coherent formality. The struggle became evident when translating complex sentences or object descriptions with adjectives, which needed heavy post-editing for accuracy/consistency. 

Kakao & Naver

Naver, a popular engine in Korea, appeared to handle general legal and survey texts with bearable qualities. However, variations in tone, non-translated results, and spacing errors showed up quite recurrently. The areas where it specifically lacked included manuals and creative text. This was due to issues such as tone inconsistency, non-translated and omitted text, making the engine somewhat unreliable. 

Kakao, unfortunately, also lagged behind and was visibly the least reliable of the platforms. It consistently presented accuracy issues, unnatural sentence structures and awkward tonality, making it unreliable for serious usage. Kakao is also the only engine that disregarded the source paragraph divisions in the original text. 

In conclusion, selecting the right MT engine depends on the specific requirements of the content. By leveraging the strengths of each engine and understanding their limitations, we can effectively optimise the integrated MT workflow. This ultimately provides the best MTPE solution for our clients across many different fields. 

Contact the Locaria team to find out  how we can help you achieve effective localisation and translation using MTPE.

Get in touch

Insights: Rene Hsu & Thorsten Brueckner
Post-edits: Daniel Purnell