Assesing the Accuracy of Translation Result of Kataku Version 1 . 1 and Transtool 10 from English to Indonesian and Its Implication on Language Teaching

In the context of translating from English into Indonesian, machine translations such as Kataku and Transtool have been mostly employed as pedagogical tools in learning and teaching reading of English as a Foreign Language. Whether teachers and students rely heavily or just refer to the machine translations as a helping tool in understanding the content of the text through reading, the question on the accuracy of the translation result produced by those machine translations is inevitable. Therefore, the purpose of this research is to assess the accuracy of the translation result produced by Transtool 10 and Kataku version 1.1, as the latest version of the time this study was designed, also as the most used machine translations and then draw its pedagogic implication on teaching and learning English as a Foreign Language, especially on reading skills. This study employed qualitative methods that are descriptive in nature. The qualitative paradigm applied is embedded case study with document analysis, Focus Group Discussion, and questionnaires as data collection techniques. The data are clauses and phrases from three different English textbooks: “Islamic Life and Thought”, “Biology Second Edition”, and “Nelson Book of Pediatrics”. The data analysis techniques used are domain analysis, taxonomy analysis, componential analysis and finding cultural values. The result of the study shows that Transtool


A.
For years, translation has been unpopular in the language teaching community (Machida, 2011;Dagilienė, 2012;Marqués-Aguado & Solís-Becerra, 2013).The reasons for this issue are numerous.First, translation as a method of teaching and learning can interfere with the natural acquisition of L2, in this case particularly English, because it involves L1.It is believed to deprive learners from adequate input of L2.In addition to that, it could cause some L2 learning errors because of grammatical confusion with the mother tounge (Pan & Pan, 2012).Next, translation is considered to be a boring and time-consuming activity.
Nevertheless, the belief on translation as a method of teaching and learning EFL has experieced some shift recently.Translation now has far gone from a method of learning into a tool of learning foreign languages.It has become a pedagogical tool in teaching and learning EFL (Pan & Pan, 2012;Dagilienė, 2012).In addition to that, translation along with reading, grammar exercises, and other activities are considered beneficial to language teaching and learning (Dagilienė, 2012).Even further, it is perceived as a fifth skill in language learning (Campbell, 2002), after listening, speaking, reading, and writing, and is the most crucial social skill because it fosters communication and understanding.
As a pedagogical tool, translation has been used to aid comprehension in all language skills.However, the skill that benefits a lot from translation is reading.Translation can help students better understand texts.Translation can become a way out when students find a difficult time understanding a text.By translating the text into their mother tounge language, they can understand the text better.In doing the translation, they often refer to dictionaries as a helping tool.In the newest fashion, machine translations are very much popular for its simplicity and instant.
Introduction of technology in language learning and teaching has brought a lot of changes and insights (Merzifonluoğlu & Gonulal, 2018).This notion is materialized in the case of machine translations.As previously described, they are preferred for its instant work in producing a translation.The instances of machine translations that are popular are Kataku and Transtool.Both have been utilized by many for many purposes of translation, especially for the purpose of teaching and learning EFL.
Technically, Kataku and Transtool work just like Google Translate.They all share the same feature of works.They use a statistical translation model.They match word or words with a bilingual text corpus with equal word class.For instance, a noun will be translated as a noun.In this regard, the translation tends to be isolated (Giannetti, 2016).Therefore, literal translation is possible to happen.
Studies on how translation contributes to language education have been conducted (Campbell, 2002;Pan & Pan, 2012;Marqués-Aguado & Solís-Becerra, 2013;Belenkova & Davtyan, 2016).The result brings forward a concept that translation can play a major role on being a pedagogical tool in language teaching and learning.More specifically, the role of machine translations as a pedagogical tool has been studied (Giannetti, 2016).However, none of the studies has touched the acuracy of the translation result produced by the machine translation.While, it should become an essential focus because in order for us to rely on their service, we need to know how trustworthy they are in doing translation.
Actually, a research on the contribution of machine translation on language education as well as the quality of the output has been done (Briggs, 2018).However, the translation quality that is assessed in the study is only acceptability.Acceptability deals with the form or style of the translation.It is about whether or not the translation is suitable with the culture and norm of target language (Nababan, 2012).On the other hand, according to Nida & Taber (1982), translation is a transfer of equivalence of source language into target language, first in terms of the message, and then in terms of the style.It means that the first most important aspect in translation that needs considering is the message.The style or the form, or the so-called acceptability, falls into second most important category.Therefore, a research that studies the accuracy of the message of the machine translations as well as to draw its pedagocial implication on teaching and learning EFL has yet to be conducted.
Based on such a need, this study focuses on assessing the accuracy of the translation result of Kataku 1.1 and Transtool 10 and drawing its pedagogical implication on teaching and learning reading of English as a foreign language.Kataku 1.1 and Transtool 10 are chosen as instances of machine translations in this regard because, as of the moment, they are the most used and the easiest accessed machine translation that are available.Also, as explained previously, they are mostly utilized as a pedagogical tool in teaching and learning reading of EFL.They are referred to in trying to find the meaning of unfamiliar phrases or clauses in English texts.Hence, this study focuses on the translation from English into Indonesian.

Literature Review B.
Translation and The Assessment 1.
Of translation, House (2018) describes that translation is a replacement procedure of text from source language into target language.In this sense, it seems, as though, the focus is mainly about the text or the form.Furthermore, Newmark (1988) proposes that translation is a transfer of meaning of a text from a language into another in the way the original author of the text intended the text.On the contrary, this definition sounds too much focused on the meaning or the message without bringing about the fact that translation includes a linguistic form movement from one language into another.While, in fact, translation includes both transfer of message and transfer of linguistic form.Nida puts it in detail that translation is a reproduction of closest natural equivalent of the source language in the target language, first in terms of the message, and second in terms of the style or the form (Nida & Taber, 1982).It is very clearly stated that what matters the most in translation is the message and then the form comes after that.
Based on the nature of translation as explained above, Nababan (2012) proposed a model of translation quality assessment.The model consists of three aspects of quality that are measured.They are accuracy, acceptability, and readability.Accuracy deals with how accurate the message from source language is transferred into target language.Next, acceptability deals with how suitable the translation is with the norm and culture of target language.These two aspects are derived from the nature of translation explained by Nida that translation is a transfer of message and also style.Finally, the third aspect that should be considered in translation is readability.It deals with how easy the text is understood by target readers.This concept comes forward because at the end of the day, a translation is made for target readers.Translation may fulfill a good accuracy of message and a good style suitable with the culture of target language, but if it is difficult to understand, it will be troublesome for target readers.Therefore, this concept is incorporated into the translation quality assessment proposed by Nababan.Therefore, this is by far the most holistic approach in translation quality assessment since it takes many aspects into account.
Other assessment such as one proposed by House (2015) focuses too much on the linguistic aspect.The model is very much theoretical rather than practical.It is rather difficult to apply since it tends to be conceptual.

Machine Translation 2.
Machine translation is a machine that is expected to be capable of translating automatically, fast, and efficiently from source language into target language without the help of human beings in the working process (Hutchins & Homers, 1991).The machine is designed and intended to save time and money.Machine translation analyzes structure of sentences and then breaks them down into smaller linguistic constituents.After that, the smaller linguistic constituents are translated and restructured back into sentences in their target language.That being said, the translation is rather isolated in smaller units without considering the existence of particular contexts within the sentence, most especially context of culture and situation.This becomes a major setback for machine translation.The translation tends to be literal than contextual.
In accordance with that, machine translation cannot be said to be equal to human translation for some reasons.First, machine translation cannot think like human brain.No matter how advanced technology now is, machine translation has yet to catch up with the complexity of human's brain.In the process of analyzing and understanding source text, human brain operates on several levels: 1) semantic level, that is understanding a word in isolation, or out of context; 2) syntactic level, that is understanding a word as a constituent in a sentence; 3) pragmatic level, that is knowing words that need to be translated in relation with the context of culture and situation (Craciunescu et al, 2004).Machine translation can only reach as far as to the syntactic understanding, leaving understanding of context behind.
Moreover, in doing translation, there are some knowledge aspects that need to be fulfilled.They are knowledge of source text, knowledge of target text, knowledge of equivalence between source and target text, knowledge of subject field or general knowledge, and finally knowledge of socio-cultural aspects.Among those, machine translation cannot possibly match human brain in terms of knowledge of subject field and knowlegde of socio-cultural aspects.Therefore, given the complexity of translation process and also human brain, machine translation cannot be claimed to pair or to thoroughly substitute human translation in producing a quality translation (Craciunescu et al, 2004).
Next, there are some types of machine translation: one that is assisted by humans and another that is fully automatic.The machine translation that is referred to in this study is the one of which translating work needs no human help at all, except for the inputting process of the source text.Instances of such a machine translation that is popular among people, especially students and teachers are Kataku and Transtool.

Machine Translation's Pedagogic Implication 3.
As already explained in the introduction, the progress of language education has seen machine translation as a pedagogical tool (Pan & Pan, 2012; Dagilienė, 2012); Giannetti, 2016;Briggs, 2018).Machine translation has been often utilized in EFL classes.It is often used as a resource of writing (Giannetti, 2016) and also reading.Lately, machine translation has been used as a reference to translate reading material in order to help students better understand the text.Using machine translation is proved to be a lot faster than when students consult dictionaries.They ahve to look up words one by one and then relate to each other to find out the possible meaning of unfamiliar phrases in a sentence they find in a text.Using machine translation, the process can be a lot simpler.Students only need to type or input the source text into the platform and the rest is instant.Machine translation is believed to make reading process faster and easier.

Research Methodology C.
This research is a qualitative study with embedded case study approach.In addition to that, this research is a product oriented translation study.The sources of data are documents, informants and respondents.They were chosen by criterion based sampling.Documents are in the form of three different English textbooks namely "Islamic Life and Thought", "Biology Second Edition", and "Nelson Book of Pediatrics".They were chosen for their different genres.The informants are raters who assessed the accuracy of the translation result of Kataku 1.1 and Transtool 10.The raters were chosen based on some criteria such as: 1) they must be translation scholars, meaning at least they have master degree in translation studies 2) they have experiences with using translation quality assessment proposed by Nababan (2012).Respondents are general people who filled in the questionnaires.The criterion for respondents is that they must be students, lecturers, or teachers.
The data in this study are divided into linguistic data and translation data.Other than that, the data are also the opinion on what machine translation are used by the most people.The linguistic data are in the form of clauses and phrases in the first chapter of three different textbooks.They were sampled using purposive sampling technique.This research intended to assess the accuracy of the translation result of Kataku 1.1 and Transtool 10.For that matter, first chapter analysis on accuracy may represent the total quality of the whole book (Nababan, 2012).The translation data are the score of accuracy from raters.
The data were collected using document analysis, Focus Group Discussion, and questionnaires.Document analysis was employed to gather linguistic data in the form of clauses and phrases from the first chapter of three different English textbooks.Focus Group Discussion was used to gather translation data in the form of accuracy assessment of the translation result of Kataku 1.1 and Transtool 10.In the FGD, raters gathered together with the researcher and assessed the accuracy together.Questionnaires were used to determine what machine translation that people mostly use.
Of a total 75 questionnaires were distributed and 56 respondents completed the forms and returned them to the researcher.Of these, 19 were male respondents and 37 were female respondents.The age of respondents varied; 46 respondents aged between 20-30 years and 7 respondents aged 31-40 and 3 respondents aged over 40 years.Their works also varied, such as students (46), lecturers (9) and teachers (1).The data showed implicitly that the main targets of the questions in the questionnaire were the students and professors, who regarded as the main users of Kataku and Transtool.
When asked if they had difficulties in translating English text into Indonesian or vice versa, their majority answers or 50 of them had trouble and 6 respondents had ever experienced having the trouble.According to them, the best way to resolve the issue was to (1) ask help to a friend who were not as a translator (2 respondents), (2) ask assistance to a translator (5 respondents), and (3) using machine translation (49 respondents).From these responses it was clear that all the respondents had difficulty in translating English text into Indonesian and most of them utilized machine translation to overcome the difficulties.
All these surveyed respondents were already familiar with Transtool.In contrast, there were 12 respondents who did not know Kataku at all.Even, 36 respondents had already installed Transtool on their computers.The amount of respondents who installed Transtool were more the Kataku, that is, as many as 14 respondents.That makes sense because the Transtool program was launched earlier in the market than Kataku program.Generally, the respondents got Information about both computer-assisted translation from mouth to mouth and advertisement on the Internet.
Compared with Kataku, Transtool Program was used more often by respondents in translating English text into Indonesian.The main reason they used both computer programs of translating was to obtain translations quickly and at very low cost and even free of charge.Their statements were apparently in line with the views of most respondents that Kataku and Transtool could translate well enough and the performance of both the translators of computer program was in accordance with their expectations.The instrument used in this study is instrument of translation quality assessment that is proposed by Nababan (2012)  The meaning of words, technical terms, phrases, clauses or sentences accurately transferred from source language into the target language; absolutely no distortion of meaning

Less accurate 2
Most of the meaning of words, technical terms, phrases, clauses or sentences of the source language has been transferred accurately into the target language.However, there is still a distortion of meaning or having double meaning (ambiguous) or there are still some eliminated meanings, which disturb to the integrity of the message.

Inaccurate 1
Meaning of words, technical terms, phrases, clauses or sentences of source language inaccurately transferred into the target language or eliminated The data analysis were conducted by applying the concept of ethnography analysis, that is using analysis of domain, taxonomy, component, and finally finding cultural values that lie beyond the relation of all aspects that are studied (Spradley, 1980) In the first stage, domain analysis was conducted.In domain analysis, the data were gathered and grouped into their respective domain.For instance, the translation of phrases and clauses from English into Indonesian were grouped into ones that are the result of Kataku 1.1 and ones that are the result of Transtool 10.After that, in taxonomy analysis, the translated phrases and clauses from two different machine translations are categorized into three different categories of accuracy, namely accurate, less accurate, and inaccurate.Finally, all aspects studied are correlated together.This section is called as componential analysis.In this stage, the translated phrase and clauses from Kataku 1.1 and Transtool 10 are linked to the category of accuracy.In the end, the researcher unfolded the reason behind such a relation.

Findings D.
There are found 31 clauses in the first chapter of "Islamic Life and Thought" (ILT), 39 clauses in "Biology Second Edition" (BSE), and 38 clauses in "Nelson Textbook for Pediatrics" (NTOP).In terms of phrases, there are found 88 phrases in the first chapter of ILT, 81 phrases in BSE, and 80 phrases in NTOP.After being analyzed in their domain and taxonomy, they are put into a componential relation presented in the table below.The table above shows the number of data in the form of translated clauses and phrases from English into Indonesian that are categorized as accurate, less accurate, and inaccurate from three different textbooks in two different machine translations.At the right-most section is presented the average of the accuracy score for each textbook.The average score of all textbooks, both translated in Kataku 1.1 and in Transtool 10, shows almost similar figure.All of the average score points at about figure 2. According to the instrument, the figure represents medium accuracy.It means that some parts of message are already delivered well in the translation, but there are some others that are not conveyed well in the translation.They might have experienced loss or deletion.If we look closer into the comparison of both machine translations and comparison of clause and phrase, it can be seen that the average score produced by Transtool 10 is slightly better than Kataku 1.1 and the average score of translated phrases are also slightly better than translated clauses.To be more delicate, a more conclusive table showing such a result is presented below.The table above clearly shows the comparison of average score of accuracy of clauses and phrases in two different machine translations.Furthermore, the phrase score of average accuracy is slightly better than the score of the average accuracy of the clause in both machine translations.The average score of accuracy of the phrase in Kataku 1.1

Source Text
Clearly we have to consider the physical and chemical properties of the elements and their compounds if we are to establish a meaningful classification.(142/TXT -12) Kataku 1.1 Secara jelas kami mesti mempertimbangkan milik fisik dan kimia elemen dan halaman tertutup mereka jika kami akan memperlihatkan klasifikasi berarti.
The instance above has a problematic equivalence in the translation.In the case of phrases, "the physical and chemical properties of the elements" is translated differently in Kataku 1.1 and Transtool 10.Kataku 1.1 translated the phrase into a literal equivalence "milik fisik and kimia elemen".It is obvious that Kataku 1.1 cannot operate on the level of contextual translation in this instance.This makes the phrase translation inaccurate.On the other hand, Transtool 10 successfully translated the phrase into a more relevant equivalence: "sifat kimia dan fisik".It is in conjunction with the context, which is physics science.Such an equivalence makes the translation rated as accurate.
Next, the phrase "compounds" is translated as "halaman tertutup" in Kataku 1.1 and as ""campuran" in Transtool 10.Again, Kataku 1.1 translated the phrase literally.It is completely out of context.While, Transtool 10 still retains the contextual meaning of the phrase in the translation.The translation of Kataku 1.1 is rated inaccurate and that of Transtool is rated accurate.
In the case of translating clause, Transtool translated better than Kataku 1.1.Kataku 1.1 did not produce a contextual equivalence for the verb "to establish".The translation is out of context and makes the intended meaning of the clause change in the translation.This contributes to the inaccuracy of the translation.On the contrary, Transtool 10 did very well on finding a contextual equivalence for the verb "to establish".It affects the whole clause to be understandable and makes sense in the translation.The translation is certainly accurate.

E.
Based on the relation that is displayed in the componential analysis, there is a link between the machine translations (Kataku 1.1 and Transtool 10), the unit of translation, and the accuracy score of the translation.The two machine translations display different behaviors in conjunction with quality of the translation produced.The unit of translation also produces different accuracy too.
First of all, the translation of phrases has a better accuracy than the translation of clauses by the two machine translations.This seemingly has something to do with the different syntactic complexity in clauses and in phrases.The machine translations seem to be unable to process a long structure of syntactic order very well and that results in a lot of errors in the translation.In case of shorter structure like phrases, they still perform well.This is in line with what Craciunescu et al (2004) argue: that machine translation cannot possibly match the complexity of human brain in doing the translation.Human brain starts the translation process by analyzing a sentence structure in a delicate way on which machine translations have limitations.Kataku 1.1 and Transtool 10 are proven to have problems in analyzing a long syntactic order in the form of clause.Therefore, they produce a better translation at phrase level, that is not too long or too complicated in syntactic order.
In addition to that, both machine translations, Kataku 1.1 and Transtool 10, generally produce a medium score of accuracy.It means that despite they successfully transfer some message into target language; there is a portion of message that they fail to convey in the target language.The failure is sometimes in the form of the inability of the machine translations to find equivalence of the words or phrases in the target language.This goes hand in hand with the result of the study of Koponen (2010).The result of the study shows that machine translation produces a rather bad score of accuracy compared to human translation.As described by Craciunescu, machine translations do not posses the knowledge of understanding context of culture and situation (Craciunescu et al, 2004).Therefore, oftentimes, such an error like literal translation as in the case of translating "quantities" in the context of physics into "jumlah" in Indonesian.Machine translations only process a word in isolation and find a corpus statistic of the equivalence and match it out of context.Therefore, a literal translation is expected to be found in many cases of translation using machine translation.
As of the case Transtool 10 produces a more accurate translation than Kataku 1.1, it is a matter of a more updated software.As explained in previous section, Transtool is a more familiar software and a more used application in the society.An improved update on such an application in terms of wider corpus collection is not surprising.Therefore, it is likely to produce a better translation in terms of the level of accuracy than Kataku 1.1 which does not have a wide market as much as Transtool does.
Considering the finding on the accuracy level of Kataku 1.1 and Transtool 10, the use of those machine translations as a pedagogical tool in EFL classes, especially for reading, should not be encouraged to be a sole reference.Students who seek help from these machines when they try to find out the meaning of a text needs to be careful.The use of machine translations as a pedagogical tool in teaching and learning reading should be accompanied by other pedagogical tools such as dictionaries and parallel texts.This is to prevent students from getting wrong translation that may lead to a wrong understanding of the content of a reading material, when machine translations are the sole reference there is.This is in accordance with the result of the study that Briggs (2018) conducted.The study intended to figure out the quality of the web translation, another type of machine translation, and its implication on Korean students in learning English as a foreign language.The study unfolded that the translation, from English into Korean, produced by the web translation is of a rather unfavorable quality.Although, the quality measure in the study is acceptability, but the result shows a similar pattern with this study.It seems that there is a high possibility that machine translations produce a translation of not only medium accuracy, but also medium acceptability and readability, even low score of all of them.However, this is not to discourage translation as a pedagogical tool in reading of EFL classes.On the other hand, this can promote human translation into more use, which can help boost translation skills of students.Apart from that, machine translation still can be used as a reference in the case of finding meaning of difficult or unfamiliar phrases found in the text.Students are suggested to input phrases in the translation using machine translation because they translate shorter syntactic orders better than clauses.Even, in this way, students can learn how to arrange a good order of meaningful clause in target language based on the available translated phrases produced by machine translations.This is to promote an active self-learning reading activity.Or rather, this could serve as a fifth skill in learning English as a foreign language (Campbell, 2002).Consequently, the purpose of translation as a pedagogic tool in learning EFL can be retained and even be developed.This is a justification for studies conducted by Pan & Pan (2012) and Dagilienė (2012).

Conclusion F.
Despite the medium accuracy of translation they produce, machine translations (Kataku 1.1 and Transtool 10) can still serve as a pedagogical tool in EFL classes, especially in teaching and learning reading.The use of these machines should be accompanied by other pedagogical tools to prevent students from gaining error translation that could lead to a wrong understanding of a reading material text.Translation of phrases by machine translations produces a better accuracy than translation of clauses.This is because machine translations translate word by word in isolation and they are proven to be unable to translate long syntactic structures correctly.Furthermore, Transtool 10 is proven to translate more accurately than Kataku 1.1, henceforth, it is more suggested to use as a pedagogical tool in reading class of EFL.