GIZA++ is an extension of the program GIZA (part of the SMT toolkit EGYPT) which was developed by the Statistical Machine Translation team during the summer workshop in 1999 at the Center for Language and Speech Processing at Johns-Hopkins University (CLSP/JHU). GIZA++ includes a lot of additional features. The extensions of GIZA++ were designed and written by Franz Josef Och.

In wordMint regard, GIZA++ is used to get character level alignments. GIZA++ is poorly documented but still README file helps. For wordMint purpose we needed character level alignment by GIZA++, which proved to be a difficult task. After exploring a work around, character level alignment of n-gram characters was done.

For this, a named entity word was assumed to be representing a sentence and characters in the word are parts of sentence. HMM and IBM model 1,3 and 4 iterations were run on word aligned bilingual parallel corpora. Input files to GIZA++ included language pair files in below format -
श क ु न ् त ल ा [Hindi Text File] < - > s h a k u n t l a [English Text File]

GIZA++ Package Programs :

• GIZA++: GIZA++ itself
• plain2snt.out: simple tool to transform plain text into GIZA format
• plain2snt.out: simple tool to transform GIZA format into plain text
• trainGIZA++.sh: Shell script to perform standard training given a corpus in GIZA format
• mkcls: Computes word classes in a monolingual corpus
• snt2cooc: Generates a coocurrence file

Run plain2snt.out located within the GIZA++ package to prepare text files for GIZA++ input
./plain2snt.out hindi english

Files created by plain2snt
english.vcb
hindi.vcb
hindienglish.snt

english.vcb consists of:
each word from the english corpus
corresponding frequency count for each word
an unique id for each word

hindi.vcb consists of:
each word from the hindi corpus
corresponding frequency count for each word
an unique id for each word

hindienglish.snt consists of:
each sentence from the parallel english and french corpi translated into the unique number for each word

Now, run GIZA++ located within the GIZA++ package
./GIZA++ -S hindi.vcb –T english.vcb –C hindienglish.snt

GIZA++ produces several files (distortion probability model, word alignments probability model, etc.). But there is one file that GIZA++ produces that is the only one we are going to care about. Its extension is: * .A3.final. This file is an alignment probability table for HMM alignment mode ( *.A3.* ). This is the one that has the final word-to-word alignment for each of the words of each line (in the same order as the input parallel corpus).

Python scripts were written to perform text processing to get required format for input files and parsing base corpora.

The corpus is a collection of about 100 songs which are written in romanized hindi and parallel hindi in devnagari.

Click on the download link below to download the complete corpus.

Download

Algorithm

1. User inputs the string to be processed/transliterated. This will be in roman script.

2. Now there are few possibilities:

i. The word entered is in English. This is an example of forward transliteration. We put the word into a language identification model (in broad sense, this term is used if the algorithm is capable of working in multiple languages. Here in our case, it will be primarily be Hindi and English. A dictionary check for English would reveal everything. Advanced methods are available in the case of multiple scripts/languages.)If the word is in English, CMU dict will be used (It lists all the English words will with their pronunciation in IPA). We can map IPA to Devanagari characters. This should not be a difficult task. These words are then considered for backward transliteration also. Move on to step 3 for that.

ii. The word entered could be in English, but the spelling may be incorrect. For example: “transliteration” or “traanslitareshan”
Here we can put a modified edit distance to check if the word entered relates to some English word. If the score is pretty good, then we can consider it for forward transliteration as well as backward transliteration (the same might not be the case that it is necessarily an English word).

iii. The word entered is a Hindi word. obviously, if the word is not an English word, and the edit distance is not very good, it should be a Hindi word. in this case we’ll move on to next step.

3. At this point, we are considering a generative model for transliteration. We’ll use the predefined probabilities generated by the alignment process (i.e. the training model). The word generated using the top most priority graphemes is given a score. Similarly the words generated using the lower probabilities are also given a score. Now these words are checked against the monolingual Hindi dictionary we already have. This dictionary will contain all kinds of Hindi words (we’ll have to check its size for optimization). The checking process with the monolingual dictionary is done using edit distance algorithm (different implementations of edit distance algorithm are available, see below for a little explanation provided here). If the distance score is good ( i.e. less is the distance) or say above a threshold score, we then consider it.

4. Now we have two kinds of score, one is generated using the generative model and one score is generated using the edit distance between the words. the edit distance score is calibrated to the scale of generative model (this may seem a bit complex here, but after discussion we’ve found that this may be possible). Now both the scores are added to each other and a final score is produced which decides the rank of these words.
In case the edit distance score is not able to qualify for the second stage (i.e. where the two scores are added), we may consider the edit distance score to be 0.
Now those words which were left at the stage (i) and (ii) also have to be considered (English words written correctly and which were in English but written incorrect) here. This creates a complexity for us.
As a word may have a meaning in English as well as in Hindi Example
“main” — मैं or मेन

5. We can discuss the final scoring process for such words.
Now the things which are left are, context sensitiveness of the model. Context sensitiveness here refers to the context of word. It may be an English word or a Hindi word, some different contexts are also possible. If the context prefers an English word, then the English word should be preferred. (This can be discussed if we will directly give it the first priority or include it in the final scoring process and then output the result).
This is a very important case because one spelling can refer to more than English or Hindi relevant words. We assume that this work can be done statistically. We can discuss upon the feasibility of such a modeln

Training Process:

We going to use GIZA++ for the training process. Giza++ will give the character level alignments with probabilities of particular alignments. We need to have an n-gram model. The n-gram conditional probablities will be generated by multiple iterations of training set with Giza++ with everytime the training data being segmented according to n-grams in a lookup inventory.

Edit Distance Algorithm:

Giving a short introduction to the basics. This algorithm  checks the distance between two words/string. The distance is equal to the number of steps took to convert one word into the the other.
In this conversion process, the possible actions are : deletion of a char, insertion of a character, substitution of one character with another.

Example: kitten vs sitting
1.kitten -> sitten (substitution)
2.sitten -> sittin (substitution)
3.sittin -> sitting (insertion)

Some variations to this algorithm are available. Provided is very basic stuff.

Congratulation wM family. SARAI has decide to help wordMint project. wordMint is selected for The Sarai - CSDS Independent FLOSS Fellowship 2008.

This will accelerate wordMint. We hope that with this fellowhip, the first version of wordMint will be released within six month.

Thanks SARAI - CSDS for showing confidence in us.

Kapil has suggested use of statistical algorithm. Statistical algorithms has an advantage of language neutral designs. He said that this will result a simpler design of wM.

Recently Criss and Jaiz came up with a new idea. It’s kind of modified statistical and correspondence based weighted finite state machine.

Recently Jaiz proposed that we should develope the wM project in python. He said specific advantage of python would be its interpetedness and platform indpendance also it is a language of Open Source World.

Java and C++ were close competitor but Python was beautiful enough to get them rejected. Java is yet not standardized and lacks any extendability. C++ is not exactly rejected as python can be extended by any C or C++ Module (for more information click here).

Python can be readily integrated in CGI, which would make same tool available online and offline (for more information click here and here). Although Java’s EJB could have also been an option  but it involves the J2EE servers.

Well we have open options of developing different language modules forks. The official language of wM is Python.

[You can find more comparision here]

We have already selected Python as the devlopement language. Criss is undertaking desktop inteface and CP is undertaking the web interface of WM. Kapil and Jaiz are working on the mighty algorithms. Criss will now be helping kapil and Jaiz due to overload in algorithm while he has nothing to do in desktop interface at the moment.

Thanks very much to Nalin Makar for the great looking theme HemingwayEx, though we modified it according to our logo and needs with a bit of CSS.

Now something about the project :

The project’s aim is to develop a free and open source intelligent transliteration system for english to hindi back transliteration mainly. We’ll try to make it with equal offline capabilities. For the same, artificial intelligence — statistical machine learning algorithhms are used. So that we can accurately predict what a person intended to write in Hindi.

In contrast with the traditional letter to letter mapping — grapheme model, the power of such a system lies in it’s intelligence. Where in the traditional system a person has necessarily to remember the character (it may be phonetic or not) map which makes the same work more complex.

while in the traditional model

we use character to character mapping (more technically grapheme to grapheme mapping) as

t –> ट्
th –> त्
T –> ठ्

so whenever they are used in the source grapheme, irrespective of the context or correctness, the target grapheme is generated based on the mapping.

While in our proposed method, this work will be done intelligently. For example, writing ‘namaste’ or ‘nmaste’ both would produce ‘नमस्ते’ , nulling the need of remembering the complex mapping. Only the knowledge of english keyboard and the pronunciation of different grapheme is enough.

Although different implementations are available, google indic transliteration being very popular, but no one of them is Open Source and also the offline feature is not available. Our target would be to make this system as efficient as possible.

This idea came out from the workshop on Hindi/Urdu/Kashmiri Localization organized at Sarai.

We have started our work, hope to get the first build in two months. Click on the Contact Us tab to contact us. You can send your suggestions or ideas to us by email as written on the Contact Us page.

So happy minting!!!