The goal of this lab is to experiment with word-based statistical machine translation, on an easy text type. The aim is to gain a basic understanding of the role of the translation and language models. The lab contains two parts with some sub tasks each.
In part 1 of the lab you will manually change the probabilities of translation and language models. The goal is that you should get a feeling of how the probabilities affect the system. The setup is, of course, artificial. Normally you would not manipulate probabilities in this way, but estimate them from data. You should especially never change probabilities based on a very small test set, as in this lab. Here we do it just so that you can get a feeling of how the translation and language model works. For the language model you will also train a model on data in part 2 of the lab , which is what you would normally do also for the translation model.
mkdir lab2
cd lab2
cp /local/kurs/mt/lab2/data/* .
In this lab we will use a simple word-based decoder, to translate sentences from the block world between Swedish and English. The decoder does not allow any reordering. It uses a translation model that consists of two parts, word translation probabilities and fertility probabilities, and a language model. In this lab you will adjust probabilities by hand, in order to get a feeling for how they affect translation, instead of training them on a corpus, as is normally done.
The word translation model contains the probability of a source word translating into a target word, or to the special NULL word, i.e. that it is not translated. The format is that each line contains a target word, a source word and a probability, separated by white space, as:
block blocket 1
take ta 1
the den 0.4
the det 0.1
the NULL 0.5
The fertility model contains probabilities for how many words each word can translate into. For most words in the blocks world, there will be probability 1 that they translate into 1 word. The format is again one entry per line, white space separated, containing a source word, a fertility (0-2 in the current implementation, it is enough for the blocks world) and a probability, as:
block 1 1
take 1 1
the 0 0.5
the 1 0.5
The language model contains probabilities for n-grams in the target language. It contains minimum 1-grams, but can also contain higher order n-grams. In the first part you do not need to use more than 3-grams. The LM-format is the ARPA format, but extra headings and backoff probabilities are ignored. That means that for each n-gram, it starts with a line with the tag \n-gram, followed by lines with a probability and n words, separated by white space, such as:
\1-gram
0.3 block
0.3 take
0.4 the
\2-gram
1 take the
1 the block
The given fertility models are called tmf.[swe/eng], the word translation models are called tmw.[sweeng/engswe], and the language models are called lm.[swe/eng]. The given models contains words and word pairs that you need for the lab, initialized with uniform probabilities.
/local/kurs/mt/lab2/simple_decoder/translate -lm LM-file -tmw WORD-translation-model -tmf FERTILITY-model -o -max-ngram-order
/local/kurs/mt/lab2/simple_decoder/translate -lm lm.eng -tmw tmw.sweeng -tmf tmf.swe -o 2
In this assignment we will work on the blocks world domain, which contains commands and descriptions of taking and putting different types of blocks on different positions. The model files provided contains all words in the vocabulary with equal probabilities. The sentences you should work on translating are shown below. They are also in the files test_meningar.[swe/eng] that you copied from the course area For the lab we consider the given translations the only possible, and ignore any potential ambiguities. You can thus consider full sentences correct or incorrect, and do not have to use metrics such as Bleu.
| ta en pil take an arrow |
| ställ en kon på mitt block put a cone on my block |
| hon tar blocket she takes the block |
| ställ en röd pil på en blå cirkel put a red arrow on a blue circle |
| jag tar ett blått block i take a blue block |
| ställ pilen på cirkeln put the arrow on the circle |
| han ställer det röda blocket på den blåa cirkeln he puts the red block on the blue circle |
| han ställer en pil på sin cirkel he puts an arrow on his circle |
| hon ställer sitt block på min blåa cirkel she puts her block on my blue circle |
| jag ställer konen på cirkeln på hennes blåa cirkel i put the cone on the circle on her blue circle |
The given LM and TM files contain the full vocabulary, and all needed fertilities and word translations. In part 2 you will work on language modeling, and a small corpus is provided for that. If you want to you can have a look at it now, to get a better feel for the domain. These files are named corpus.*.*
We will work on translation between Swedish and English. For non-Swedish speakers, there is a brief description of the relevant language phenomena. You should translate in both directions, but non-Swedish speakers can focus most of their discussion on translation into English.
In all assignments you should try to get one set of weights that gives the globally best results for all 10 test sentences. This might mean that a change makes the result worse for one single sentence, but better for two other ones. Your task is to find a good compromise of weights that work reasonably well across all 10 sentences.
The decoder contains a function which gives you the rank of the correct hypothesis in the n-best list, and the average rank for all sentences. If a sentence does not have a translation the rank will be approximated to 500, which means the average rank is not very trustworthy in that case. But the decoder should be able to translate all provided sentences. For evaluation the nbest-flag of the decoder is ignored even if given, and all translation hypothesis are explored. To run this function you use "-eval referenceFile" as an argument to the decoder, for instance:
# for translation from Swedish to English
/local/kurs/mt/lab2/simple_decoder/translate -lm lm.eng -tmw tmw.sweeng -tmf tmf.swe -o 2 -in test_meningar.swe -eval test_meningar.eng
# for translation from English to Swedish
/local/kurs/mt/lab2/simple_decoder/translate -lm lm.swe -tmw tmw.engswe -tmf tmf.eng -o 2 -in test_meningar.eng -eval test_meningar.swe
Hand in your report for part 2 as a pdf through the student portal. Deadline for handing in the report: April 28, 2017.
If you failed to attend the session for part 1 you also have to hand in a report for that part. If both persons in the pair missed part 1, do it together with the results from part 2. If one person missed the part 1 session, that person should hand in an individual report of part 1. In that case there should be another joint report from that pair for part 2.