双语词向量映射代码解读vecmap

在上一篇文章中无监督机器翻译《An Effective Approach to Unsupervised Machine Translation》说现在的机器翻译模型有部分优秀模型不再使用平行语料也能够完成翻译。

为什么可以不再使用双语平行语料？

双语词向量映射python代码——vecmap

为了构建自己的跨语种的词向量之间的映射操作，首先需要选择一个单语种词向量训练工具(e.g. word2vec or fasttext)，然后再用今天的工具vecmap将一种单语种映射为另一种单语种。

实验测试数据获取

./get_data.sh

多种Mapping操作

Supervised方法

python3 map_embeddings.py --supervised TRAIN.DICT SRC.EMB TRG.EMB SRC_MAPPED.EMB TRG_MAPPED.EMB

Semi-supervised方法

python3 map_embeddings.py --semi_supervised TRAIN.DICT SRC.EMB TRG.EMB SRC_MAPPED.EMB TRG_MAPPED.EMB

Identical方法

python3 map_embeddings.py --identical SRC.EMB TRG.EMB SRC_MAPPED.EMB TRG_MAPPED.EMB

Unsupervised方法

python3 map_embeddings.py --unsupervised SRC.EMB TRG.EMB SRC_MAPPED.EMB TRG_MAPPED.EMB

·SRC.EMB是源语言embedding
·TRG.EMB是目标语言embedding
·SRC_MAPPED.EMB是映射后的源语言embedding
·TRG_MAPPED.EMB是映射后的目标语言embedding
·TRAIN.DICT

Unsupervised方法实践

小demo数据集准备---白雪公主(中文，英文)
英文文件名en.txt

In the harsh winter season, the goose-like snow flakes are flying around in the sky. There is a queen sitting in a window in the palace, doing needlework for her daughter, the wind blowing snow flakes into the window, the ebony window sill There are a lot of snowflakes falling on it. She looked up and looked out the window. She did not pay attention, and a needle stuck into her finger. The red blood flowed out of the needle, and three drops of blood fell on the snow on the window. She thoughtfully stared at the red blood drops on the white snow, and looked at the ebony window sill. She said, "I hope my little daughter's skin will be white and red, and it looks like this white snow and The red blood is the same, so gorgeous, so arrogant, the hair looks like the ebony of this window is generally black and bright!"
Her little daughter has grown up, and the little girl is so beautiful that she is beautiful, beautiful and moving. Her skin is really white like snow, ruddy with blood, and black hair like ebony. So the queen gave her a name, called Snow White. But Snow White has not grown up, and her queen mother died.
Soon, the father of the king married another wife. This queen is very beautiful, but she is very proud and conceited. She is very strong and can't stand it if she hears someone is more beautiful than her. She has a mirror, she often goes to the mirror to appreciate herself and asks: "Tell me, mirror, tell me the truth! All the women here are the most beautifulTell me who she is?"
The mirror replied: "It's you, queen! You are the most beautiful woman here."
When she heard this, she would smile with satisfaction. But Snow White grew up slowly and became more and more beautiful. By the time she was seven years old, she was more dazzling than the bright spring, more beautiful than the queen. Until one day, when the queen went to ask the mirror as usual, the mirror made the answer: "The queen, you are beautiful and beautiful, but Bai Xuegong is more beautiful than you!"
She heard this, her heart filled with anger and jealousy, and her face became pale. She called a servant and said to him, "Give me Snow White to the big forest. I don't want to see her anymore." The servant took Snow White away. When he was about to kill her in the forest, she cried and begged him not to kill her. Facing the pleading of the pitiful little princess, the servant’s sympathy came to life. He said, “You are a child who loves you, I will not kill you.” In this way, he left her alone. In the forest. When the servant decided not to kill Snow White and left her there, even though he knew that in the uninhabited big forest, she would be torn into pieces by the beast, but thought that he did not have to kill her by hand. He felt that a heavy stone that was pressing on his heart fell.
After the servant left, Snow White was very scared. She was everywhere in the forest, looking for a way out. The beast screamed beside her, but did not hurt her. In the evening, she came to a small house. When she determined that there was no one in the house, she pushed the door and went to rest, because she was really unable to move. As soon as she entered the door, she found that everything in the house was well organized and clean. A table was covered with white cloth with seven small plates, each with a piece of bread and some other food. There were seven glasses filled with wine next to the plate, seven knives. And the fork, etc., and the wall is also discharged with seven small beds. At this time she felt hungry and thirsty, and she did not care who it was. She went up to cut a small piece of bread from each piece of bread and drank a little bit of wine in each glass. After eating and drinking, she felt very tired and wanted to lie down and rest. So she came to the bed and almost tried every single of the seven beds. It was not too long, it was too short. It was not until the seventh bed was tried. She lay down on it and soon fell asleep.

中文文件名zh.txt

严冬时节，鹅毛一样的大雪片在天空中到处飞舞着，有一个王后坐在王宫里的一扇窗子边，正在为她的女儿做针线活儿，寒风卷着雪片飘进了窗子，乌木窗台上飘落了不少雪花。她抬头向窗外望去，一不留神，针刺进了她的手指，红红的鲜血从针口流了出来，有三点血滴飘落在窗子的雪花上。她若有所思地凝视着点缀在白雪上的鲜红血滴，又看了看乌木窗台，说道：“但愿我小女儿的皮肤长得白里透红，看起来就像这洁白的雪和鲜红的血一样，那么艳丽，那么骄嫩，头发长得就像这窗子的乌木一般又黑又亮！”
她的小女儿渐渐长大了，小姑娘长得水灵灵的，真是人见人爱，美丽动人。她的皮肤真的就像雪一样的白嫩，又透着血一样的红润，头发像乌木一样的黑亮。所以王后给她取了个名字，叫白雪公主。但白雪公主还没有长大，她的王后妈妈就死去了。
不久，国王爸爸又娶了一个妻子。这个王后长得非常漂亮，但她很骄傲自负，嫉妒心极强，只要听说有人比她漂亮，她都不能忍受。她有一块魔镜，她经常走到镜子面前自我欣赏，并问道：“告诉我，镜子，告诉我实话！这儿所有的女人谁最漂亮？告诉我她是谁？”
镜子回答道：“是你，王后！你就是这儿最漂亮的女人。”
听到这样的话，她就会满意地笑起来。但白雪公主慢慢地长大，并出落得越来越标致漂亮了。到了七岁时，她长得比明媚的春光还要艳丽夺目，比王后更美丽动人。直到有一天，王后像往常一样地去问那面魔镜时，镜子作出了这样的回答：“王后，你是美丽漂亮的，但是白雪公主要比你更加漂亮！”
她听到了这话，心里充满了愤怒和妒忌，脸也变得苍白起来。她叫来了一名仆人对他说：“给我把白雪公主抓到大森林里去，我再也不希望看到她了。”仆人把白雪公主带走了。在森林里他正要动手杀死她时，她哭泣着哀求他不要杀害她。面对楚楚动人的可怜小公主的哀求，仆人的同情之心油然而生，他说道：“你是一个人见人爱的孩子，我不会杀害你。”就这样，他把她单独留在了森林里。当仆人决定不再杀害白雪公主，而把她留在那儿时，尽管他知道在那荒无人际的大森林里，她十有八九会被野兽撕成碎片，但想到他不必亲手杀害她，他就觉得压在心上的一块沉重的大石头落了下来。
仆人走了以后，白雪公主一个人非常害怕，她在森林里到处徘徊，寻找出去的路。野兽在她身旁吼叫，但却没去伤害她。到了晚上，她来到了一间小房子跟前。当她确定这间房子没有人时，就推门走进去想休息一下，因为她已经实在走不动了。一进门，她就发现房子里的一切都布置得井井有条，十分整洁干净。一张桌子上铺着白布，上面摆放着七个小盘子，每个盘子里都装有一块面包和其它一些吃的东西，盘子旁边依次放着七个装满葡萄酒的玻璃杯，七把刀子和叉子等，靠墙还并排放着七张小床。此时她感到又饿又渴，也顾不得这是谁的了，走上前去从每块面包上切了一小块吃了，又把每只玻璃杯里的酒喝了一点点。吃过喝过之后，她觉得非常疲倦，想躺下休息休息，于是来到那些床前，七张床的每一张她几乎都试过了，不是这一张太长，就是那一张太短，直到试了第七张床才合适。她在上面躺下来，很快就睡着了。

数据处理代码

from gensim.models import fasttext
from gensim.models import word2vec
import jieba

def get_zh_embedding():
    sentance = []
    with open('zh.txt', 'r', encoding='utf8') as f:
        line = f.readline().strip()
        while line:
            sentance.append(line)
            line = f.readline().strip()

    ## 对句子进行分词分词
    def segment_sen(sen):
        sen_list = []
        try:
            sen_list = jieba.lcut(sen)
        except:
                pass
        return sen_list
    # 将数据变成gensim中 word2wec函数的数据格式
    sens_list = [segment_sen(i) for i in sentance]

    model = word2vec.Word2Vec(sens_list,min_count=1,iter=20)
    model.wv.save_word2vec_format('SRC.EMB', binary=False)


def get_en_embedding():
    sentance = []
    with open('en.txt', 'r', encoding='utf8') as f:
        line = f.readline().strip().lower()
        while line:
            sentance.append(line)
            line = f.readline().strip().lower()

    # 将数据变成gensim中 word2wec函数的数据格式
    sens_list = [i.strip().split(' ') for i in sentance]


    model = word2vec.Word2Vec(sens_list, min_count=1, iter=20)
    model.wv.save_word2vec_format('TRG.EMB', binary=False)

get_en_embedding()
get_zh_embedding()

SRC.EMB文件

TRG.EMB文件

运行代码

python3 map_embeddings.py --unsupervised SRC.EMB TRG.EMB SRC_MAPPED.EMB TRG_MAPPED.EMB

vecmap映射之后得到的SRC_MAPPED.EMB

vecmap映射之后得到的TRG_MAPPED.EMB

评估

TEST.DICT文件内容如下，格式：(源语词+空格+目标语词)同义词形式，尽量是前面两个embedding里有的词。

就 on
里 in
王后 queen
白雪公主 snow
白雪公主 white
他 he
在 in
又 also
我 i
漂亮 pretty
一样 same
长 long

训练语句

python3 eval_translation.py SRC_MAPPED.EMB TRG_MAPPED.EMB -d TEST.DICT

训练结果(因为是一个小demo，所以模型按照以上文件格式能够正常运行，但效果要好，可以通过原始的get数据来测试，或者自己用大语料，多次迭代来训练单语言词向量)