The key/value/query formulation of attention is from the paper attention is all you need. For this, they use a sinusoidal embedding:. Do they use already pretrained such as off the shelf word2vec or glove embeddings ?
THEY'RE HIDING THIS FROM YOU (nutritionist reacts) YouTube
I believe you were somehow confused by some folks saying that the masked attention is essential for causality. 自然语言处理中，attention is all you need。 attention is all you need这篇文章的重要性不只是提出了attention这一概. 本视频对ashish vaswani和他的合著者的著名论文《attention is all you need》进行了全面的研究。该论文介绍了transformer架构，transformer架构广泛应用于自然语言处理和其他领域的.
In attention is all you need paper, regarding encoder (and decoder) input embeddings:
How should one understand the queries, keys, and values the key/value/query concept is analogous. Number of params in the models. I just wanted to add that causality is important during testing; But numbers from my calculation do not match.
Only then if the results were interesting, they read the paper more thoroughly. In attention is all you need, the authors implement a positional embedding (which adds information about where a word is in a sequence). So, the main idea of the attention is all you need paper was to replace the rnn layers completely with attention mechanism. 如何理解谷歌团队的机器翻译新作《attention is all you need》？ 本题已收录至知乎圆桌：机器之能 x 语言之美 ，更多「人工智能」相关话题欢迎关注讨论。 谷歌团队 6月13号发表在 arxiv 上的文章：.
