|
|
|
|
|
<div style="max-width: 940px; margin: 16px 0; border:1px solid #2a2f3a; border-radius:8px; background:#0b0f19; font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, 'Liberation Mono', 'Courier New', monospace; color:#e5e7eb;"> |
|
|
<div style="display:flex; align-items:center; gap:8px; padding:8px 10px; border-bottom:1px solid #1f2430; background:#111827; border-top-left-radius:8px; border-top-right-radius:8px;"> |
|
|
<span style="width:10px; height:10px; background:#ef4444; border-radius:50%; display:inline-block;"></span> |
|
|
<span style="width:10px; height:10px; background:#f59e0b; border-radius:50%; display:inline-block;"></span> |
|
|
<span style="width:10px; height:10px; background:#22c55e; border-radius:50%; display:inline-block;"></span> |
|
|
<span style="margin-left:8px; font-size:12px; color:#9ca3af;">attention-mask-visualizer</span> |
|
|
</div> |
|
|
<div style="padding:12px 14px; overflow:auto; font-size:12.5px; line-height:1.4;"> |
|
|
<pre style="margin:0; white-space:pre; tab-size:2;"> |
|
|
ATTN MASK β GPT-2 (causal) |
|
|
Tokens: [The, cat, sat, on, the, mat] |
|
|
Legend: x = can attend, . = masked (future) |
|
|
|
|
|
The cat sat on the mat |
|
|
The x |
|
|
cat x x |
|
|
sat x x x |
|
|
on x x x x |
|
|
the x x x x x |
|
|
mat x x x x x x |
|
|
|
|
|
|
|
|
ATTN MASK β PaliGemma-style (bidirectional prefix + causal suffix) |
|
|
Prefix: [<i0> <i1> <i2> <i3> <i4> What is this] |
|
|
Suffix: [A great duck] |
|
|
Legend: β = can attend, β = cannot |
|
|
|
|
|
<i0><i1><i2><i3><i4> What is this | A great duck |
|
|
<i0> β β β β β β β β β β β |
|
|
<i1> β β β β β β β β β β β |
|
|
<i2> β β β β β β β β β β β |
|
|
<i3> β β β β β β β β β β β |
|
|
<i4> β β β β β β β β β β β |
|
|
What β β β β β β β β β β β |
|
|
is β β β β β β β β β β β |
|
|
this β β β β β β β β β β β |
|
|
-------------------------------------------------------------------- |
|
|
A β β β β β β β β β β β |
|
|
great β β β β β β β β β β β |
|
|
duck β β β β β β β β β β β |
|
|
</pre> |
|
|
</div> |
|
|
</div> |
|
|
|