Multi-Branch Transformer
Collection
18 items โข Updated โข 2
YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
| Configuration | Value |
|---|---|
| Block allocation | 3 / (4 + 4 + 4 + 4) / 4 |
| Outer d / m / f | 1152 / 576 / 576 |
| Branch d / m / f | 640 / 384 / 384 |
| Outer Q / K / V heads | 18 / 9 / 9 |
| Branch Q / K / V heads | 10 / 5 / 5 (shared K/V) |
| Head dimension | 64 |
| MoE | 64 experts, top 8 |
| Context | 4,096 tokens |
Megatron-native model-only safetensors checkpoint.