Multi-Branch Transformer
Collection
18 items โข Updated โข 2
YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
| Configuration | Value |
|---|---|
| Block allocation | 3 / (6 + 6) / 3 |
| Outer d / m / f | 1152 / 576 / 576 |
| Branch d / m / f | 896 / 464 / 464 |
| Outer Q / K / V heads | 18 / 9 / 9 |
| Branch Q / K / V heads | 14 / 7 / 7 (shared K/V) |
| Head dimension | 64 |
| MoE | 64 experts, top 8 |
| Context | 4,096 tokens |
Megatron-native model-only safetensors checkpoint.