YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

MBT-2-branch-W2

Configuration Value
Block allocation 3 / (6 + 6) / 3
Outer d / m / f 1152 / 576 / 576
Branch d / m / f 896 / 464 / 464
Outer Q / K / V heads 18 / 9 / 9
Branch Q / K / V heads 14 / 7 / 7 (shared K/V)
Head dimension 64
MoE 64 experts, top 8
Context 4,096 tokens

Megatron-native model-only safetensors checkpoint.

Downloads last month
49
Safetensors
Model size
1B params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including gaotang/MBT-2-branch-W2