YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

MBT-4-branch-W2

Configuration Value
Block allocation 3 / (4 + 4 + 4 + 4) / 4
Outer d / m / f 1152 / 576 / 576
Branch d / m / f 640 / 384 / 384
Outer Q / K / V heads 18 / 9 / 9
Branch Q / K / V heads 10 / 5 / 5 (shared K/V)
Head dimension 64
MoE 64 experts, top 8
Context 4,096 tokens

Megatron-native model-only safetensors checkpoint.

Downloads last month
49
Safetensors
Model size
1B params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including gaotang/MBT-4-branch-W2