YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Standard-W3

Configuration Value
Block allocation 16 sequential blocks
Outer d / m / f 1280 / 640 / 640
Branch d / m / f โ€”
Outer Q / K / V heads 20 / 10 / 10
Branch Q / K / V heads โ€”
Head dimension 64
MoE 64 experts, top 8
Context 4,096 tokens

Megatron-native model-only safetensors checkpoint.

Downloads last month
41
Safetensors
Model size
2B params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including gaotang/Standard-W3