Datasets:
document_id stringclasses 30
values | format stringclasses 3
values | text stringlengths 1.36k 21k |
|---|---|---|
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 1: Why Byte Latent Transformer? Motivation and Contributions
# Limitations of tokenization: heuristic compression, domain/modality bias, noise sensitivity, orthographic blind spots, multilingual inequities
Most mo... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 10: Fixed-Inference Scaling: Patches Scale Better Than Tokens
# Hypothesis: fewer global steps on larger patches frees compute to scale the global model
The central hypothesis driving the fixed-inference scaling e... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 11: Robustness and Long-Tail Generalization
# Character-level noising (AntSpeak, Drop, RandomCase, Repeat, UpperCase): BLT’s consistent gains
Character-level noise exposes a weakness in models that only see compre... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 12: Ablations, Design Trade-offs, and Practical Guidance
# Entropy model size and context window: diminishing returns after ~50M params
The entropy model is the small byte-level autoregressive LM used to estimate n... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 13: Bridging from Token Models: Byte-ifying LLaMA
# Initialization of global parameters and differential learning rates
One efficient way to train a Byte Latent Transformer (BLT) is to reuse the already-trained kn... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 14: Related Work and Positioning
# Character-level RNNs and CNNs; early Transformer character LMs
The study of language at the byte or character level predates modern transformer-based large language models. Early... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 15: Limitations, Engineering Considerations, and Future Directions
# Compute-optimal ratios may differ from token models; need BLT-specific scaling laws
One of the paper’s explicit caveats is that the scaling laws... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 16: Summary and Practitioner’s Checklist
# Core idea: allocate global-transformer compute via entropy-driven patches
The central idea of Byte Latent Transformer (BLT) is to move away from a fixed-vocabulary tokeniz... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 2: Background: From Tokens to Bytes and Information-Theoretic Metrics
# Tokenization basics: BPE/subword vocabularies and their trade-offs
Modern large language models (LLMs) almost always begin by converting raw ... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 3: From Individual Bytes to Patches: Patching Functions and Their Costs
# Formal definition of a patching function and patch size
Patching is the process of grouping a raw byte sequence into larger units—patches—t... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 4: Entropy Patching in Depth
# Training a small byte-level LM to estimate next-byte entropy
Entropy patching in BLT begins with an auxiliary byte-level auto-regressive language model (the entropy model) whose sole ... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 5: BLT at a Glance: The Global–Local Architecture
# Three-block design: local encoder, latent global transformer, local decoder
The Byte Latent Transformer (BLT) is built around a simple but powerful three-block de... |
arxiv/ByteLatent | textbook | \title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"}
Chapter 6: Local Encoder: From Bytes to Patch Representations
# Byte embeddings and motivation for augmenting local context
The local encoder is the component of the Byte Latent Transformer (BLT) whose job is to turn raw ... |
Auxiliary Views Knowledge Acquisition
This repository contains the cleaned source documents, training paraphrases, auxiliary views, and evaluation probes used in Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views (arXiv:2609.04180).
News
- August 21, 2026: Our paper was accepted to Findings of EMNLP 2026.
Configurations
| Configuration | Split | Rows |
|---|---|---|
documents |
train | 30 |
factual_cloze |
test | 6,435 |
factual_mcqa_5shot |
test | 4,515 |
inference_cloze |
test | 430 |
inference_mcqa_5shot |
test | 322 |
paraphrases |
train | 270 |
auxiliary_views |
train | 702 |
Both multiple-choice configurations include five-shot prompts in
formatted_question_5shot.
Training views
Auxiliary views generated by GPT-5 mini.
| Configuration | Columns |
|---|---|
paraphrases |
document_id, paraphrase_index, text |
auxiliary_views |
document_id, format, text |
Paraphrases contain versions 0–8, the nine selected by the training recipe.
Auxiliary formats are textbook, blog, and QA, with one chapter, blog post,
or QA file per row. Both configurations cover the same 30 released source
documents. document_id links each row to the source attribution in
ATTRIBUTION.md and the documents configuration; applicable
source terms and the mixed-rights notice in LICENSE.md still apply.
from datasets import load_dataset
paraphrases = load_dataset(
"jiosephlee/auxiliary-views-knowledge-acquisition", "paraphrases", split="train"
)
views = load_dataset(
"jiosephlee/auxiliary-views-knowledge-acquisition", "auxiliary_views", split="train"
)
Copyright and source documents
This is a mixed-rights repository; see LICENSE.md and ATTRIBUTION.md. The 30 released documents comprise:
- 6 arXiv papers licensed CC BY 4.0;
- 12 BMJ Case Reports articles licensed CC BY-NC 4.0; and
- 12 public-domain U.S. federal judicial opinions obtained via CourtListener.
Intended use and limitations
The probes are evaluation data, not training/validation splits. Medical source documents are case reports and must not be used for diagnosis, treatment, or clinical decision-making. The corpus is small, English-only, intentionally heterogeneous, and is not representative of any of its three domains.
Citation
@misc{lee2026knowledgeacquisitionpretraininglarge,
title = {Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views},
author = {Joseph Lee and Yidi Huang and Dokyoon Kim and Shu Yang and Li Shen},
year = {2026},
eprint = {2609.04180},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2609.04180},
}
- Downloads last month
- 99