Dataset Viewer
Auto-converted to Parquet Duplicate
document_id
stringclasses
30 values
format
stringclasses
3 values
text
stringlengths
1.36k
21k
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 1: Why Byte Latent Transformer? Motivation and Contributions # Limitations of tokenization: heuristic compression, domain/modality bias, noise sensitivity, orthographic blind spots, multilingual inequities Most mo...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 10: Fixed-Inference Scaling: Patches Scale Better Than Tokens # Hypothesis: fewer global steps on larger patches frees compute to scale the global model The central hypothesis driving the fixed-inference scaling e...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 11: Robustness and Long-Tail Generalization # Character-level noising (AntSpeak, Drop, RandomCase, Repeat, UpperCase): BLT’s consistent gains Character-level noise exposes a weakness in models that only see compre...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 12: Ablations, Design Trade-offs, and Practical Guidance # Entropy model size and context window: diminishing returns after ~50M params The entropy model is the small byte-level autoregressive LM used to estimate n...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 13: Bridging from Token Models: Byte-ifying LLaMA # Initialization of global parameters and differential learning rates One efficient way to train a Byte Latent Transformer (BLT) is to reuse the already-trained kn...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 14: Related Work and Positioning # Character-level RNNs and CNNs; early Transformer character LMs The study of language at the byte or character level predates modern transformer-based large language models. Early...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 15: Limitations, Engineering Considerations, and Future Directions # Compute-optimal ratios may differ from token models; need BLT-specific scaling laws One of the paper’s explicit caveats is that the scaling laws...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 16: Summary and Practitioner’s Checklist # Core idea: allocate global-transformer compute via entropy-driven patches The central idea of Byte Latent Transformer (BLT) is to move away from a fixed-vocabulary tokeniz...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 2: Background: From Tokens to Bytes and Information-Theoretic Metrics # Tokenization basics: BPE/subword vocabularies and their trade-offs Modern large language models (LLMs) almost always begin by converting raw ...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 3: From Individual Bytes to Patches: Patching Functions and Their Costs # Formal definition of a patching function and patch size Patching is the process of grouping a raw byte sequence into larger units—patches—t...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 4: Entropy Patching in Depth # Training a small byte-level LM to estimate next-byte entropy Entropy patching in BLT begins with an auxiliary byte-level auto-regressive language model (the entropy model) whose sole ...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 5: BLT at a Glance: The Global–Local Architecture # Three-block design: local encoder, latent global transformer, local decoder The Byte Latent Transformer (BLT) is built around a simple but powerful three-block de...
arxiv/ByteLatent
textbook
\title{A Textbook about the Paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"} Chapter 6: Local Encoder: From Bytes to Patch Representations # Byte embeddings and motivation for augmenting local context The local encoder is the component of the Byte Latent Transformer (BLT) whose job is to turn raw ...
End of preview. Expand in Data Studio

Auxiliary Views Knowledge Acquisition

This repository contains the cleaned source documents, training paraphrases, auxiliary views, and evaluation probes used in Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views (arXiv:2609.04180).

News

  • August 21, 2026: Our paper was accepted to Findings of EMNLP 2026.

Configurations

Configuration Split Rows
documents train 30
factual_cloze test 6,435
factual_mcqa_5shot test 4,515
inference_cloze test 430
inference_mcqa_5shot test 322
paraphrases train 270
auxiliary_views train 702

Both multiple-choice configurations include five-shot prompts in formatted_question_5shot.

Training views

Auxiliary views generated by GPT-5 mini.

Configuration Columns
paraphrases document_id, paraphrase_index, text
auxiliary_views document_id, format, text

Paraphrases contain versions 0–8, the nine selected by the training recipe. Auxiliary formats are textbook, blog, and QA, with one chapter, blog post, or QA file per row. Both configurations cover the same 30 released source documents. document_id links each row to the source attribution in ATTRIBUTION.md and the documents configuration; applicable source terms and the mixed-rights notice in LICENSE.md still apply.

from datasets import load_dataset

paraphrases = load_dataset(
    "jiosephlee/auxiliary-views-knowledge-acquisition", "paraphrases", split="train"
)
views = load_dataset(
    "jiosephlee/auxiliary-views-knowledge-acquisition", "auxiliary_views", split="train"
)

Copyright and source documents

This is a mixed-rights repository; see LICENSE.md and ATTRIBUTION.md. The 30 released documents comprise:

  • 6 arXiv papers licensed CC BY 4.0;
  • 12 BMJ Case Reports articles licensed CC BY-NC 4.0; and
  • 12 public-domain U.S. federal judicial opinions obtained via CourtListener.

Intended use and limitations

The probes are evaluation data, not training/validation splits. Medical source documents are case reports and must not be used for diagnosis, treatment, or clinical decision-making. The corpus is small, English-only, intentionally heterogeneous, and is not representative of any of its three domains.

Citation

@misc{lee2026knowledgeacquisitionpretraininglarge,
  title         = {Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views},
  author        = {Joseph Lee and Yidi Huang and Dokyoon Kim and Shu Yang and Li Shen},
  year          = {2026},
  eprint        = {2609.04180},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.04180},
}
Downloads last month
99

Paper for jiosephlee/auxiliary-views-knowledge-acquisition