You need to agree to share your Hugging Face information to access this dataset

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this dataset content.

Research use only, non-commercial. Turn-taking timing in the skel_* batches is derived from CC-BY-NC-4.0 corpora (Seamless Interaction, SpokenWOZ); no audio or transcript of those corpora is included. Speech is synthetic.

fullduplex_model shared training data, v1

This folder is private. Do not re-share it publicly or pass the link on. Generation is still running: files are added over time, and the text files are replaced until their runs finish.

Layout

MANIFEST.jsonl                       one row per uploaded file: path, bytes, sha256, created (UTC)
README.md
text/<run>.jsonl.zst                 spoken-instruct text dialogues, one JSON record per line (replaced while the run grows)
text/stats.json                      counts per run, template and exclusion reason
speech/<batch>/<batch>-00000.tar     two-channel speech dialogues, uncompressed tar, about 4 GiB, never rewritten
speech/<batch>/<batch>-00000.index.jsonl   one row per dialogue in that tar
speech/<batch>/<batch>-00000.meta.json     bytes and sha256 of that tar, dialogue count, hours
qc/speech_qc.jsonl                   ASR check per dialogue: drop decision and turns to mask (rebuilt every round)
qc/unchecked.txt                     <batch>/<key> of packed dialogues the ASR check has not reached yet
qc/stats.json                        counts behind the two files above
packs/<family>/<arm>.tar             optional: tokenised proxy-training packs, with <arm>.meta.json

MANIFEST.jsonl lists only files whose upload was verified. A shard appears once it is sealed; a batch that is still being generated has a last, unsealed shard that is not here yet.

Download and verify

rclone copy <remote>:fullduplex_share/v1 ./v1 --checksum --transfers 4 -P
cd v1 && python3 - <<'EOF'
import hashlib, json
for line in open("MANIFEST.jsonl"):
    r = json.loads(line)
    h = hashlib.sha256()
    with open(r["path"], "rb") as f:
        while chunk := f.read(1 << 22):
            h.update(chunk)
    print("ok  " if h.hexdigest() == r["sha256"] else "BAD ", r["path"])
EOF

A later rclone copy with --checksum fetches only new and replaced files. Each flac inside a tar can be checked against flac_sha256 in the index.

Text: spoken-instruct dialogues

Read with zstd -dc text/<run>.jsonl.zst or Python zstandard. The files hold accepted records of prompt template spoken_instruct_v5 only, one per seed. Rejected records, records of earlier templates and dialogues that share a 13-word sequence with a benchmark are left out; text/stats.json counts each group.

Each dialogue was written in one call by gpt-6.1-terra from a seed: the first user turn of a Dolci-Instruct-SFT conversation, used as the user's intent. The seed's reply is not copied.

Field Meaning
seed_id, subset, license, content_hash the seed: its id, Dolci subset, that subset's licence, hash of the seed text
plan what the generator was asked for: persona, situation, fillers (per 100 words), follow_ups (kinds of the later user turns), cut (the follow-up that interrupts, or null), user_bc, openers, fillers_inside, frame, budget (turns and marker counts)
turns[] role (user / assistant); text as written, with markers; plain the text-only view (markers removed, the words after a [cut] dropped); say the full text to voice (markers removed, the words after a [cut] kept); markers[]
turns[].markers[] type bc (the listener says text right after the word with 0-based index after in this turn) or cut (the other speaker interrupts right after that word)
written marker counts found in the dialogue (user_bc, assistant_bc, cut, ov)
marker_issues, style, flags checker output: marker problems, style leak tags, and limitation / digits / praise_opener flags
reject, skip, error always null in this export
call_id, model, family, template, template_hash, spec_version, opener_fit generation lineage

Numbers are written in digits, as a transcript shows them. The dialogue text can contain public contact details that the seed or the model brought in (organisation phone numbers and the like).

Speech: two-channel dialogues

Each dialogue is <key>.flac (24 kHz, 16-bit, channel 0 = user, channel 1 = assistant, no noise added) and <key>.json. Both are in exactly one tar. Members are stored uncompressed, so a dialogue can be read straight from the tar with flac_offset / flac_bytes and json_offset / json_bytes from the index.

All speech is synthetic. The user's words come from an LLM user simulator (gpt-6.1-terra) and are spoken by a realtime speech model acting as a voice actor; the assistant is a realtime speech model (gpt-realtime-2.1 requested) that hears the user audio and answers, grounded in an excerpt of a web or PDF document (doc_id).

Index row (<shard>.index.jsonl): key, batch, shard, source, source_pass, spec, hours, n_samples, sr, flac_bytes, flac_sha256, flac_offset, json_bytes, json_sha256, json_offset, end.

JSON field Meaning
key, doc_id dialogue id (16 hex digits) and the grounding document
sr, n_samples sample rate and length; equal to the FLAC's
card the scenario given to the simulator: persona, accent, style, situation, environment (a tag only; the audio is clean), topic, hint (first sentence of the document), voices, target words per second and speed per speaker
voices user and assistant voice names
events[] every piece of audio placed on the timeline, in order of placement. ch (user / assistant), kind, start / end in samples, text
events[].kind utterance; utterance_cut (stopped early because the other speaker barges in); barge_in (starts inside the other speaker's turn and takes the floor); backchannel (short listener token inside the other speaker's turn)
events[].text user: the simulator's text. assistant and backchannel: the speech model's own transcript. For a cut event this is the full planned text while the audio stops early
events[].spoken_transcript user events: the speech model's transcript of what it said
events[].cut onset and stop in samples from the event start: where the interruption begins and where the audio ends
events[].turn, skel_fto_s, rendered_fto_s, skel_frac skeleton batches: index of the skeleton turn, its gap to the previous turn in the real dialogue and as rendered (seconds, negative = overlap), and for a backchannel its position as a fraction of the turn
models, provenance, timing, spec, seed, time generator models, family / modality, placement parameters, generator version, seed, generation time
source, source_dialogue_id, source_kind, skeleton, source_pass, turns_rendered, bc_bank skeleton batches only, see below
duet duet_* batches only: the sampled interaction style (probabilities of barge-in, overlap, backchannel)

source is duet in the index for duet_* batches (their JSON has no source field) and source_pass is null.

What comes from real dialogues

The skel_* batches copy their turn-taking timing from one real dialogue each, taken from the train splits of Seamless Interaction (seamless_naturalistic), otoSpeech full-duplex 280 h (otospeech_fd280) and SpokenWOZ (spokenwoz). The JSON stores this much of the real dialogue and nothing else:

  • source, source_dialogue_id: the dataset tag and that dataset's dialogue id.
  • skeleton.window_s: start and end, in seconds of the real recording, of the stretch that was used.
  • skeleton.turns[]: per floor turn who (the role given to that speaker), src_start (start in the real recording, seconds), dur_s, fto_s (gap to the previous turn, negative = overlap), interrupts, onset_frac (where in the previous turn the interruption starts), bc_fracs (positions of the listener's backchannels as fractions of the turn).
  • source_pass: 1 for the first use of a real dialogue, 2 when it is used a second time with another document, persona and voices (skel_p2_*). JSONs written before the field existed lack it; the index fills in 1.

No real audio and no real transcript is included: every word in events was generated. bc_bank is the log of the generated backchannel clips per speaker (word asked for, attempts, kept or not, what was spoken). The duet_* batches use sampled timing and no real dialogue.

Licences of the timing sources as recorded in our notes: Seamless Interaction CC-BY-NC-4.0, SpokenWOZ CC-BY-NC-4.0, otoSpeech CC-BY-4.0. Confirm them before any use beyond internal research.

The generated words can repeat public contact details from the grounding document (an e-mail address or phone number on a web page), in card.hint and in events[].text.

Applying the ASR check

Every event was transcribed with Qwen3-ASR-1.7B and compared with its script. A scored turn is a non-backchannel event with at least 3 script words. qc/speech_qc.jsonl has one row per checked dialogue:

Field Meaning
key, batch, source, hours the dialogue
dialogue_wer word errors over script words, summed over the scored turns
n_turns_scored, n_turns_wer_gt_0.5 scored turns, and how many of them have WER above 0.5
drop, drop_reasons drop is true when dialogue_wer > 0.1 (dialogue_wer_gt_0.1) or two or more scored turns have WER > 0.5 (two_turns_gt_0.5)
flagged_eids indices into the JSON's events of scored turns with WER > 0.5: mask these turns in training when the dialogue is kept
qc_run the ASR run the row comes from

Three cases per packed dialogue:

  • row with drop true: leave the dialogue out.
  • row with drop false: use it, masking the events in flagged_eids.
  • no row (listed in qc/unchecked.txt): the ASR check has not reached it. It is neither dropped nor cleared. Wait for a later speech_qc.jsonl, or use it knowing the share of checked dialogues that were dropped (qc/stats.json).

The shards are packed before the check finishes, so decisions are not baked into the tars; always apply the newest speech_qc.jsonl.

import json
qc = {r["key"]: r for r in map(json.loads, open("qc/speech_qc.jsonl"))}
def use(key):                      # -> None (leave out), or the set of event indices to mask
    r = qc.get(key)
    if r is None:
        return None                # unchecked: decide your own policy
    return None if r["drop"] else set(r["flagged_eids"])

Packs (optional)

packs/<family>/<arm>.tar holds one token-pack directory as written by our mixing code (meta.json, train.bin, val*.bin). meta.json records the tokenizer's sha256; the tokenizer file is not in this folder, and the paths inside meta.json are paths on the generating server. Packs built from real transcripts (real arms) are not included.

Downloads last month
1,161