Research use only, non-commercial. Turn-taking timing in the skel_* batches is derived from CC-BY-NC-4.0 corpora (Seamless Interaction, SpokenWOZ); no audio or transcript of those corpora is included. Speech is synthetic.
fullduplex_model shared training data, v1
This folder is private. Do not re-share it publicly or pass the link on. Generation is still running: files are added over time, and the text files are replaced until their runs finish.
Layout
MANIFEST.jsonl one row per uploaded file: path, bytes, sha256, created (UTC)
README.md
text/<run>.jsonl.zst spoken-instruct text dialogues, one JSON record per line (replaced while the run grows)
text/stats.json counts per run, template and exclusion reason
speech/<batch>/<batch>-00000.tar two-channel speech dialogues, uncompressed tar, about 4 GiB, never rewritten
speech/<batch>/<batch>-00000.index.jsonl one row per dialogue in that tar
speech/<batch>/<batch>-00000.meta.json bytes and sha256 of that tar, dialogue count, hours
qc/speech_qc.jsonl ASR check per dialogue: drop decision and turns to mask (rebuilt every round)
qc/unchecked.txt <batch>/<key> of packed dialogues the ASR check has not reached yet
qc/stats.json counts behind the two files above
packs/<family>/<arm>.tar optional: tokenised proxy-training packs, with <arm>.meta.json
MANIFEST.jsonl lists only files whose upload was verified. A shard appears once it is sealed; a batch that is still
being generated has a last, unsealed shard that is not here yet.
Download and verify
rclone copy <remote>:fullduplex_share/v1 ./v1 --checksum --transfers 4 -P
cd v1 && python3 - <<'EOF'
import hashlib, json
for line in open("MANIFEST.jsonl"):
r = json.loads(line)
h = hashlib.sha256()
with open(r["path"], "rb") as f:
while chunk := f.read(1 << 22):
h.update(chunk)
print("ok " if h.hexdigest() == r["sha256"] else "BAD ", r["path"])
EOF
A later rclone copy with --checksum fetches only new and replaced files. Each flac inside a tar can be checked
against flac_sha256 in the index.
Text: spoken-instruct dialogues
Read with zstd -dc text/<run>.jsonl.zst or Python zstandard. The files hold accepted records of prompt template
spoken_instruct_v5 only, one per seed. Rejected records, records of earlier templates and dialogues that share a 13-word
sequence with a benchmark are left out; text/stats.json counts each group.
Each dialogue was written in one call by gpt-6.1-terra from a seed: the first user turn of a Dolci-Instruct-SFT
conversation, used as the user's intent. The seed's reply is not copied.
| Field | Meaning |
|---|---|
seed_id, subset, license, content_hash |
the seed: its id, Dolci subset, that subset's licence, hash of the seed text |
plan |
what the generator was asked for: persona, situation, fillers (per 100 words), follow_ups (kinds of the later user turns), cut (the follow-up that interrupts, or null), user_bc, openers, fillers_inside, frame, budget (turns and marker counts) |
turns[] |
role (user / assistant); text as written, with markers; plain the text-only view (markers removed, the words after a [cut] dropped); say the full text to voice (markers removed, the words after a [cut] kept); markers[] |
turns[].markers[] |
type bc (the listener says text right after the word with 0-based index after in this turn) or cut (the other speaker interrupts right after that word) |
written |
marker counts found in the dialogue (user_bc, assistant_bc, cut, ov) |
marker_issues, style, flags |
checker output: marker problems, style leak tags, and limitation / digits / praise_opener flags |
reject, skip, error |
always null in this export |
call_id, model, family, template, template_hash, spec_version, opener_fit |
generation lineage |
Numbers are written in digits, as a transcript shows them. The dialogue text can contain public contact details that the seed or the model brought in (organisation phone numbers and the like).
Speech: two-channel dialogues
Each dialogue is <key>.flac (24 kHz, 16-bit, channel 0 = user, channel 1 = assistant, no noise added) and
<key>.json. Both are in exactly one tar. Members are stored uncompressed, so a dialogue can be read straight from the
tar with flac_offset / flac_bytes and json_offset / json_bytes from the index.
All speech is synthetic. The user's words come from an LLM user simulator (gpt-6.1-terra) and are spoken by a realtime
speech model acting as a voice actor; the assistant is a realtime speech model (gpt-realtime-2.1 requested) that
hears the user audio and answers, grounded in an excerpt of a web or PDF document (doc_id).
Index row (<shard>.index.jsonl): key, batch, shard, source, source_pass, spec, hours, n_samples,
sr, flac_bytes, flac_sha256, flac_offset, json_bytes, json_sha256, json_offset, end.
| JSON field | Meaning |
|---|---|
key, doc_id |
dialogue id (16 hex digits) and the grounding document |
sr, n_samples |
sample rate and length; equal to the FLAC's |
card |
the scenario given to the simulator: persona, accent, style, situation, environment (a tag only; the audio is clean), topic, hint (first sentence of the document), voices, target words per second and speed per speaker |
voices |
user and assistant voice names |
events[] |
every piece of audio placed on the timeline, in order of placement. ch (user / assistant), kind, start / end in samples, text |
events[].kind |
utterance; utterance_cut (stopped early because the other speaker barges in); barge_in (starts inside the other speaker's turn and takes the floor); backchannel (short listener token inside the other speaker's turn) |
events[].text |
user: the simulator's text. assistant and backchannel: the speech model's own transcript. For a cut event this is the full planned text while the audio stops early |
events[].spoken_transcript |
user events: the speech model's transcript of what it said |
events[].cut |
onset and stop in samples from the event start: where the interruption begins and where the audio ends |
events[].turn, skel_fto_s, rendered_fto_s, skel_frac |
skeleton batches: index of the skeleton turn, its gap to the previous turn in the real dialogue and as rendered (seconds, negative = overlap), and for a backchannel its position as a fraction of the turn |
models, provenance, timing, spec, seed, time |
generator models, family / modality, placement parameters, generator version, seed, generation time |
source, source_dialogue_id, source_kind, skeleton, source_pass, turns_rendered, bc_bank |
skeleton batches only, see below |
duet |
duet_* batches only: the sampled interaction style (probabilities of barge-in, overlap, backchannel) |
source is duet in the index for duet_* batches (their JSON has no source field) and source_pass is null.
What comes from real dialogues
The skel_* batches copy their turn-taking timing from one real dialogue each, taken from the train splits of
Seamless Interaction (seamless_naturalistic), otoSpeech full-duplex 280 h (otospeech_fd280) and SpokenWOZ
(spokenwoz). The JSON stores this much of the real dialogue and nothing else:
source,source_dialogue_id: the dataset tag and that dataset's dialogue id.skeleton.window_s: start and end, in seconds of the real recording, of the stretch that was used.skeleton.turns[]: per floor turnwho(the role given to that speaker),src_start(start in the real recording, seconds),dur_s,fto_s(gap to the previous turn, negative = overlap),interrupts,onset_frac(where in the previous turn the interruption starts),bc_fracs(positions of the listener's backchannels as fractions of the turn).source_pass: 1 for the first use of a real dialogue, 2 when it is used a second time with another document, persona and voices (skel_p2_*). JSONs written before the field existed lack it; the index fills in 1.
No real audio and no real transcript is included: every word in events was generated. bc_bank is the log of the
generated backchannel clips per speaker (word asked for, attempts, kept or not, what was spoken). The duet_* batches use sampled timing and no real dialogue.
Licences of the timing sources as recorded in our notes: Seamless Interaction CC-BY-NC-4.0, SpokenWOZ CC-BY-NC-4.0, otoSpeech CC-BY-4.0. Confirm them before any use beyond internal research.
The generated words can repeat public contact details from the grounding document (an e-mail address or phone number
on a web page), in card.hint and in events[].text.
Applying the ASR check
Every event was transcribed with Qwen3-ASR-1.7B and compared with its script. A scored turn is a non-backchannel event
with at least 3 script words. qc/speech_qc.jsonl has one row per checked dialogue:
| Field | Meaning |
|---|---|
key, batch, source, hours |
the dialogue |
dialogue_wer |
word errors over script words, summed over the scored turns |
n_turns_scored, n_turns_wer_gt_0.5 |
scored turns, and how many of them have WER above 0.5 |
drop, drop_reasons |
drop is true when dialogue_wer > 0.1 (dialogue_wer_gt_0.1) or two or more scored turns have WER > 0.5 (two_turns_gt_0.5) |
flagged_eids |
indices into the JSON's events of scored turns with WER > 0.5: mask these turns in training when the dialogue is kept |
qc_run |
the ASR run the row comes from |
Three cases per packed dialogue:
- row with
droptrue: leave the dialogue out. - row with
dropfalse: use it, masking the events inflagged_eids. - no row (listed in
qc/unchecked.txt): the ASR check has not reached it. It is neither dropped nor cleared. Wait for a laterspeech_qc.jsonl, or use it knowing the share of checked dialogues that were dropped (qc/stats.json).
The shards are packed before the check finishes, so decisions are not baked into the tars; always apply the newest
speech_qc.jsonl.
import json
qc = {r["key"]: r for r in map(json.loads, open("qc/speech_qc.jsonl"))}
def use(key): # -> None (leave out), or the set of event indices to mask
r = qc.get(key)
if r is None:
return None # unchecked: decide your own policy
return None if r["drop"] else set(r["flagged_eids"])
Packs (optional)
packs/<family>/<arm>.tar holds one token-pack directory as written by our mixing code (meta.json, train.bin,
val*.bin). meta.json records the tokenizer's sha256; the tokenizer file is not in this folder, and the paths inside
meta.json are paths on the generating server. Packs built from real transcripts (real arms) are not included.
- Downloads last month
- 1,161