histlearn/notas-comunidade-ptbr

Community Notes BR: Enriched Portuguese Dataset for NLP and Text Mining

Community Notes BR is a large-scale Portuguese-language corpus of X Community Notes enriched for Natural Language Processing (PLN), text mining, and empirical research on collaborative content moderation. Built from the official Community Notes public data dump, it integrates note-level consensus metadata, neural topic and macrotheme representations, hybrid named-entity recognition (NER), source-domain extraction, locally re-estimated Matrix Factorization bridging scores, an optional Universal Dependencies (UD) morphosyntactic layer, and comprehensive quality-control tables.

To comply with platform terms of service and privacy standards, the dataset redistributes only official Community Notes texts and public identifiers (noteId, tweetId, and anonymized note_author_participant_id), omitting original post text. Researchers can hydrate source posts via tweetId under their own API access and legal conditions.

• Associated preprint: Davi Machado da Rocha, Community Notes no Brasil: consenso, polarização e aceitabilidade algorítmica (SciELO Preprints, 2026). https://doi.org/10.1590/SciELOPreprints.15996
• Upstream source: Official Community Notes public data (snapshot 2026-04-07) — https://communitynotes.x.com/guide/en/under-the-hood/download-data
• Temporal coverage: 2021-10-13 to 2026-04-05
• Scale at a glance:
  • 142,448 Portuguese Community Notes (fastText lid.176 confidence $\ge 0.80$)
  • 90,822 unique target tweet IDs and 25,483 anonymized note-author IDs
  • 421,912 validated, non-overlapping entity mentions across 21 domain-specific categories (100% exact offset match against the normalized note text)
  • 898 fine-grained BERTopic clusters organized into 38 hierarchical macrothemes (covering 94,889 notes, plus 47,162 HDBSCAN noise/outlier notes preserved with topic_id = -1)
  • 292,275 UD sentences and 5,599,679 morphosyntactically annotated tokens (UDPipe Bosque 2.17)

Resumo em português

Este repositório disponibiliza um corpus em larga escala das Community Notes (Notas da Comunidade) em língua portuguesa, enriquecido em múltiplas camadas para pesquisas em Processamento de Língua Natural (PLN), mineração de texto, checagem de fatos e moderação colaborativa. O recurso reúne 142.448 notas (selecionadas por critério estritamente linguístico via fastText lid.176 com confiança $\ge 0,80$), 90.822 tweets distintos, 421.912 menções de entidades nomeadas distribuídas em 21 categorias sem sobreposição de spans, 898 tópicos específicos agrupados em 38 macrotemas, parâmetros de pontuação reestimados pelo algoritmo oficial de fatoração matricial sobre o subgrafo lusófono e uma camada sintática opcional no padrão Universal Dependencies (UDPipe Bosque).

O conjunto foi projetado com foco em rastreabilidade e reprodutibilidade: todos os spans da camada atual de entidades possuem correspondência exata de caracteres na base textual de extração, e o diretório quality_audit/ inclui tanto tabelas determinísticas de integridade referencial quanto os dados e scripts de uma auditoria humana estratificada de precisão e coerência temática.

Repository Structure and Files

File or folder	Rows	Size	Granularity and analytical role
notes_pt.parquet	142,448	42.4 MB	Primary note-level table (one row per Portuguese Community Note)
entities.parquet	421,912	57.2 MB	Validated, non-overlapping entity mentions (SHA-256: c106cfdd...ee4afb591b)
topic_metadata.parquet	898	78.3 KB	Metadata, keywords, and structured descriptions for each valid BERTopic cluster
macrotheme_metadata.parquet	38	9.4 KB	High-level thematic clusters aggregating the 898 fine-grained topics
DATA_DICTIONARY.md	-	-	Complete column-level data dictionary, types, and controlled vocabularies
taxonomy.json	-	5.0 KB	Entity taxonomy definitions, confidence thresholds, and regex aliases
syntax_ud_bosque/sentences/*.parquet	292,275	22.8 MB	Sentence-level spans and IDs over summary_clean
syntax_ud_bosque/tokens/*.parquet	5,599,679	89.7 MB	Token-level CoNLL-U morphosyntactic and dependency annotations
syntax_ud_bosque/notes_udpipe_bosque.conllu.zst	-	42.3 MB	Compressed CoNLL-U export with note-level metadata headers
quality_audit/entities_removed_2026-08.parquet	88,440	11.0 MB	Archive of spans removed during August 2026 structural deduplication (R1/R2)
quality_audit/	-	90.7 MB	Deterministic integrity, summary_clean offset mapping, and UD alignment tables
quality_audit/human_audit/	-	0.6 MB	Stratified human evaluation data, topic-intruder answer keys, and evaluation scripts
cross_modal_validation/	-	64.6 KB	Empirical cross-tabulations of macrothemes against entity types and cited domains
source_code/	-	2.3 MB	Reproducible pipeline notebooks (ETL, filtering, topics, NER, MF scoring, UD)

Primary join key: noteId links notes_pt.parquet, entities.parquet, syntax_ud_bosque/, and quality_audit/. The Dataset Viewer displays the primary analytical tables and the UD syntax layer; auxiliary audit archives remain directly downloadable from the repository tree.

Licensing summary: The core dataset (notes_pt.parquet, entities.parquet, topic_metadata.parquet, macrotheme_metadata.parquet, and audit tables) is dedicated to the public domain under CC0 1.0 Universal, matching the upstream Community Notes license. The optional syntax_ud_bosque/ directory was produced with the UDPipe portuguese-bosque-ud-2.17-251125 model (CC BY-NC-SA; see syntax_ud_bosque/udpipe_bosque_metadata.json).

Schema Overview

See DATA_DICTIONARY.md for exhaustive column specifications. Key analytical fields are summarized below.

notes_pt.parquet

Column	Description
noteId	Primary key: official Community Notes identifier (SHA-256 of table: 0bec0496...c1772cbc5f).
tweetId	Target post identifier for cross-referencing or external hydration.
note_author_participant_id	Anonymized contributor identifier provided by the platform.
created_at, year_month	UTC creation timestamp and YYYY-MM aggregation key.
summary	Original note text as published in the upstream dump.
summary_clean	Normalized text with URLs stripped, used as input for semantic embeddings and UD parsing.
classification_algoritmo	Author-assigned note category (e.g., MISINFORMED_OR_POTENTIALLY_MISLEADING, NOT_MISLEADING).
idioma, confianca_idioma	fastText lid.176 language code (pt) and confidence score ($\ge 0.80$).
status_final, consenso	Latest platform status and compact class: CRH (Currently Rated Helpful), CRNH (Currently Rated Not Helpful), NMR (Needs More Ratings), or Outro.
n_ratings, n_notas_no_tweet	Total ratings received by the note and total competing notes on the same target post.
coreNoteIntercept, coreNoteFactor1	Helpfulness intercept and polarity factor re-estimated by running the official Matrix Factorization algorithm over the Portuguese rating subgraph (populated for notes meeting the minimum rating threshold).
topic_id, topic_name, contexto_central, categoria_ampla	Fine-grained BERTopic cluster ID and structured LLM descriptors (topic_id = -1 indicates HDBSCAN unclustered outliers).
macrotheme_id, macrotheme_label	Hierarchical macrotheme identifier (38 classes) and descriptive label.
stratum	Sampling stratum indicator (top_notas, bat_signals, aleatorio) for the 4,055-note hydration subset; null otherwise.

entities.parquet

Column	Description
entity_id	Unique mention identifier (<row_id>_<j>), with zero duplicates across all 421,912 rows.
noteId, tweetId	Foreign keys linking each mention to its source note and target post.
texto_entidade	Extracted surface string.
texto_canonico	Normalized form (lowercased, accent-stripped, and mapped through canonical alias dictionaries for major political actors, parties, agencies, and platforms).
tipo_entidade	Canonical entity category (21 domain-specific types; see taxonomy.json).
label_modelo	Raw GLiNER class label or specific deterministic regex rule name.
inicio, fim	Codepoint character offsets anchored to the whitespace-normalized summary text (trim(regexp_replace(summary, '\s+', ' ', 'g'))), with 100% exact recovery across all 421,912 rows.
score	Model confidence score (for GLiNER spans) or deterministic rule indicator (1.0).
fonte_extracao	Provenance layer: gliner or specific regex extractor (regex_url, regex_data, regex_estatistica, regex_valor, regex_lei, regex_processo).
contexto	Local character window surrounding the mention.
papel_no_texto	Functional role in the note: mencao (topical actor/object) or fonte_ou_evidencia (cited domain, norm, date, or statistic).
dominio	Normalized hostname for URL_DOMINIO mentions.
topic_id, macrotheme_id	Denormalized topic and macrotheme IDs inherited from notes_pt.parquet.

Construction Methodology and Design Rationale

1. Upstream ETL and Medallion Architecture

The five official Community Notes tables (notes, noteRatings, noteStatusHistory, userEnrollment, batSignals) were ingested into DuckDB with cryptographic hash verification and shard-level schema validation:

• Bronze layer: raw string-typed ingestion preserving upstream records verbatim.
• Silver layer: type casting, timestamp normalization, and snake_case schema standardization.
• Gold layer: analytical joins and Snappy-compressed Parquet exports.

2. High-Precision Language Filtering

Notes were filtered using fastText lid.176.bin applied to summary_clean, retaining records with idioma = pt and confianca_idioma >= 0.80 (142,448 notes). fastText provides deterministic, sub-millisecond inference across hundreds of thousands of short texts, and the conservative 0.80 threshold prioritizes precision over recall to prevent Spanish or English code-mixed notes from distorting downstream topic clusters. Manual inspection of a 300-note sample stratified across confidence deciles confirmed high linguistic purity.

3. Instruction-Guided Topic and Macrotheme Modeling

To cluster notes by the factual dispute under verification rather than surface phrasing or recurring platform formulas ("O post é falso porque..."), document representations were computed with intfloat/multilingual-e5-large-instruct using a task-specific Portuguese instruction prompt.

• Fine-grained clustering: BERTopic was fitted over UMAP + HDBSCAN with hyperparameters selected via Density-Based Clustering Validation (DBCV) grid search, yielding 898 valid topics (94,889 notes) and retaining 47,162 diffuse notes as explicit outliers (topic_id = -1) rather than forcing noisy cluster assignments.
• Grammar-constrained labeling: Each cluster was summarized using Marco-Mini-Instruct under grammar-constrained JSON decoding, ensuring 100% schema-compliant outputs (topic_name, contexto_central, categoria_ampla) from representative notes and c-TF-IDF keywords.
• Hierarchical macrothemes: The 898 topic representations were hierarchically agglomerated into 38 macrothemes, providing a two-level thematic taxonomy suitable for both macro-level discourse analysis and micro-level event tracking.

4. Hybrid Named-Entity Recognition and Structural Deduplication

Community Notes combine informal social-media prose with dense evidentiary citations (links, statutes, court dockets, and statistics). To capture both open-class actors and structured evidentiary markers accurately, the NER pipeline combines two complementary layers over 142,051 valid notes (using a 1,500-character sliding window with 200-character overlap for long notes):

1. Zero-shot neural layer (urchade/gliner_multi-v2.1): extracts 16 semantic entity types tuned to Brazilian public debate (political actors, public figures, institutions, courts, security forces, media outlets, platforms, public programs, and political events) using class-specific confidence thresholds (0.40–0.55).
2. Deterministic regex layer: extracts 6 structured evidentiary patterns (URL_DOMINIO, DATA, ESTATISTICA, VALOR_MONETARIO, LEI_NORMA, PROCESSO_JUDICIAL).
3. Priority overlap resolution and multi-round curation: When a neural span overlaps a deterministic span ($\text{IoU} \ge 0.50$), the deterministic span takes precedence. Eight iterative rounds of lexical and UD/POS-assisted post-processing removed generic role nouns (e.g., "o presidente", "o autor"), resolved systematic class ambiguities, and unified major aliases via canonical dictionaries.

Pipeline Evolution and August 2026 Structural Deduplication (421,912 rows)

During the preparation of the canonical revision (d4f438b3acba66a56b0124f5ebc57e2a48dd76a6), a full-corpus structural audit against the whitespace-normalized summary text identified and eliminated two sources of span redundancy from the previous 510,352-row revision:

• Rule R1 — Removal of GLiNER spans embedded inside URL slugs (88,300 rows): Because extraction runs on the full note text (with URLs intact), the neural model frequently tagged words inside URL paths as second entity mentions (e.g., tagging globo inside https://g1.globo.com/... as VEICULO_MIDIA alongside the existing URL_DOMINIO span). Removing spans contained within URL_DOMINIO boundaries eliminated double-counting — particularly in VEICULO_MIDIA (from 43,292 to 11,170 nominal mentions) and PLATAFORMA_DIGITAL (from 27,889 to 12,835) — while leaving core actor classes such as ATOR_POLITICO virtually unchanged (from 33,617 to 32,045; $-4.7%$).
• Rule R2 — Removal of unrecoverable character offsets (140 rows): Eliminated residual spans whose [inicio, fim] indices did not slice back to texto_entidade.

Pipeline stage	Entity rows	Key transformation
1. Raw hybrid extraction	536,373	Initial GLiNER + regex extraction with sliding window and alias dictionaries
2. Structural & lexical cleanup	521,992	Removal of generic role descriptors, extraction artifacts, and malformed URL tokens
3. Morphosyntactic (UD/POS) filtering	510,352	Removal of 11,640 single-token common-noun false positives via POS/morphology rules
4. Person-category refinement	510,352	Reclassification of 1,282 political figures to ATOR_POLITICO and 126 non-political figures to PESSOA_PUBLICA
5. Canonical structural deduplication (Aug 2026)	421,912	Removal of 88,300 intra-URL duplicates (R1) and 140 invalid offsets (R2); 100% exact offset match and zero span overlaps

All 88,440 removed rows are archived with their removal rule in quality_audit/entities_removed_2026-08.parquet (SHA-256: bcde3c0d...a0f45c9), ensuring full backward reproducibility. Because the canonical layer (421,912 rows) contains zero overlapping spans, it can be projected directly onto flat sequence-labeling schemes such as BIO/BILOU.

5. Subgraph Matrix Factorization Scoring

The official open-source Community Notes Matrix Factorization algorithm was executed over the Portuguese rating subgraph, producing coreNoteIntercept (note helpfulness score) and coreNoteFactor1 (raters' viewpoint bridging factor) for the 72.7% of notes that meet the minimum rating density required by the algorithm. These parameters capture consensus dynamics within the Portuguese-speaking contributor network and complement the categorical consenso state (CRH, CRNH, NMR).

6. Morphosyntactic Annotation (syntax_ud_bosque/)

To support linguistic, stylistic, and syntactic analyses of fact-checking argumentation, summary_clean was parsed with UDPipe using the portuguese-bosque-ud-2.17-251125 model, yielding 292,275 segmented sentences and 5,599,679 tokens annotated with Universal POS tags, morphological features, lemmas, and dependency heads/relations.

Corpus Statistics (Canonical Revision d4f438b3)

Note Consensus Distribution

consenso	Platform status	Notes (n)	Share (%)
NMR	NEEDS_MORE_RATINGS	116,741	82.0%
CRH	CURRENTLY_RATED_HELPFUL	15,857	11.1%
CRNH	CURRENTLY_RATED_NOT_HELPFUL	4,423	3.1%
Outro	Unmapped or transitional	5,427	3.8%

Thematic Coverage

Assignment status	Notes (n)	Share (%)
Assigned to a valid topic (898 topics) and macrotheme (38 macrothemes)	94,889	66.6%
Unclustered HDBSCAN outlier (topic_id = -1)	47,162	33.1%
Outside topic-modeling run (topic_id null)	397	0.3%

Validated Entity Distribution by Type (421,912 mentions)

Across the corpus, 126,527 notes (89.1%) contain at least one extracted entity (mean of 3.33 mentions per annotated note and 2.97 overall), and 116,000 notes (81.7%) include at least one explicit URL citation. Person mentions (PESSOA + ATOR_POLITICO + PESSOA_PUBLICA) total 78,196.

tipo_entidade	Extraction layer	Canonical count (n)	Share of entities (%)	Pre-deduplication count (30865ddf)
URL_DOMINIO	Deterministic (regex)	174,582	41.38%	174,624
PESSOA	Neural (GLiNER)	45,135	10.70%	58,543
LOCAL	Neural (GLiNER)	39,368	9.33%	49,022
ORGANIZACAO	Neural (GLiNER)	36,542	8.66%	46,835
ATOR_POLITICO	Neural (GLiNER)	32,045	7.60%	33,617
ORGAO_PUBLICO	Neural (GLiNER)	13,548	3.21%	16,740
DATA	Deterministic (regex)	13,287	3.15%	13,292
PLATAFORMA_DIGITAL	Neural (GLiNER)	12,835	3.04%	27,889
VEICULO_MIDIA	Neural (GLiNER)	11,170	2.65%	43,292
PARTIDO	Neural (GLiNER)	8,647	2.05%	9,016
ESTATISTICA	Deterministic (regex)	7,514	1.78%	7,519
EVENTO_POLITICO	Neural (GLiNER)	6,262	1.48%	7,347
PROGRAMA_PUBLICO	Neural (GLiNER)	5,120	1.21%	5,732
VALOR_MONETARIO	Deterministic (regex)	4,197	0.99%	4,224
ORGAO_SEGURANCA	Neural (GLiNER)	3,869	0.92%	4,196
ORGAO_JUDICIARIO	Neural (GLiNER)	2,535	0.60%	2,828
LEI_NORMA	Deterministic (regex)	1,783	0.42%	1,789
OPERACAO_POLICIAL	Neural (GLiNER)	1,073	0.25%	1,170
PESSOA_PUBLICA	Neural (GLiNER)	1,016	0.24%	1,159
PROCESSO_JUDICIAL	Hybrid (GLiNER / regex)	790	0.19%	888
FONTE_CITADA	Neural (GLiNER)	594	0.14%	630
Total	All 21 types	421,912	100.00%	510,352

Validation and Quality Audit

The repository provides two complementary validation layers: stratified human evaluation (quality_audit/human_audit/) and full-corpus deterministic integrity audits (quality_audit/).

1. Stratified Human Evaluation (quality_audit/human_audit/)

A stratified sample (seed 42, canonical revision d4f438b3acba66a56b0124f5ebc57e2a48dd76a6) was evaluated in context to measure extraction precision and thematic separability. All raw decisions, sampling manifests, answer keys, and evaluation scripts are published in quality_audit/human_audit/.

Named-Entity Precision (230 notes / 1,016 spans)

To ensure coverage across all 16 neural entity categories, the sampler selected notes from the rarest to the most frequent model type until reaching at least $n \ge 30$ mentions per type (659 GLiNER mentions), and included all co-occurring deterministic spans in those same notes (357 regex mentions, of which 314 are URL_DOMINIO).

Extraction layer	Audited spans (n)	Strict precision	Wilson 95% CI	Error profile
Deterministic layer (regex)	357	99.4%	[98.0%, 99.8%]	Only 2 false positives across 357 spans
Neural model layer (GLiNER)	659	80.6%	[77.4%, 83.4%]	120 over-extractions and 8 type corrections; 0 boundary errors
Combined sample	1,016	87.2%	[85.0%, 89.1%]	Reported separately above due to URL_DOMINIO prevalence in regex

• High reliability in core analytical classes: Precision is highest in the central categories used for political and institutional analyses, including ATOR_POLITICO (100.0%, 114/114), URL_DOMINIO, and PARTIDO, whereas open-ended discourse categories such as FONTE_CITADA (16.1%) exhibit higher over-extraction and can be filtered out when high precision is required.

Topic Separability and Label Coherence (148 topics across volume deciles)

Across 148 topics stratified by note-volume decile, a topic-intruder detection task (identifying 1 intruder note drawn from a distinct macrotheme among 3 true topic notes, against a 25% random baseline) achieved 96.6% accuracy (143/148, Wilson 95% CI [92.3%, 98.5%]; 97.9% excluding 2 abstentions). On a 4-point ordinal scale, the LLM-generated topic labels achieved a mean coherence rating of 3.37 / 4.00, with 85.8% ([79.3%, 90.5%]) rated acceptable (3) or well-described (4).

To reproduce the audit metrics locally from quality_audit/human_audit/:

python analyze_entities.py
python analyze_topics.py gold_topics_cross_macro_single_annotator.json

2. Full-Corpus Deterministic Integrity and Cross-Layer Alignment

• 100% Exact Offset Match on Extraction Base: In entities.parquet (421,912 rows), 100% of character offsets [inicio, fim] match texto_entidade exactly against the whitespace-normalized summary text (which preserves URLs).
• Mapping to URL-Stripped Text (summary_clean) and UD Sentences: Because the UD syntax layer (syntax_ud_bosque/) is parsed over summary_clean (where URLs are stripped), offsets of mentions that appear after a stripped URL shift relative to summary_clean. The diagnostic tables in quality_audit/ (entity_offset_audit.parquet and entity_ud_alignment.parquet, computed on the 510,352-row pre-deduplication table) explicitly map which non-URL mentions remain positionally aligned with summary_clean and UD sentence spans (237,738 entities directly aligned to UD sentences).
• Referential and Token Integrity: 100% of entity rows match their parent note's topic_id and macrotheme_id; 292,149 of 292,275 UD sentences (99.96%) locate in exact sequential order within summary_clean, and 100% of sentences have sequential CoNLL-U token IDs.

Recommended Use Cases

• Collaborative Moderation & Helpfulness Modeling: Training and interpreting tabular, neural, or hybrid classifiers predicting consensus status (CRH vs. CRNH) from textual, topical, entity, and evidentiary features.
• Political Communication & Actor-Network Analysis: Studying co-occurrence networks, citation patterns, and thematic framing around political actors (ATOR_POLITICO, 100% audited sample precision), parties, public institutions, and judicial bodies in Portuguese.
• Source & Citation Ecology in Fact-Checking: Analyzing the 174,582 URL_DOMINIO mentions in conjunction with topic clusters and note helpfulness scores (coreNoteIntercept) to investigate which domains foster cross-partisan consensus.
• Argumentation & Morphosyntactic Analysis: Combining the 5.6M-token UDPipe Bosque layer with consensus labels to study linguistic markers of neutrality, hedging, and evidentiary framing in crowd-sourced fact-checking.
• Distant-Supervision & Domain-Specific NER Research: Inspecting domain-adapted entity extraction over Brazilian social-media text or projecting the non-overlapping 421,912 mentions onto BIO tagging schemes for weakly supervised experiments.

Scope and Methodological Considerations

To support accurate downstream interpretation, users should account for the following design boundaries:

1. Linguistic vs. Geographic Scope: The corpus is defined by language (fastText lid.176 Portuguese confidence $\ge 0.80$), not user geolocation. While entity and topic distributions reflect predominantly Brazilian public debate, the filter captures Portuguese-language notes globally and may exclude very short or heavily code-mixed notes.
2. Algorithmic Consensus vs. Ground-Truth Factuality: CRH (Currently Rated Helpful) and CRNH (Currently Rated Not Helpful) reflect the output of the Community Notes bridging-based consensus algorithm at the 2026-04-07 snapshot, not professional fact-checking verdicts.
3. Subgraph Matrix Factorization Estimates: coreNoteIntercept and coreNoteFactor1 are re-estimated using the official algorithm restricted to the Portuguese rating subgraph (and are null for the 27.3% of notes below the minimum rating threshold). They should be interpreted as subgraph-specific consensus and polarity coordinates rather than X's global production parameters.
4. Audit Scope: The human audit in quality_audit/human_audit/ was conducted by a single annotator (the dataset author) over a type-stratified sample. Consequently, the reported entity figures measure strict sample precision (not corpus-wide prevalence-weighted precision or recall), and single-annotator ratings do not include inter-annotator agreement ($\kappa$).

Reproducibility and Public Interfaces

The source_code/ directory provides seven sanitized Jupyter notebooks covering the complete pipeline:

Notebook	Stage
01_etl_snapshot.ipynb	Upstream TSV ingestion, hash verification, and DuckDB medallion tables
02_portuguese_filter.ipynb	fastText language identification and rating/status history joins
03_topic_modeling_macrothemes.ipynb	multilingual-e5-large-instruct embeddings, BERTopic, and Marco-Mini-Instruct labeling
04_named_entity_recognition.ipynb	Hybrid GLiNER + regex extraction, alias canonicalization, and POS-assisted filtering
05_matrix_factorization_scoring.ipynb	Execution of the official Matrix Factorization algorithm on the Portuguese subgraph
06_helpfulness_classification.ipynb	Helpfulness classification and interpretability experiments (CRH vs. CRNH)
07_udpipe_bosque_enrichment_colab.ipynb	Morphosyntactic annotation pipeline (UDPipe Bosque 2.17)

Interactive web interfaces for exploring the corpus without local setup:

• Project Catalog & Overview: https://communitynotesbr.netlify.app/
• Argument & Entity Explorer: https://explorador-argumentos.netlify.app/

Citation / Como citar

If you use this dataset in your research, please cite both the associated paper and the Hugging Face dataset repository:

@article{Machado_da_Rocha_2026,
  title     = {Community Notes no Brasil: consenso, polarização e aceitabilidade algorítmica},
  url       = {https://doi.org/10.1590/SciELOPreprints.15996},
  doi       = {10.1590/SciELOPreprints.15996},
  publisher = {FapUNIFESP (SciELO)},
  author    = {Machado da Rocha, Davi},
  year      = {2026},
  month     = aug
}

@misc{communitynotesbr2026,
  title        = {Community Notes BR: An NLP-Enriched Portuguese Dataset for Collaborative Misinformation Moderation},
  author       = {Rocha, Davi Machado da},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  url          = {https://huggingface.co/datasets/histlearn/notas-comunidade-ptbr},
  note         = {Canonical revision d4f438b3acba66a56b0124f5ebc57e2a48dd76a6}
}

AI and Automated Tooling Disclosure

This resource incorporates neural and generative models as documented components of its data-enrichment pipeline:

• intfloat/multilingual-e5-large-instruct for instruction-tuned semantic embeddings;
• Marco-Mini-Instruct (with grammar-constrained JSON decoding) for structured topic and macrotheme descriptors;
• urchade/gliner_multi-v2.1 (combined with deterministic regex rules and canonical dictionaries) for named-entity recognition;
• UDPipe (portuguese-bosque-ud-2.17-251125) for morphosyntactic parsing.

Language models also assisted in copy-editing documentation prose. All released tables are deterministic outputs of the versioned scripts in source_code/ and quality_audit/, verified by cryptographic SHA-256 hashes.

Maintainer: Davi Machado da Rocha (Universidade de São Paulo — USP). Upstream source: X Community Notes public data (2026-04-07 snapshot, CC0 1.0).