mlsanityData to 7 Oct 2026

Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP

arXiv 2109.03777 ↗First posted 8 Sep 2021cs.CL cs.IR cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Graph neural networks have triggered a resurgence of graph-based text classification methods, defining today's state of the art. We show that a wide multi-layer perceptron (MLP) using a Bag-of-Words (BoW) outperforms the recent graph-based models TextGCN and HeteGCN in an inductive text classification setting and is comparable with HyperGAT. Moreover, we fine-tune a sequence-based BERT and a lightweight DistilBERT model, which both outperform all state-of-the-art models. These results question the importance of synthetic graphs used in modern text classifiers. In terms of efficiency, DistilBERT is still twice as large as our BoW-based wide MLP, while graph-based models like TextGCN require setting up an O(N2) graph, where N is the vocabulary plus corpus size. Finally, since Transformers need to compute O(L2) attention weights with sequence length L, the MLP models show higher training and inference speeds on datasets with long sequences.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 46 of its 73 references that are in the map. First posted on arXiv 8 Sep 2021.

Builds on

46 of 73 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2017Graph Attention Networkscited by 8,854
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
34 more
  1. 2019Simplifying Graph Convolutional Networkscited by 1,470
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  5. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  8. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  9. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  10. 2021Pay Attention to MLPscited by 367
  11. 2018Graph Convolutional Networks for Text Classificationcited by 345
  12. 2019Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approachescited by 181
  13. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  14. 2014A Convolutional Neural Network for Modelling Sentencescited by 163
  15. 2019Text Classification Algorithms: A Surveycited by 163
  16. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  17. 2020Deep Learning Based Text Classification: A Comprehensive Reviewcited by 140
  18. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  19. 2015PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networkscited by 96
  20. 2021GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddingscited by 90
  21. 2020Be More with Less: Hypergraph Attention Networks for Inductive Text Classificationcited by 73
  22. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  23. 2021Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNetcited by 61
  24. 2018Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanismscited by 53
  25. 2020Tensor Graph Convolutional Networks for Text Classificationcited by 49
  26. 2020Comparing BERT against traditional machine learning text classificationcited by 45
  27. 2016Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Poolingcited by 39
  28. 2020HeteGCN: Heterogeneous Graph Convolutional Networks for Text Classificationcited by 21
  29. 2019Graph Star Net for Generalized Multi-Task Learningcited by 11
  30. 2021A Comparison of Deep-Learning Methods for Analysing and Predicting Business Processescited by 11
  31. 2018Using Deep Learning for Title-Based Semantic Subject Indexing to Reach Competitive Performance to Full-Textcited by 10
  32. 2019Efficient Path Prediction for Semi-Supervised and Weakly Supervised Hierarchical Text Classificationcited by 5
  33. 2017Using Titles vs. Full-text as Source for Automated Semantic Document Annotationcited by 4
  34. 2020Combine Convolution with Recurrent Networks for Text Classificationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 2; 2024: 6; 2025: 4; 2026: 1.

17 papers.

Sort:

By area:

  1. 2022Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasetscited by 11
  2. 2023Bag of States: A Non-sequential Approach to Video-based Engagement Measurementcited by 5
  3. 2024Graph with Sequence: Broad-Range Semantic Modeling for Fake News Detectioncited by 4
  4. 2024Vulnerability Detection in Ethereum Smart Contracts via Machine Learning: A Qualitative Analysiscited by 2
  5. 2025GLaMoR: Consistency Checking of OWL Ontologies using Graph Language Modelscited by 2
  6. 2025Guarded Query Routing for Large Language Modelscited by 2
  7. 2022Are We Really Making Much Progress in Text Classification? A Comparative Reviewcited by 1
  8. 2022Memorization of Named Entities in Fine-tuned BERT Modelscited by 1
  9. 2024MLPs Compass: What is learned when MLPs are combined with PLMs?cited by 1
  10. 2024When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantizationcited by 1
  11. 2024Efficient Continual Learning for Small Language Models with a Discrete Key-Value Bottleneckcited by 1
  12. 2025Automated Design Optimization via Strategic Search with Large Language Modelscited by 1
5 more
  1. 2022Event and Entity Extraction from Generated Video Captionscited by 0
  2. 2023Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification Using Graph Neural Networks?cited by 0
  3. 2024Graph Neural Networks on Discriminative Graphs of Wordscited by 0
  4. 2025Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representationscited by 0
  5. 2026Influence of Prompt Engineering on Small Language Models for Guarded Query Routingcited by 0