Key Concepts of GNNs

Graph Representation Learning

Graph Neural Networks (GNNs) operate on graph-structured data, where entities are represented as nodes and relationships as edges. The core idea is to learn low-dimensional embeddings for nodes, edges, or entire graphs that capture structural and feature-based information. Given a graph G = (V, E), where V is the set of nodes and E the set of edges, GNNs leverage message passing to propagate information across the graph.

Here, h_v^(k) is the embedding of node v at layer k, σ is a non-linear activation function, W^(k) and B^(k) are learnable weight matrices, and AGGREGATE is a permutation-invariant function (e.g., sum, mean, or max) over the neighbors 𝒩(v) of node v.

Message Passing Framework

The message passing paradigm consists of three steps: (1) message computation, where each node generates messages for its neighbors; (2) message aggregation, where each node combines received messages; and (3) node update, where each node updates its embedding based on aggregated messages. This framework is iterative, allowing information to propagate across multiple hops.

Here, ϕ, ρ, and ψ are differentiable functions, and e_u→v represents edge features.

Graph Convolutional Networks (GCNs)

A widely used GNN variant, GCNs simplify message passing by employing a normalized sum aggregation with symmetric adjacency matrix normalization. The layer-wise propagation rule is:

where à = A + I is the adjacency matrix with self-loops, D̃ is the diagonal degree matrix of Ã, and H^(k) is the node feature matrix at layer k.

Attention Mechanisms in GNNs

Graph Attention Networks (GATs) introduce attention weights to dynamically prioritize messages from different neighbors. The attention coefficient α_uv between nodes u and v is computed as:

where a is a learnable attention vector, W is a weight matrix, and ∥ denotes concatenation.

Inductive vs. Transductive Learning

GNNs can operate in transductive settings (fixed graph, all nodes observed during training) or inductive settings (generalize to unseen nodes or graphs). Inductive models, such as GraphSAGE, sample neighborhoods and aggregate features dynamically, enabling scalability to large graphs.

Practical Considerations

• Over-smoothing: Deep GNNs may suffer from node embeddings becoming indistinguishable due to excessive message passing.
• Scalability: Sampling techniques (e.g., neighborhood sampling, subgraph sampling) are critical for large-scale graphs.
• Heterophily: Traditional GNNs assume homophily (connected nodes are similar); adaptations are needed for heterophilous graphs.

Message Passing in GNNs

The core operation in graph neural networks (GNNs) is message passing, where node representations are iteratively updated by aggregating information from neighboring nodes. This process enables GNNs to capture both local and global structural patterns in graph-structured data.

Mathematical Formulation

At layer l, the message passing operation for node v can be decomposed into three key steps:

1. Message Construction: Each neighbor u ∈ N(v) computes a message:

where φ is a message function (typically an MLP), h are node features, and e are edge features.

2. Message Aggregation: Messages from all neighbors are combined:

Common aggregation operators ⊕ include sum, mean, or max pooling.

3. Node Update: The target node's representation is updated:

where ψ is an update function (often another MLP).

Practical Considerations for Citation Networks

When applying message passing to citation datasets like Cora or PubMed:

• Directionality: Citation graphs are directed but often treated as undirected to ensure information flow between citing and cited papers.
• Edge Features: Publication years or citation counts can be incorporated via e_uv.
• Self-loops: Crucial for preserving original node features through skip connections.

Advanced Variants

Several refined message passing schemes have shown improved performance:

where α_uv are attention weights (as in GATs) or learnable edge-type parameters (as in R-GCNs). For heterogeneous citation networks with multiple node types (papers, authors, venues), meta-paths can guide message passing routes.

Computational Efficiency

The sparse nature of citation networks enables efficient matrix-based implementations:

where  is the normalized adjacency matrix with self-loops. For large graphs, neighborhood sampling or subgraph batching becomes essential - techniques like GraphSAGE's fixed-size sampling or Cluster-GCN's graph partitioning maintain scalability while preserving the majority of the message passing benefits.

Popular GNN Architectures

Graph Convolutional Networks (GCNs)

The Graph Convolutional Network (GCN) introduced by Kipf and Welling in 2017 is a foundational architecture for semi-supervised node classification. It operates via layer-wise propagation rules that aggregate features from neighboring nodes. The forward propagation rule for a single layer is:

where H^(l) represents node embeddings at layer l, W^(l) is the trainable weight matrix, σ is a nonlinear activation function, Ã = A + I is the adjacency matrix with self-loops, and D̃ is the diagonal degree matrix of Ã. This symmetric normalization ensures stable gradient propagation during training.

Graph Attention Networks (GATs)

Graph Attention Networks replace the fixed aggregation weights of GCNs with learnable attention mechanisms. Each node computes attention coefficients over its neighbors:

where α_ij is the attention coefficient between nodes i and j, W is a shared linear transformation, and a is a learnable attention vector. Multi-head attention extends this by concatenating or averaging K independent attention heads.

GraphSAGE

GraphSAGE (Sample and AggregatE) generalizes GCNs to inductive settings by learning aggregator functions that combine features from a node's local neighborhood. The key innovation is its sampling-based approach:

• Neighbor sampling: Fixed-size uniform sampling from each node's k-hop neighborhood
• Aggregator functions: Mean, LSTM, or pooling operators combine sampled features

The update rule for node v at layer k is:

Gated Graph Neural Networks (GGNNs)

GGNNs incorporate gated recurrent units (GRUs) to propagate information across graph edges. The message passing mechanism resembles that of recurrent neural networks:

where W_edge is an edge-type specific weight matrix. This architecture excels in tasks requiring multi-step reasoning over graph structures.

Graph Isomorphism Networks (GINs)

GINs provide theoretical guarantees for distinguishing graph structures by using injective multiset functions. The update rule for node v at layer k is:

where MLP denotes a multi-layer perceptron and ε is a learnable parameter. GINs achieve maximum discriminative power among GNNs in the Weisfeiler-Lehman graph isomorphism test framework.

Practical Considerations for Citation Networks

When applying these architectures to citation datasets like Cora or PubMed:

• GCNs perform well for transductive tasks with fixed graphs
• GATs capture asymmetric citation relationships via attention
• GraphSAGE handles dynamic graphs where new papers may be added
• GINs are preferred when structural similarity detection is crucial

Batch normalization and dropout (applied to node features rather than edges) typically improve performance. For large graphs, sampling-based methods like GraphSAGE or Cluster-GCN become necessary to manage memory constraints.

Overview of Citation Datasets (Cora, Citeseer, PubMed)

Citation datasets are fundamental benchmarks for evaluating graph neural networks (GNNs) in semi-supervised node classification tasks. These datasets model academic papers as nodes and citations as edges, with node features derived from text and labels representing paper topics. Three widely used citation datasets are Cora, Citeseer, and PubMed, each offering distinct characteristics in terms of scale, sparsity, and feature dimensionality.

Cora Dataset

The Cora dataset consists of 2,708 machine learning papers categorized into 7 classes. Each paper is represented as a node, with 5,429 citation links forming directed edges. Node features are binary word vectors indicating the presence of 1,433 unique words from a dictionary. The graph exhibits a power-law degree distribution, with most nodes having few connections while a small number act as hubs. The dataset is typically split into 140 training nodes, 500 validation nodes, and 1,000 test nodes.

This extreme sparsity challenges GNNs to effectively propagate information across the graph while avoiding over-smoothing.

Citeseer Dataset

Citeseer contains 3,312 scientific publications classified into 6 categories, with 4,732 citation edges. The feature vectors span 3,703 dimensions, representing word occurrences from a larger vocabulary than Cora. Notably, Citeseer includes many isolated nodes (no citations) and about 12% of citation links are missing (only present in one direction). The standard split uses 120 nodes for training, 500 for validation, and 1,000 for testing.

PubMed Dataset

PubMed is significantly larger, comprising 19,717 medical research papers on diabetes from the PubMed database, grouped into 3 classes. With 44,338 citation edges, it offers a denser connectivity pattern (sparsity ≈ 0.9998) compared to Cora and Citeseer. Each node is represented by a TF-IDF weighted word vector of 500 dimensions, derived from the paper abstracts. The standard split allocates 60 nodes per class for training (180 total), with 500 validation and 1,000 test nodes.

Comparative Analysis

The varying properties of these datasets enable comprehensive evaluation of GNN architectures across different scenarios: Cora tests performance on small graphs with high feature dimensionality, Citeseer evaluates robustness to incomplete graph structures, and PubMed assesses scalability to larger networks.

Feature Engineering Considerations

For GNN training, the raw features often undergo preprocessing:

• Normalization: TF-IDF vectors in PubMed are L2-normalized
• Dimensionality Reduction: PCA is sometimes applied to Citeseer's high-dimensional features
• Feature Augmentation: Some approaches concatenate node degrees as additional features

The citation graphs are typically treated as undirected for GNN processing, with adjacency matrices normalized using the symmetric transformation:

where A is the adjacency matrix and D is the degree matrix. This normalization helps stabilize gradient propagation during training.

Dataset Characteristics and Preprocessing

Citation datasets such as Cora, PubMed, and CiteSeer are widely used benchmarks for evaluating Graph Neural Networks (GNNs). These datasets represent academic papers as nodes and citations as edges, forming directed or undirected graphs. Each node contains a feature vector (typically bag-of-words representations of the paper's abstract) and a label indicating its research topic.

Graph Structure and Feature Representation

The adjacency matrix A of a citation graph is sparse, with A_ij = 1 if paper i cites paper j. For undirected graphs, A is symmetric. Node features are represented as a matrix X ∈ ℝ^n×d, where n is the number of nodes and d is the feature dimension. The label vector y ∈ ℤⁿ contains class indices for each paper.

Common Citation Datasets

• Cora: 2,708 machine learning papers with 5,429 citations, 7 classes, and 1,433-dimensional binary word features.
• PubMed: 19,717 biomedical papers with 44,338 citations, 3 classes, and 500-dimensional TF-IDF features.
• CiteSeer: 3,312 computer science papers with 4,732 citations, 6 classes, and 3,703-dimensional word features.

Preprocessing Pipeline

Standard preprocessing steps include:

1. Feature Normalization: Scale features to zero mean and unit variance or apply L2 normalization:

2. Graph Normalization: Apply symmetric normalization to the adjacency matrix for better spectral properties:

where D is the degree matrix with D_ii = ∑_jA_ij.

3. Train/Val/Test Splits: Standard splits use 20 nodes per class for training, 500 for validation, and 1,000 for testing, ensuring each split contains all classes.

Handling Class Imbalance

For datasets with uneven class distributions, apply class-weighted loss functions during training:

where w_c = 1/f_c and f_c is the frequency of class c.

Edge Sampling and Subgraph Extraction

For large graphs, use neighborhood sampling or random walk-based subgraph extraction to enable mini-batch training. The GraphSAGE approach samples a fixed-size neighborhood:

where d(u,v) is the shortest path distance between nodes u and v.

Node and Edge Features in Citation Graphs

Citation graphs represent scholarly articles as nodes and citations as directed edges. The feature representation of nodes and edges is critical for graph neural networks (GNNs) to capture semantic and structural patterns. Node features typically encode document content, while edge features may represent citation context or metadata.

Node Feature Engineering

In citation networks like Cora, PubMed, or CiteSeer, node features are commonly derived from:

• Bag-of-Words (BoW): Binary or TF-IDF weighted term vectors constructed from paper abstracts or titles.
• Embeddings: Pre-trained word vectors (e.g., Word2Vec, GloVe) averaged across document tokens.
• Graph-Aware Features: Centrality measures (degree, betweenness) or positional encodings (Laplacian eigenvectors).

where X_i is the feature vector for node i, E_k denotes word embeddings, and σ is a normalization function.

Edge Feature Representation

Edge features in citation graphs often encode:

• Directionality: Binary indicators for forward/backward citations.
• Temporal Context: Time delta between publication dates.
• Citation Importance: Weighted by section (e.g., introduction vs. related work).

For a citation from paper i to j, edge features may be constructed as:

where ⊕ denotes concatenation and Δt_ij is the publication year difference.

Heterogeneous Feature Integration

Advanced GNN architectures like Graph Attention Networks (GATs) dynamically reweight features during aggregation:

where W is a learnable weight matrix and a is an attention vector. This allows the model to focus on semantically relevant citations.

Data Splitting Strategies for Citation Graphs

Citation graphs, such as Cora, PubMed, and CiteSeer, exhibit unique structural properties that necessitate specialized data splitting strategies. Unlike traditional tabular or image datasets, citation graphs contain interconnected nodes where edges represent citation relationships, introducing dependencies that complicate random splitting.

Challenges in Graph Data Splitting

Randomly splitting nodes into training, validation, and test sets risks data leakage due to message passing between connected nodes. If a test node is connected to a training node, the model may indirectly access test information during training, leading to overoptimistic performance estimates. This violates the fundamental assumption of independent and identically distributed (i.i.d.) data.

Common Splitting Strategies

Transductive Splitting

In transductive learning, all graph nodes are visible during training, but only a subset of labels are available. The standard split for Cora uses:

• 140 nodes (5.2%) for training
• 500 nodes (18.5%) for validation
• 1000 nodes (37.2%) for testing

This approach preserves the full graph structure while masking labels, allowing GNNs to leverage topological information without direct access to test labels.

Inductive Splitting

For inductive scenarios where the test graph may differ from the training graph, researchers employ:

• Time-based splitting: Nodes are split by publication date, simulating real-world chronological constraints
• Community detection: Clusters nodes by modularity before splitting to maintain community structure

Advanced Techniques

The disjoint split method enforces strict separation between splits:

• No edges connect training and test nodes
• 2-hop neighbors of test nodes are excluded from training
• Requires recomputing adjacency matrices for each split

For large-scale graphs, approximate splitting methods use graph partitioning algorithms like METIS to minimize inter-split edges while maintaining balance:

Practical Considerations

When implementing splits in PyTorch Geometric, the RandomNodeSplit transform provides basic functionality, while custom splitters should:

• Preserve class distribution across splits (stratification)
• Account for node degree distribution
• Handle directed/undirected and weighted edges appropriately

from torch_geometric.transforms import RandomNodeSplit

transform = RandomNodeSplit(
    num_val=0.1,
    num_test=0.2,
    key='y'
)
dataset = dataset.transform(transform)

Recent work on distributionally robust splitting introduces adversarial validation techniques to identify and mitigate potential covariate shift between splits, particularly important for graphs evolving over time.

3.2 Loss Functions and Optimization

Training Graph Neural Networks (GNNs) on citation datasets requires carefully designed loss functions and optimization strategies to handle the semi-supervised nature of the task. The primary objective is to minimize prediction errors for node classification while leveraging the graph structure.

Cross-Entropy Loss for Node Classification

For multi-class node classification tasks, the standard loss function is the cross-entropy loss applied to the labeled nodes. Given a set of labeled nodes V_L with ground truth labels y_i and predicted class probabilities ŷ_i, the loss is computed as:

where C is the number of classes, and y_i,c is a one-hot encoded vector. This formulation penalizes deviations between predicted probabilities and true labels.

Regularization and Graph-Based Loss Terms

To exploit the graph structure, additional regularization terms are often incorporated. A common approach is to enforce smoothness in the learned node representations by minimizing the graph Laplacian regularization term:

where H is the matrix of node embeddings, L is the graph Laplacian, and λ controls the regularization strength. This term encourages connected nodes to have similar embeddings.

Optimization Strategies

Stochastic Gradient Descent (SGD) or its variants (e.g., Adam) are typically used for optimization. However, GNNs pose unique challenges:

• Full-batch vs. mini-batch training: Full-batch training processes the entire graph at once but is memory-intensive. Mini-batch training samples subgraphs but requires careful handling of neighborhood aggregation.
• Learning rate scheduling: Adaptive learning rates (e.g., cosine annealing) help stabilize training, especially for deep GNNs.
• Gradient clipping: Prevents exploding gradients in deep architectures.

Handling Class Imbalance

Citation datasets often exhibit class imbalance. Techniques like weighted cross-entropy or focal loss can be applied:

where γ adjusts the rate at which easy examples are down-weighted.

Advanced Techniques: Contrastive Learning

Recent work incorporates contrastive loss to improve representation learning. For a node v_i, positive samples are its neighbors, while negative samples are randomly selected nodes. The contrastive loss is:

where sim(·,·) measures cosine similarity, and τ is a temperature hyperparameter.

3.3 Handling Class Imbalance in Citation Data

Class imbalance is a pervasive issue in citation datasets, where certain paper categories may be significantly underrepresented compared to others. For instance, in the Cora dataset, the "Neural Networks" category might dominate, while "Genetic Algorithms" appears infrequently. This skew biases GNNs toward majority classes, degrading performance on minority classes. Addressing imbalance requires specialized techniques beyond standard cross-entropy loss.

Resampling Strategies

Resampling adjusts the training distribution by either oversampling minority classes or undersampling majority classes. For graph data, these methods must preserve topological structure:

• Node-level oversampling: Duplicates minority-class nodes and their local neighborhoods. The new nodes inherit edges based on a similarity metric:

where h denotes node embeddings and γ controls edge sharpness.

• Graph-aware undersampling: Removes majority-class nodes while maintaining connectivity via betweenness centrality:

Nodes with lowest centrality are pruned first to minimize disruption.

Loss Function Modifications

Reweighting schemes adjust the loss function to emphasize minority classes:

Common weighting strategies include:

• Inverse frequency: w_c ∝ 1/n_c
• Focal loss: Downweights well-classified examples via w_c = (1 - p_c)^γ

For graph data, w_c can incorporate node degree to account for structural importance:

Graph-Specific Approaches

Recent methods exploit graph topology for imbalance mitigation:

• TopoBalance: Generates synthetic minority nodes in sparsely connected regions using graph autoencoders.
• GraphSMOTE: Interpolates node features and constructs edges via k-NN in embedding space.
• ImGAGN: Adversarial generation of minority-class graphs with topology preservation.

These methods typically outperform classical resampling by 5-12% in macro-F1 on benchmarks like Cora and PubMed.

Evaluation Metrics

Accuracy becomes meaningless under imbalance. Preferred metrics include:

• Macro-F1: Unweighted mean of per-class F1 scores
• G-mean: Geometric mean of class-wise recall
• Class-Weighted Accuracy: 1/C ∑_c w_c · accuracy_c

For citation graphs, the macro-F1 is particularly informative as it equally values all research topics regardless of prevalence.

4.1 Common Evaluation Metrics

Evaluating graph neural networks (GNNs) on citation datasets requires metrics that capture both node classification accuracy and the structural integrity of predictions. Unlike traditional machine learning tasks, GNNs must account for relational dependencies between nodes, necessitating specialized evaluation approaches.

Node Classification Metrics

For multi-class node classification tasks, standard supervised learning metrics apply, but with adaptations for graph-structured data:

• Accuracy: Measures the fraction of correctly classified nodes. While straightforward, it can be misleading in class-imbalanced datasets.
• F1 Score: The harmonic mean of precision and recall, particularly useful for imbalanced classes. The macro-averaged F1 is commonly reported for multi-class scenarios.
• ROC-AUC: Area under the receiver operating characteristic curve, which evaluates the model's ability to rank positive instances higher than negatives across all thresholds.

Edge Prediction Metrics

When evaluating link prediction or edge classification tasks, different metrics are employed:

• Average Precision (AP): Summarizes the precision-recall curve as a weighted mean of precisions at each threshold.
• Area Under Curve (AUC): Probability that a random positive edge receives a higher score than a random negative edge.
• Hit@k: Fraction of cases where a true edge appears in the top-k predicted edges.

Graph-Level Metrics

For tasks requiring graph-level predictions (e.g., graph classification), additional metrics are used:

• Graph Classification Accuracy: Simple ratio of correctly classified graphs.
• Mean Average Precision (mAP): Extension of AP to multi-class scenarios, averaging across classes.
• Normalized Mutual Information (NMI): Measures the quality of cluster assignments compared to ground truth.

Specialized GNN Metrics

Citation graph analysis often employs domain-specific evaluation approaches:

• Citation Prediction Accuracy: Measures how well the model predicts future citations in temporal citation networks.
• Rank Correlation Metrics: Kendall's τ or Spearman's ρ to evaluate ranking quality in citation recommendation tasks.
• Novelty Detection: Evaluates the model's ability to identify previously uncited but relevant papers.

When benchmarking GNNs on standard citation datasets like Cora, Citeseer, or PubMed, researchers typically report multiple metrics to provide a comprehensive performance assessment. The choice of metrics should align with the specific downstream application, whether it's academic search, recommendation systems, or knowledge graph completion.

Benchmarking Against Baselines

When evaluating Graph Neural Networks (GNNs) on citation datasets such as Cora, PubMed, or CiteSeer, rigorous benchmarking against established baselines is essential to validate performance improvements. Baselines typically include traditional machine learning methods, shallow graph embeddings, and simpler neural architectures.

Traditional Machine Learning Baselines

Classical approaches like logistic regression, support vector machines (SVMs), and random forests operate on hand-engineered features derived from the graph structure. For citation networks, common features include:

• Bag-of-words representations of node attributes
• Graph statistics (degree centrality, PageRank)
• Co-citation or bibliographic coupling measures

These methods serve as important sanity checks since they represent the performance achievable without leveraging graph structure or deep learning.

Shallow Graph Embedding Methods

Shallow embedding techniques like DeepWalk, node2vec, and LINE provide node representations by optimizing objective functions that preserve structural properties:

where f is the embedding function, σ is the sigmoid function, and P_n is the negative sampling distribution. These methods are computationally efficient but lack the ability to incorporate node features or learn task-specific representations.

Simple Neural Architectures

Multi-layer perceptrons (MLPs) applied directly to node features provide a crucial baseline for assessing whether graph structure actually improves performance. The MLP objective is:

where L is the number of layers and x_v are the input features of node v. Comparing GNNs against MLPs reveals whether message passing provides meaningful gains over feature-only approaches.

Evaluation Metrics and Protocols

Standard evaluation practices for citation networks include:

• Fixed splits: Using the standard 20/30/50 train/val/test splits from Yang et al.
• Multiple runs: Reporting mean and standard deviation over 10+ random initializations
• Metrics: Accuracy for multi-class classification, often with macro-F1 for imbalanced cases

The table below shows typical baseline performance ranges on Cora:

Advanced Considerations

For rigorous benchmarking, several subtle factors must be controlled:

• Feature preprocessing: Normalization of input features affects MLP and GNN performance differently
• Hyperparameter tuning: Equal tuning budgets must be allocated to all methods
• Graph perturbations: Testing robustness to noisy or missing edges

The PyTorch Geometric library provides standardized implementations of these baselines, ensuring fair comparisons through shared data loading and evaluation protocols.

Interpreting Model Predictions

Feature Attribution in GNNs

Understanding why a GNN makes specific predictions requires analyzing feature attribution. Unlike traditional models, GNNs propagate information through graph structures, making interpretation non-trivial. Feature attribution methods like GNNExplainer identify influential nodes and edges by optimizing a mask over the input graph:

where M is a learnable mask, G denotes element-wise multiplication, and I measures mutual information between the prediction Y and the masked graph. The L1 penalty encourages sparsity.

Semi-Supervised Learning with GNNs

5.2 Scalability and Large-Scale Citation Graphs

5.3 Incorporating Node Attributes and Metadata

class AttributeAwareGCN(MessagePassing):
    def __init__(self, in_channels, out_channels):
        super().__init__(aggr='add')
        self.lin = Linear(in_channels, out_channels)
        
    def forward(self, x, edge_index):
        # x: [num_nodes, in_channels]
        return self.propagate(edge_index, x=x)
    
    def message(self, x_j):
        return x_j
    
    def update(self, aggr_out, x):
        # Concatenate node features with aggregated messages
        return torch.cat([x, aggr_out], dim=-1)

6.1 Key Research Papers on GNNs and Citation Data

6.2 Open-Source Implementations and Libraries

6.3 Recommended Books and Tutorials