Paul Harrald’s Post

Sometimes data visualisation stops you in your tracks... As part of my exploration of the Yale/UNC-CH Geophysical Waveform Inversion Kaggle competition, I’ve been studying how machine learning models can recover subsurface velocity fields from synthetic seismic waveform data. The dataset consists of thousands of examples of simulated wave propagation, each represented as a 2D array of traces — amplitude over time across multiple receivers — paired with a corresponding 2D ground truth velocity map. The task is to learn a mapping from waveform observations to velocity structure: an inverse problem that is inherently non-local, underdetermined, and sensitive to small features in the data. In investigating the capacity of transformer-based encoders to model this problem, I produced the following visualization: a t-SNE projection of 250 latent embeddings, one for each downsampled time step, output by the attention encoder before decoding. What emerged was a striking spiral-shaped manifold — a smooth, continuous embedding where early time steps (shown in purple) evolve gracefully into later ones (yellow). Despite minimal or even no training on the downstream task, the model appears to have self-organized the temporal input into a latent space with topological coherence. This suggests that it is already distinguishing signal features — likely wavefronts, reflections, or arrival phases — in a structured, learnable way. That is, while the attention encoder is only minimally trained, this visualization likely reflects the intrinsic structure of the input waveforms themselves — not a learned representation in the strict sense, but a revealing alignment between data regularity and model inductive bias. That the model even in this early stage organizes the temporal input into such a structured manifold suggests a promising fit between the architecture and the physics of the domain. This kind of emergent structure is a strong signal that attention mechanisms are particularly well-suited to waveform inversion. Unlike convolutional models, which operate locally and isotropically, attention layers allow each voxel or time step to condition on global context. This enables the model to learn non-local causal dependencies — exactly what’s needed in a physical system where an event at time t may depend on multiple reflections, diffractions, or source-receiver geometries. Ultimately, I suspect that attention architectures — especially when coupled with cross-attention to learnable spatial grids — will provide a more natural and physically grounded basis for high-resolution seismic inversion.