From Prompt to Token traced the text-LLM loop: symbols → integers → vectors → (linear algebra) → one random draw → integers → symbols. That is one point in a much larger design space. The useful way to see the rest is along four questions:

  • Input representation — how the world becomes numbers.
  • Where symbols and numbers cross — the boundaries from that post. Sometimes zero times.
  • Generation mechanism — autoregressive, denoising, message-passing, state-recurrence, or a single forward pass.
  • Output representation — and crucially, not always back to language.

Here is the landscape on one page:

A table of model families as input to mechanism to output rows: text LLM, diffusion, vision transformer, speech recognition, neural field, graph neural network, and embedding models, colour-coded by whether each stage is discrete symbols or continuous numbers.

Still language — but not word-tokens

  • Byte / character models (ByT5, CANINE, MEGABYTE, Byte Latent Transformer) read raw UTF-8 bytes — no tokenizer, a 256-value "vocabulary." Same round-trip as an LLM, but the input boundary is bytes, not learned tokens. (Why that boundary matters at all is the subject of The Tokenizer Tax.)
  • State-space models (Mamba, S4) still take a sequence, but the mechanism is not attention over discrete positions — it is a continuous internal state evolving step by step, in linear time.

Continuous signals in — the input is never discrete

For a spectrogram or an image patch, the "symbol → number" boundary at the input does not exist: the input is already numbers.

  • Speech-to-text (Whisper): a waveform becomes a log-mel spectrogram — a continuous 2-D array — and only the output is text tokens.
  • Audio / music LMs (AudioLM, MusicGen, EnCodec): audio becomes learned discrete acoustic codes via a vector-quantized codec, an LM runs over those, and the output is audio you hear — not language.
  • Vision transformers (ViT): an image is cut into 16×16 patches and linearly projected into vectors — continuous, never tokenized — for a label, caption, or features.
  • Image generation (diffusion): the "input" is literally Gaussian noise plus a conditioning vector, and the image is produced by iterative denoising — no autoregression and no tokens at all. That mechanism is worth its own read: The Denoising Engine.

Structured inputs — no sequence at all

  • Graph neural networks: the input is a graph (nodes, edges, features); the mechanism is message-passing between neighbors; the output is node labels, link predictions, or molecule properties. This is how drug-discovery and fraud-graph models work.
  • Tabular models (gradient-boosted trees, TabTransformer): a row of numeric and categorical features in, a number or class out. The language framing never applies.
  • Point clouds and sets (PointNet, DeepSets): unordered 3-D points or sets in, embeddings or labels out.

No tokenization anywhere — pure continuous math

The family closest to SVAC's own work never lets symbols and numbers trade sides:

  • Neural fields / PINNs / NeRF: the input is raw continuous coordinates (x, y, z, t); the output is a continuous field value — a PDE solution, or a density and color you render into a 3-D scene. Numbers in, numbers out, no symbols ever. (See What a PINN Is and Is Not.)
  • Operator learning (DeepONet, Fourier Neural Operator): the input is a whole function or field and the output is another function — a map between infinite-dimensional spaces, used to stand in for a PDE solver.

Output isn't language — representation and action

  • Embedding models (CLIP, sentence-transformers): text or image in, a single vector out. The output is numbers consumed by a search index; it never returns to human language.
  • RL, control, world models, and vision-language-action robotics: the input is environment state (pixels, sensor readings, proprioception); the output is actions an agent takes. The loop closes in the world, not in text.

The one-line version

The text-LLM round trip — discrete symbols in, autoregressive discrete output, detokenize, language — is the special case where input and output are both human language. Drop that one constraint and everything else falls out: continuous-in (audio, vision, coordinates), structured-in (graphs, tables), continuous-out (images, fields, actions, embeddings), and mechanisms that replace the token loop entirely. Knowing which corner of the zoo a system lives in tells you what it can be good at — and where it can go wrong. That is the whole reason to understand the machine you are using.