new theory of cortex?
a new paper on diffusion models for image generation tackles the interesting question of why diffusion models don’t just vomit training examples, citing inductive biases leading to creativity,
https://arxiv.org/pdf/2412.20292
TODO unpack the results more
Interestingly, their analytic results are confined to pure diffusion models, but they consider how and why models augmented with self-attention do a slightly better job, by introducing non local interactions.
the complementary roles of local and non local interactions is reminiscent of similar considerations in cortical connectivity in the brain: the local, statistically anisotropic connections and complemented by long range, non local connections (including time delays) which may play a similar role as attention.