←Back to Beyond Market Intelligence
•5 min read•from Machine Learning

Generating Bad Apple autonomously from a single initial state using a tiny recurrent dynamical system (417k params) [P]

Our take

Researchers have achieved autonomous generation of the iconic "Bad Apple" animation using a remarkably compact recurrent dynamical system—a mere 417,000 parameters. This innovative approach, detailed on GitHub (SEBADA321/BadAppleRNN), moves beyond static memorization, enabling the system to generate the full 6,500-frame video from a single initial state. Key to this success were techniques like learned latent teacher tables and state perturbation noise, ensuring stability across extended sequences.
Generating Bad Apple autonomously from a single initial state using a tiny recurrent dynamical system (417k params) [P]

The recent demonstration of generating the entirety of the "Bad Apple" animation – a notoriously challenging benchmark for video compression – autonomously from a single initial state using a remarkably small recurrent dynamical system is a significant achievement. It builds upon earlier work, such as Improved compression of Bad Apple into a Neural Network, which explored SIREN networks for implicit memorization of the video as a coordinate function. However, this new approach, detailed by SEBADA321, represents a leap forward by eschewing explicit timestamp inputs and instead training a network to learn the continuous temporal flow within a latent space. This allows the model to generate the full 6,573-frame video without needing to be told when each frame should appear, showcasing a move towards truly autonomous generative capabilities, a concept explored more broadly in Variational Autoencoders (VAEs) Explained: From Theory to ELBO and the Reparameterization Trick, which highlights the theoretical foundations of generative modeling. The efficiency of this system, boasting a mere 417,000 parameters and achieving over 200 frames per second on an RTX 4080, underscores the potential for resource-constrained AI applications.

The ingenuity of this project extends beyond the core architecture. The training process employed a series of clever techniques to overcome the inherent challenges of learning long-term dependencies in recurrent networks. The use of "Learned Latent Teacher Tables" to enable parallel segment training, the progressive "Rollout Horizon Curriculum," and the novel "Second-Difference Acceleration Regularization" all played crucial roles in stabilizing training and preventing the model from collapsing into a brittle, 1D trajectory. These innovations highlight a shift in focus from simply increasing model size to carefully crafting training methodologies that promote stability and generalization. The author’s candid acknowledgement of the training process's somewhat ad-hoc nature, and the reliance on AI assistance for writing the post and README, adds a layer of refreshing transparency to the research, further reinforcing the exploratory nature of the work. The finding that the checkpoint with the lowest training loss didn’t necessarily produce the best autonomous generation results is a particularly insightful observation about the complexities of evaluating generative models.

The implications of this work reach beyond simply recreating a classic animation. It demonstrates a powerful new approach to video generation, one that prioritizes learning the underlying dynamics of a system rather than explicitly memorizing individual frames. This has implications for applications ranging from generating realistic simulations to creating interactive art experiences. The model’s architecture, notably the absence of skip connections, normalization layers, or attention mechanisms, is also noteworthy, suggesting that surprisingly simple architectures can achieve remarkable results when coupled with appropriate training techniques. The author’s intention to focus on optimizing the decoder, recognizing it as a limiting factor, signals potential for even greater improvements in quality and efficiency. The success of compressing "Bad Apple" into a neural network, as previously demonstrated in I Compressed Bad Apple into a 3MB Neural Network, continues to inspire innovative approaches to data representation and generative AI.

Looking ahead, it will be fascinating to see how this approach scales to more complex and longer videos. The current success with "Bad Apple" represents a proof of concept, but the real challenge lies in applying these techniques to generate content with greater diversity and realism. Can similar recurrent dynamical systems be trained to model the complexities of human movement, natural phenomena, or even abstract concepts? Furthermore, exploring the potential of alternative architectures and training regimes – particularly those that incorporate more sophisticated forms of feedback and control – could unlock even greater levels of autonomy and creativity in generative AI. The question remains: how far can we push the boundaries of what's possible with surprisingly small and elegantly designed recurrent networks?

Generating Bad Apple autonomously from a single initial state using a tiny recurrent dynamical system (417k params) [P]

A few weeks ago, I saw this post where the author trained a SIREN MLP to implicitly memorize Bad Apple as a coordinate function: (t, y, x) to pixel.

That got me curious about a slightly different formulation: instead of handing the network a timestamp t, could a small recurrent dynamical system (RNN-ish) learn the continuous temporal flow in latent space and generate the entire ~6,500-frame full resolution video autonomously from a single initial condition (h_0, c_0)?

The code, weights, and analysis tools with rollout scripts, plots, and standalone models are shared here: GitHub: SEBADA321/BadAppleRNN.

Architecture & Inference Footprint

At inference time, the system receives no timestamp inputs and evaluates in a closed loop:

(h_t, c_t) -> Recurrent Transition (CTF) -> (h_{t+1}, c_{t+1}) | h_t -> Frame Decoder -> 384x512 Grayscale Frame
  • Latent Dimension: 64-D for h_t (decoded) and 64-D for c_t (internal memory manifold to separate visually similar frames at different timestamps).
  • Recurrent Transition (ctf): 4-gate LSTM-style recurrence with orthogonal initialization (16,640 parameters, 65 KB).
  • Frame Decoder (fd): 4-stage bilinear upsampling with depthwise-separable convolutions (400,361 parameters, 1.56 MB).
  • Initial State: A single pair of 64-dim vectors (h_0, c_0) (128 floats, 0.5 KB).
  • Total Inference Model: 417,129 parameters (~1.60 MB in FP32).
  • Runtime Performance: >200 FPS on an RTX 4080, with ~17.2 MB peak active VRAM during decoding.

Training an autonomous system across ~6,573 steps from t=0 directly was probably computationally unstable due to vanishing/exploding gradients and compounding error. The training setup I was circling around used several targeted techniques:

  • Learned Latent Teacher Tables: During training, I optimize a pair of tables h_table[t] and c_table[t] alongside the model. This allows parallel segment training starting at arbitrary timestamps over a finite horizon K. These tables are scaffolding and are discarded entirely at inference.
  • Rollout Horizon Curriculum: I started training with K = 2 and progressively doubled the rollout length (K = 2 -> 4 -> 8 -> 16 -> 32 -> 64 -> 128 -> 256 -> 512). Each horizon doubling produced a characteristic jump in loss before the transition function adapted to the longer trajectory.
  • State Perturbation Noise (sigma = 0.005): To prevent the model from learning a brittle 1D line that diverges under small numerical errors, Gaussian noise was added to the state before passing it into the transition function (z_hat_{t+1} = F(z_t + epsilon)), while evaluating the loss against the clean target. This encourages the recurrent map to contract small deviations back toward the trajectory.
  • Second-Difference Acceleration Regularization: Penalizing velocity (||h_{t+1} - h_t||) risks collapsing the trajectory. Instead, I penalized discrete acceleration (jitter) via second differences: ||h_{t+2} - 2h_{t+1} + h_t||_2^2. This enforces smooth trajectories without penalizing motion.
  • Optimizer & Momentum Management: I used AdamW (1x10^-5) for the decoder/tables and Muon (0.005, momentum 0.95) for the recurrent weights. To prevent accumulated momentum from acting as stale inertia when K doubled, momentum buffers were scaled by 0.2 every 10 epochs starting from the epoch 500.
  • Chunked Decoding: To handle long horizons at K = 512 without overflowing VRAM during training, the decoder was evaluated in temporal chunks of 32 frames.

Some interesting things

  1. The model could unroll the full 6.5k sequence even if it was, technically, trained on up to 512 frames. So that was a success.
  2. Training loss vs. autonomous rollout: The checkpoint with the lowest numerical training loss was not necessarily the best at autonomous generation. Because K changes across the curriculum, raw training losses are not directly comparable across stages, and short-horizon teacher-forced agreement does not guarantee long-horizon stability.
  3. Dynamical stability over parameter scale: The challenge was not increasing parameter count (the recurrent transition is only 16k params), but conditioning the dynamics through noise injection and acceleration penalties so error doesn't compound over thousands of recurrent steps.
  4. I need to improve the decoder a lot, I was mostly focused in getting the recurrent part right, and training was slow. Now that I gave gotten a successful result I will focus more into optimizing the CNN decoder.
  5. There are no skip connections nor normalization, which was interesting too. Obviously no attention either since I wanted to keep it simple.I also didn't want to use Truncated BPTT.

Not completely scientific, since I was doing some changes mid run or many at once, which makes it kinda not clear what contributed more. I used 'AI' to help with writting the post and README. Part of the code was also generated that way, but the architecture is what I came up with on my own and from a previous project too. There are probably many parts to improve too, so glad to get some feedback!

submitted by /u/SEBADA321
[link] [comments]

Read on the original site

Open the publisher's page for the full experience

View original article →

Tagged with

#Bad Apple#Recurrent Dynamical System#RNN#LSTM#SIREN MLP#Autonomous Generation#Latent Space#Frame Decoder#Pixel Generation#Rollout Horizon#Curriculum Learning#Vanishing Gradients#Exploding Gradients#State Perturbation Noise#Second-Difference Acceleration#AdamW#Muon Optimizer#Grayscale Frame#CTF (Recurrent Transition)#Initial State (h_0, c_0)