ByteCraft: Generating Video Games and Animations Through Bytes

1. Introduction

Current methods for video game generation. While a lot of research is done on generating videos [1,2,3], very little effort is done on the video game front. Current state-of-the-art models for generating video games focus mainly on controllable video generation [4,5,6,7,8,9], which effectively consists of walking simulators with a few seconds of memory and increasingly blurry images over time. More recently, Large Language Models (LLMs) [10,11,12,13,14] have started being used to generate video games through code. This approach is promising, but generating complex games requires many rounds of human-AI interactions and user-provided graphics and audio assets.

Figure 1. Screenshots of files generated by ByteCraft.

Figure 1. Screenshots of files generated by ByteCraft.

A promising direction: directly generating bytes. A promising new line of direction is the generation of bytes from various types of files (e.g., images, videos, text, programs, etc.) using language models [15,16,17,18,18]. By staying in the byte world, one can generate any type of file found on a computer.

Why is byte generation not widespread? This direction has received very little attention for many reasons. First, simple files can take an enormous amount of bytes, leading to extremely large context lengths. For example, a simple 1Mb file requires 1 million byte tokens. Second, most modalities that people care about have specific neural network architecture and methods that have been developed and iterated over many years by researchers to make them excellent (e.g., images using 2D convolutions). Treating these modalities as bytes is challenging and may require many improvements in order to beat existing methods.

What are the problems associated with using bytes and how do we deal with them? The main difficulties when generating bytes of games and animations are 1) scaling (due to high-context length and limited data) and 2) overfitting. To handle longer context length than byte-level models, we tokenize bytes into 108K tokens, leading to approximately 2.29 bytes per token (with some extreme cases at 4-5 bytes per token). With a 32K sequence length, we can generate games of around 73 KB in size. To reduce the risks of overfitting, we produced multiple prompts per sequence of bytes.

The first-of-a-kind. In this work, we make the first attempt at building a generator of Small Web Format (SWF) [19] games and animations through bytes. SWF is a complex multi-modal format containing images, videos, code, fonts, and more data types. A single incorrectly placed byte could break the generated file. We are the first to tackle such a challenging task. We do so by fine-tuning an LLM (Qwen2.5-7B) to generate bytes conditional on a text prompt describing the game/animation, making our approach fully end-to-end. Our model, ByteCraft, was trained on limited resources (4 GPUs) for many months. It can generate up to 32K bytes (games/animations smaller than 73KB on average, with the largest we have seen at around 140KB).

Figure 2. Checkered patterns in motion generated in different colors by ByteCraft.

Figure 2. Checkered patterns in motion generated in different colors by ByteCraft.

2. ByteCraft

2.1. Architecture

We need to be able to condition on both text description and bytes. Since no model exists for generating bytes of video games and animations, this part has to be learned. LLMs are particularly good with unstructured text prompts.

We considered the following open-source LLMs as base model: Llama-3.1-8B [20], Ministral-8B [21], Qwen2-VL-7B [22], and Qwen2.5-7B [23]. In preliminary experiments, we found Qwen2.5-7B to be the best; thus, we used it as our base model.

2.2. Tokenization

We used Byte Pair Encoding (BPE) [24] to encode the bytes of video games and animations into 108K new tokens containing on average 2.29 bytes per token (going up to 4-5 bytes per token in rare cases). This allowed us to scale to larger games than would be possible with byte-level generation. These new tokens were added to the tokens of the pre-trained model.

2.3. Data Augmentation on Prompts

Using Qwen2.5-7B, we generated multiple prompts per sequence of bytes in order to produce more diversity and reduce risks of overfitting.

2.4. Training

We fine-tuned ByteCraft with progressively bigger context length (4K, 8K, 32K) using AdamW [25,26]. For the early stages of training, we used the Muon optimizer [27], which accelerated training. Modern techniques such as Fully Sharded Data Parallel(FSDP) [28] and fused kernels [29] were used to accelerate training and reduce memory cost. Training took around 4 months in total using 4 GPUs. The model was trained until it reached a cross-entropy loss of 0.15 (Perplexity (PPL) of 1.16).

2.5. Usage

The user provides a text prompt to describe the video game or animation that they want to be generated. It can be written in any language. We provide some prompt examples in Figure 3. Once the prompt is given, ByteCraft generates k SWF files per prompt, where k is set by the user. These files can then be opened with the Ruffle player [30].

Using vLLM v0.7.3 [31], the model can simultaneously generate 25 files at 32K context length in around 10 minutes on a single A100 with 80Gb memory.

3. Results

We tested ByteCraft qualitatively and quantitatively after generating 250 files from 250 prompts from a held-out set.

For quantitative measures, we 1) calculate the maximum Jaro-Winkler string similarity [32] between the generated file and training files ("Max similarity"), 2) the Jaro-Winkler string similarity [32] between the generated file and the true file with the associated prompt ("Truth similarity"), and 3) the percentage of "fully broken" files as determined by verifying its header and metadata; if parsing fails, the file is likely invalid.

Since this is the first attempt at generating such files, assessing the quality of the files is tricky. We thus rely on qualitative measures by manually opening each file using the Ruffle player [30] (Nightly 2025-03-14). We categorize games that are not "fully broken" as i) "blank canvas" (with a flat background color), ii) "stuck on a loading screen", iii) "showing/hearing something" (this is subjective; it means that something is shown or heard and it is not a loading screen or blank canvas), iv) "Fully Working" (the file is working and playable from a quick assessment).

The results are shown in Table 1. We see that most games are broken, but a few show something visually/audibly interesting, and a tiny percentage functions properly. On average, around 21% of the bytes in the files are novel, as determined by the maximum Jaro-Winkler similarity. The results show that the model can learn some patterns about bytes enough to produce some semi-working or working files.

4. Potential Future Improvements

ByteCraft is a first attempt at generating open-web games and animations through bytes. However, it generates many broken files. In this section, we propose some improvements that could improve the performance of this method.

4.1. Scaling

ByteCraft was trained on extremely limited resources (4 GPUs over 4 months). Given large-scale resources, ByteCraft could improve in performance and be extended to generate larger games at higher context length.

4.2. Reinforcement Learning

To reduce the number of broken files, one could i) generate many files from various prompts with the current model, ii) have an automatic visual evaluator (possibly using a visual LLM such as the latest Qwen2.5-VL [33]) from a screenshot giving a reward between 0 and 1 for broken to fully working, and iii) use reinforcement learning to push the model toward generating valid working files or more aligned with the prompt.

4.3. Test-time Compute

Test-time scaling methods [34,35] have the potential to increase quality and diversity of the generated samples at inference without retraining.

4.4. Better Generalization on Small Data Using Data Augmentations on Bytes

Data augmentation is currently applied only to prompts: we generate a different prompt for each sequence of bytes. However, to truly enable the model to understand and generalize given a limited amount of training examples, one could i) randomly permute the order in which file components are stored (e.g., image 1, font1, code1, image2 → image2, image1, code1, font1). Other data augmentations are also possible, such as ii) having an AI rewrite the programming language code differently or iii) adding noise to the image assets contained in the file or changing their format (e.g., PNG [36]→ JPEG [37]). The difficulty with all the data augmentation strategies proposed is that they require a good understanding of the file format, as one would need to determine where each asset starts and ends with their format and possibly where they are referred to in the code contained in the file. Ideally, this could be inferred directly from bytes, but it is possible that this would instead require expensive reverse engineering.

5. Conclusions

We built ByteCraft, the first generator of games and animations through bytes. Contrary to existing approaches, ByteCraft is fully end-to-end. Given our limited resources (4 GPUs), it is limited to small files (32K context-length, around 73KB on average, but can be as big as 140KB). ByteCraft will require more scaling and could be extended to more types of games, including compressed versions of larger video games. This approach of generating files from text is not limited to video games and animations; it could be used to generate any kind of file end-to-end on a computer.

Parallel with early molecule generation. A parallel exists between ByteCraft and autoregressive molecule generation. Molecules can be represented as SMILES strings [38] and their context length is generally small (around 10-250 tokens without BPE). We show below some of the progress of molecule generation over time on the Zinc-250K dataset [39]:

• GVAE [40]: 0.7% valid molecules (← ByteCraft is here)
• CVAE [41]: 7.2% valid molecules
• RVAE [42]: 34.9% valid molecules
• GFVAE [43], STGG [44], and many others: 100% valid molecules, but not necessarily realistic and synthesizable
• STGG+AL [45]: 100% valid molecules with high synthesizability and out-of-distribution properties (← a future ByteCraft version could be here)

In our case, we tackle the much harder problem of autoregressively generating SWF files with up to 32K tokens. Currently, some of the generated files are partially working (4.8-12%) or fully working (0.4-1.2%). We are thus at step 1, the equivalent of GVAE for molecule generation in 2016. Our end goal is generating 100% valid files with high novelty. We propose some potential directions of improvements in Section 4 on how to get there. We hope this crazy project inspires researchers and hobbyists toward the lofty goal of generating games through bytes.