Step 0399: Improve Metrics: Tile ID Diversity and Playable State

📋 Executive Summary

Step 0398 revealed that the "tilemap 100%" metric was misleading: the tilemap was full of non-zero bytes, but they were alltile ID 0x00(without diversity). This step improves the detection metrics to includetile ID diversityandplayable state.

Lesson Learned from Step 0398

Problem: Counting bytes != 0x00 can be misleading if all values ​​are the same. A "full" tilemap does not imply playable state if all tiles are the same ID.

Solution: Verifydiversity(count unique tile IDs) and combine multiple metrics to determine playable state.

Key Results

• Zelda DX: Tilemap 100% but only1 unique tile ID(all 0x00) →gameplay_state=NO ✅
• Tetris DX: 69-256 unique tile IDs → gameplay_state=YESfrom Frame 720 ✅
• No regressions: Correct detection in ROMs that worked before ✅

🔧 Hardware Concept

Why Tile ID Diversity Matters

According toPan Docs - Tile Maps, the tilemap (0x9800-0x9BFF or 0x9C00-0x9FFF) is an array of 32×32 bytes (1024 tiles) where each byte is aTile ID(0-255) which references a tile in VRAM.

Misleading vs Correct Metrics

Playable State vs Initialization State

a game inplayable statehas:

1. TileData with data: ≥200 non-zero bytes at 0x8000-0x97FF (tiles loaded from ROM)
2. Tilemap Diversity: ≥10 unique tile IDs (not just initialization to 0x00)
3. Complete tiles: ≥10 tiles with ≥8 non-zero bytes (real tiles, not single bytes)

a game ininitialization stateyou can have "full" tilemap but no diversity:

• All tiles point to 0x00 (initial tilemap cleanup)
• Empty VRAM (tiles not yet loaded from ROM)
• Example: Zelda DX in Frame 1-1200 from Step 0398

Reference: Pan Docs - Background & Tiles

The diversity of tiles is essential for real rendering. A game that loads its main screen typically uses 50-256 unique tile IDs to represent:

• Static background (e.g. sky, grass, walls)
• Interactive elements (e.g. doors, items)
• Text and UI (ex: life bar, points)
• Characters and sprites (referenced by tilemap in some games)

⚙️ Implementation

1. Helper: count_unique_tile_ids_in_tilemap()

Files: src/core/cpp/PPU.hpp, src/core/cpp/PPU.cpp

Aim: Count how many unique tile IDs there are in the tilemap (diversity).

Implementation:

int PPU::count_unique_tile_ids_in_tilemap() const {
    if (mmu_ == nullptr) {
        return 0;
    }
    
    uint8_t lcdc = mmu_->read(IO_LCDC);
    // Tilemap active according to LCDC bit 3
    uint16_t vram_offset = (lcdc & 0x08) ? 0x1C00 : 0x1800;  // 0x9C00 or 0x9800
    
    // Use array of booleans to track unique tile IDs (0-255)
    bool tile_ids_seen[256] = {false};
    int unique_count = 0;
    
    // Read full tilemap (32×32 = 1024 bytes)
    for (uint16_t offset = 0; offset< 0x0400; offset++) {
        uint8_t tile_id = mmu_->read_vram_bank(0, vram_offset + offset);
        if (!tile_ids_seen[tile_id]) {
            tile_ids_seen[tile_id] = true;
            unique_count++;
        }
    }
    
    return unique_count;
}

Concept: Unlike counting bytes != 0x00, this counts how many tile IDsdifferentthere is. A tilemap with all tiles = 0x00 has diversity = 1 (only a unique ID).

2. Helper: is_gameplay_state()

Files: src/core/cpp/PPU.hpp, src/core/cpp/PPU.cpp

Aim: Determine if the game is in playable state based on combined metrics.

Implementation:

bool PPU::is_gameplay_state() const {
    // Check TileData
    int tiledata_nonzero = count_vram_nonzero_bank0_tiledata();
    if (tiledata_nonzero< 200) {
        return false;  // VRAM vacía o casi vacía
    }
    
    // Verificar diversidad de tilemap
    int unique_tile_ids = count_unique_tile_ids_in_tilemap();
    if (unique_tile_ids < 10) {
        return false;  // Tilemap sin diversidad (estado de inicialización)
    }
    
    // Verificar tiles completos
    int complete_tiles = count_complete_nonempty_tiles();
    if (complete_tiles < 10) {
        return false;  // Pocos tiles completos (datos incompletos)
    }
    
    return true;  // Todas las métricas cumplen → estado jugable
}

Concept: Combines three independent metrics. If only 1-2 criteria are met, it is probably initialization or transition state.

3. Update of vram_has_tiles_ with Diversity Criterion

Archive: src/core/cpp/PPU.cpp(functionrender_scanline(), LY=0)

Change: Include diversity criteria in VRAM detection.

Previous logic (Step 0397):

// Only checked non-zero or complete tile bytes
vram_has_tiles_ = (tiledata_nonzero >= 200) || (complete_tiles >= 10);

Improved logic (Step 0399):

// Check diversity of tile IDs in tilemap
int unique_tile_ids = count_unique_tile_ids_in_tilemap();

// Improved triple criteria:
// 1. TileData has data (>= 200 bytes) AND full tiles (>= 10)
// 2. Tilemap has diversity (>= 5 unique tile IDs)
bool has_tiles_data = (tiledata_nonzero >= 200) || (complete_tiles >= 10);
bool has_tilemap_diversity = (unique_tile_ids >= 5);

bool old_vram_has_tiles = vram_has_tiles_;
vram_has_tiles_ = has_tiles_data && has_tilemap_diversity;

// Log when state changes (max 10 changes)
if (vram_has_tiles_ != old_vram_has_tiles) {
    printf("[VRAM-STATE-CHANGE] Frame %llu | has_tiles: %d -> %d | "
           "TileData: %d/6144 (%.1f%%) | Complete: %d | Unique IDs: %d\n",
           frame_counter_ + 1, old_vram_has_tiles ? 1 : 0, vram_has_tiles_ ? 1:0,
           tiledata_nonzero, (tiledata_nonzero * 100.0/6144),
           complete_tiles, unique_tile_ids);
}

Concept: Now requiresbothcriteria: data in VRAMANDdiversity in tilemap. This prevents false positives ("full" tilemap with no real data).

4. Periodic Metrics Update [VRAM-REGIONS]

Archive: src/core/cpp/PPU.cpp(functionrender_scanline(), every 120 frames)

Change: Include diversity of tile IDs and playable state in periodic logs.

Previous log (Step 0397):

[VRAM-REGIONS] Frame 1080 | tiledata_nonzero=... | tilemap_nonzero=... | 
               complete_tiles=... | vbk=... | vram_is_empty=... | vram_has_tiles=...

Improved log (Step 0399):

[VRAM-REGIONS] Frame 1080 | tiledata_nonzero=0/6144 (0.0%) | 
               tilemap_nonzero=2048/2048 (100.0%) | unique_tile_ids=1/256 | 
               complete_tiles=0/384 (0.0%) | vbk=0 | gameplay_state=NO

Concept: The fieldunique_tile_idsIt immediately reveals if there is diversity. The fieldgameplay_statesummarizes the result of the combination of metrics.

✅ Tests and Verification

Compilation

Command executed:

cd /media/fabini/8CD1-4C30/ViboyColor
python3 setup.py build_ext --inplace

Result: ✅ Successful build without errors

Extended Test: Zelda DX (60 seconds)

Command executed:

timeout 60s python3 main.py roms/Oro.gbc > logs/step0399_zelda_dx_extended.log 2>&1

Results (metrics every 120 frames):

[VRAM-REGIONS] Frame 120 | tiledata_nonzero=0/6144 (0.0%) | tilemap_nonzero=2048/2048 (100.0%) | unique_tile_ids=1/256 | complete_tiles=0/384 (0.0%) | vbk=0 | gameplay_state=NO
[VRAM-REGIONS] Frame 240 | tiledata_nonzero=0/6144 (0.0%) | tilemap_nonzero=2048/2048 (100.0%) | unique_tile_ids=1/256 | complete_tiles=0/384 (0.0%) | vbk=0 | gameplay_state=NO
...
[VRAM-REGIONS] Frame 1200 | tiledata_nonzero=0/6144 (0.0%) | tilemap_nonzero=2048/2048 (100.0%) | unique_tile_ids=1/256 | complete_tiles=0/384 (0.0%) | vbk=0 | gameplay_state=NO

Analysis:

• ✅ Tilemap 100% butonly 1 tile unique ID(all 0x00)
• ✅ gameplay_state=NOcorrectly detected for 1200 frames
• ✅ Confirm that Zelda DX is in initialization state, not playable

Regression Test: Tetris DX (30 seconds)

Command executed:

timeout 30s python3 main.py roms/tetris_dx.gbc > logs/step0399_tetris_dx.log 2>&1

Key results:

[VRAM-STATE-CHANGE] Frame 678 | has_tiles: 0 -> 1 | TileData: 2938/6144 (47.8%) | Complete: 221 | Unique IDs: 69
[VRAM-REGIONS] Frame 720 | tiledata_nonzero=1416/6144 (23.0%) | tilemap_nonzero=259/2048 (12.6%) | unique_tile_ids=256/256 | complete_tiles=98/384 (25.5%) | vbk=0 | gameplay_state=YES
[VRAM-STATE-CHANGE] Frame 735 | has_tiles: 1 -> 0 | TileData: 0/6144 (0.0%) | Complete: 0 | Unique IDs: 1
[VRAM-STATE-CHANGE] Frame 745 | has_tiles: 0 -> 1 | TileData: 3479/6144 (56.6%) | Complete: 253 | Unique IDs: 185
[VRAM-REGIONS] Frame 840 | tiledata_nonzero=3479/6144 (56.6%) | tilemap_nonzero=2012/2048 (98.2%) | unique_tile_ids=185/256 | complete_tiles=253/384 (65.9%) | vbk=0 | gameplay_state=YES
[VRAM-REGIONS] Frame 960 | tiledata_nonzero=3479/6144 (56.6%) | tilemap_nonzero=2012/2048 (98.2%) | unique_tile_ids=185/256 | complete_tiles=253/384 (65.9%) | vbk=0 | gameplay_state=YES

Analysis:

• ✅ Frame 678: Transitionhas_tiles: 0 -> 1with69 unique tile IDs
• ✅ Frame 720:gameplay_state=YESwith256 unique tile IDs(maximum diversity)
• ✅ Frame 735-745: Temporary transition (possible screen clear during menu mode)
• ✅Frame 840+:gameplay_state=YESstable with185 unique tile IDs
• ✅ No regressions: correct playable state detection

Regression Test: Pokemon Red (30 seconds)

Command executed:

timeout 30s python3 main.py roms/pkmn.gb > logs/step0399_pokemon_red.log 2>&1

Result: ✅ No state changes detected (expected behavior, does not reach playable state in 30s or does not have significant transitions)

Native Validation

✅ C++ module compiled and working correctly

The new helperscount_unique_tile_ids_in_tilemap()andis_gameplay_state()They run natively in C++ with no Python overhead.

📊 Results and Conclusions

Metric Comparison: Step 0397 vs Step 0399

Improvements Achieved

1. Diversity Metric: unique_tile_idscorrectly detects initialization status (Zelda DX: 1/256)
2. Playable Status: gameplay_statesummarizes combination of metrics (TileData + diversity + complete tiles)
3. No False Positives: Zelda DX no longer reports "tilemap 100%" as if it were playable
4. No Regressions: Tetris DX still detects playable state correctly (Frame 720, 256 unique IDs)

Lessons Learned

Simple Metrics Can Be Misleading:

• ❌ Counting bytes != 0x00 does not guarantee diversity
• ✅ Counting unique values ​​reveals the true state

Playable state requires multiple criteria:

• TileData with data (tiles loaded from ROM)
• Tilemap with diversity (not just initialization to 0x00)
• Complete tiles (not just single bytes)

Next Steps

With the improved metrics, we can now:

1. Accurately detect when a game reaches a playable state
2. Identify state transitions (e.g. menu → gameplay)
3. Investigate why Zelda DX does not load tiles from ROM (possible emulation or timing problem)

📁 Modified Files

• src/core/cpp/PPU.hpp- Helper declarationscount_unique_tile_ids_in_tilemap()andis_gameplay_state()
• src/core/cpp/PPU.cpp- Implementation of helpers and updating of VRAM detection logic
• logs/step0399_zelda_dx_extended.log- Zelda DX extended log (60 seconds)
• logs/step0399_tetris_dx.log- Tetris DX regression log (30 seconds)
• logs/step0399_pokemon_red.log- Pokemon Red regression log (30 seconds)