The Limits of Hierarchical Processing

The conventional model suggests that our visual system is structured like a deep neural network—with early layers detecting edges and colors, middle layers identifying shapes and textures, and deeper layers recognizing objects and scenes. This neat, linear progression feels intuitive, but it fails to explain several remarkable human visual capabilities.

Consider a thought experiment: imagine the letter “A” composed of tiny letter “A”s, which themselves are made of even smaller “A”s, continuing infinitely. A hierarchical system would eventually hit a limit—running out of processing layers to handle deeper levels of recursion. Yet humans effortlessly perceive the pattern at multiple scales simultaneously. We recognize both the large “A” and, upon closer inspection, the smaller “A”s that compose it, regardless of how many levels deep the pattern goes.

This recursive pattern recognition isn’t limited to letters. We see it in our ability to process nested visual structures in nature, art, and design. Whether it’s recognizing a forest, then the trees that comprise it, then the branches, leaves, and cellular structures—our visual system seems to apply the same recognition mechanisms recursively at different scales.

The Recursive Alternative

What if, instead of a hardwired hierarchy, our visual system employs a more flexible, recursive approach? Drawing inspiration from computer science, we might compare the brain to a CPU handling recursive functions. Despite having limited physical depth, a single processor can navigate arbitrarily deep recursive structures using a stack mechanism—pushing and popping information as needed.

Similarly, the brain might use a core set of neural operations that can be dynamically applied at different levels of abstraction. Rather than having dedicated neural circuits for each level of visual complexity, the same neural machinery might process different levels of detail through recurrent connections and attentional mechanisms.

This perspective aligns with several emerging theories in neuroscience:

1. Predictive coding models suggest the brain uses a common operation—minimizing prediction error—throughout the visual system, rather than fundamentally different operations at each level.

2. Recurrent processing theories emphasize that visual processing isn’t just a feedforward cascade but involves extensive feedback and lateral connections, allowing the same neural populations to participate in different levels of processing.

3. Global workspace theories propose that conscious perception involves broadcasting information to a shared neural workspace where the same mechanisms can operate on different levels of abstraction.

Implications for Artificial Intelligence

If this recursive model better reflects human vision, it suggests we might need to rethink the purely hierarchical architectures that dominate computer vision today. While modern CNNs have achieved remarkable results, they still struggle with the kind of flexible, scale-invariant processing that humans perform effortlessly.

Systems incorporating more recurrent processing, attention mechanisms, and working memory-like components might better capture the recursive nature of visual intelligence. Rather than adding more layers, we might need architectures that can dynamically apply the same operations at different scales and levels of abstraction.

A New Perspective

The recursive view of visual processing invites us to question a fundamental assumption in neuroscience and AI. Perhaps the brain’s remarkable efficiency stems not from having specialized circuits for every level of visual complexity, but from its ability to apply the same fundamental operations recursively across different scales and contexts.

This perspective might help explain why humans show remarkable invariance to scale, position, and viewpoint that still challenges deep learning systems. We’re not just processing images through a fixed hierarchy—we’re dynamically engaging with visual information at multiple levels of abstraction using the same core neural mechanisms.

As we continue to unravel the mysteries of visual perception, perhaps it’s time to look beyond the comforting ladder of hierarchy and embrace the more versatile spiral of recursion.

The Mystery of Sensory Comparison

One of the most remarkable yet overlooked aspects of perception is our ability to compare sensations across different regions of our sensory fields. Consider color: when you look at a blue sky, the “blueness” you experience in the upper left corner of your visual field is immediately comparable to the “blueness” in the upper right corner. Despite these sensations originating from entirely different photoreceptors in physically separate parts of your retina, you effortlessly recognize them as the same color.

This capability extends beyond vision. With touch, we can feel similar sensations on widely separated parts of our body and recognize their similarity. A gentle pressure applied to your left hand feels qualitatively similar to the same pressure applied to your right foot, despite these sensations originating from completely different nerve endings separated by considerable physical distance.

The same principle applies to temperature, itch, pain, and other bodily sensations. We can compare the intensity of an itch on our back with one on our arm, or recognize that the temperature of water on our left hand matches that on our right.

Beyond Connectivity

What makes this ability so remarkable is that there’s no direct physical connection between the receptors involved. The photoreceptors in one part of your retina don’t connect directly to those in another part. The touch receptors in your hand have no direct neural pathway to those in your foot.

Yet somehow, our brain constructs a unified sensory field where these disparate signals can be compared, contrasted, and integrated into a coherent experience. The signals from physically distant receptors must be transformed into a common “neural language” or representational space where comparison becomes possible.

This transformation transcends the physical arrangement of our sensory apparatus. It suggests that our brain doesn’t just process sensory data—it creates a unified experiential field where sensations acquire qualities that can be compared across space.

Implications for Consciousness and Artificial Intelligence

This observation has profound implications for our understanding of consciousness and the development of truly feeling artificial systems.

For consciousness, it suggests that creating a unified field of comparable sensations might be one of the fundamental operations that transforms mere signal processing into subjective experience. The ability to compare sensations across space may be integral to what philosophers call “qualia”—the subjective qualities of experience.

For artificial intelligence and robotics, this insight points to a crucial missing piece in our attempts to create machines that can “feel.” A robot with sensors distributed across its body doesn’t just need to register input from these sensors—it needs neural architecture that transforms these spatially distinct signals into a unified representational space where sensations can be compared and integrated.

Current AI systems process inputs from different sensors separately or merge them in simplistic ways. They lack the unified sensory field that allows for the rich, qualitative comparisons that characterize human experience. Even the most advanced deep learning systems don’t create the kind of representational space where a sensation in one part of the system can be qualitatively compared to a sensation in another part.

The Bridge to Feeling

Understanding how our brain creates this unified field of comparable sensations may be the key to developing artificial systems that truly “feel” rather than merely “sense.” It might be the bridge between data processing and consciousness—the ability to create a unified experiential field from distributed sensory inputs.

This is not merely a technical challenge but a profound philosophical one. It requires us to understand how physical systems like brains transform spatially distinct signals into a unified field of comparable experiences—how the firing of neurons becomes the rich, integrated tapestry of conscious perception.

By exploring this fundamental aspect of perception, we might not only deepen our understanding of human consciousness but also chart a path toward artificial systems that don’t just mimic the outward behaviors of sentient beings but share something of their inner experience as well.

The ability to compare blue in one part of your visual field with blue in another might seem trivial, but it points to one of the deepest mysteries of consciousness—how physically separate neural processes give rise to a unified field of experience where sensations can be compared across space.

Introduction to SFENO

SFENO (Structured Framework for Exploring Network ODEs) is a flexible framework designed to model and simulate biological networks using ordinary differential equations (ODEs). It was developed as an improvement over the existing Cellbox model, addressing key limitations in flexibility, hardware compatibility, and experimental design.

Key Limitations of Cellbox

The original Cellbox model had several limitations:

1. Tightly Coupled Architecture: The ODE equations were deeply embedded within the Cellbox model itself, making it difficult to modify or experiment with different mathematical approaches without rewriting the entire class.

2. Limited Learning Methods: The learning code was designed specifically for the approach described in the paper, lacking flexibility for applying various learning techniques.

3. Hardware Constraints: Being implemented in TensorFlow, the code was not easily adaptable to different hardware environments without significant modifications.

SFENO’s Architectural Improvements

SFENO addresses these limitations through several key architectural changes:

1. Modular Architecture

The architecture clearly separates different components:

• LitSfeno: The top-level wrapper that integrates PyTorch Lightning functionality
• GeneralModel: A middle-layer that handles ODE solving and model coordination
• User-defined ODE modules: Customizable components that define how the state changes

This separation allows researchers to modify just the ODE equations without changing the rest of the system, significantly enhancing experimental flexibility.

Here’s an example of the LitSfeno class showing how it wraps the model and integrates PyTorch Lightning:

class LitSfeno(pl.LightningModule):
    def __init__(self, config, args, ud_class_nm):
        super().__init__()
        
        # Create model using the user-defined ODE class
        self.model = GeneralModel.build_model(config=config,
                                              args=args,
                                              user_defined_dxdt=ud_class_nm,
                                              master=self)
        
        self.lr = 0.001
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr=self.lr)
        
    def forward(self, t, y0, arg_dict):
        return self.model(self, t, y0, arg_dict)
        
    def get_device(self):
        return self.device
        
    def training_step(self, batch, batch_idx):
        inputs, targets = batch
        y0, additional_args = inputs
        
        outputs = self.model(t=0, y0=y0, arg_dict=additional_args)
        loss = self.loss_fn(outputs, targets)
        
        self.log("train_loss", loss, on_step=False, on_epoch=True)
        return loss

2. PyTorch Lightning Integration

SFENO replaces TensorFlow with PyTorch and introduces PyTorch Lightning, providing:

• Hardware agnostic operation across CPUs, GPUs, and TPUs without code changes
• Support for distributed training (DDP - Distributed Data Parallel)
• Built-in callbacks at appropriate points during model training
• Automatic tensor management across devices

The following code demonstrates how SFENO leverages PyTorch Lightning for hardware-agnostic training:

# Example of setting up training with different hardware configurations
if pargs.ddp:  # Distributed Data Parallel (multi-GPU)
    trainer = Trainer(devices=pargs.num_gpu,
                      accelerator="gpu",
                      max_epochs=config.config['max_epoch'],
                      strategy="ddp_find_unused_parameters_false")
else:  # Single GPU
    trainer = Trainer(devices=pargs.num_gpu,
                      accelerator="gpu",
                      max_epochs=config.config['max_epoch'])

# Train the model with the same code regardless of hardware configuration
trainer.fit(lit_sfeno, datamodule=lit_datamodule)

3. Flexible Masking System

SFENO implements a generalized masking system for network constraints:

• Four possible interaction types between nodes: no influence, bidirectional influence, positive-only influence, or negative-only influence
• User-defined network structure information can be directly incorporated
• The StructInfoLayer class manages these constraints through an elegant filtering mechanism

Here’s how the masking system is implemented:

class StructInfoLayer(nn.Module):
    def __init__(self, config, master):
        super().__init__()
        self.master = [master]
        
        # Initialize adjacency matrix
        if config.adj_mat==1:
            self.adj_mat = np.ones((config.config["n_x"],config.config["n_x"]))
        else:
            self.adj_mat = config.adj_mat
            
        self.node_names = config.node_nm_ls
        self.node_num = len(self.adj_mat)
        
        # Network weight parameters
        self.register_parameter(
            'W', nn.parameter.Parameter(torch.zeros((self.node_num, self.node_num), 
                                                    requires_grad=True))
        )
        
        # Define different parameter transformation functions
        def zero_param(x):
            return x * 0
            
        def R_param(x):
            return x
            
        def R_pos_param(x):
            return F.softplus(x)  # Positive-only influence
            
        def R_ngt_param(x):
            return(-1) * F.softplus(x)  # Negative-only influence
            
        self.adj_val2get_param_func = {
            0: zero_param,      # No influence
            1: R_param,         # Bidirectional influence
            2: R_pos_param,     # Positive-only influence
            3: R_ngt_param      # Negative-only influence
        }

    def load_mask(self):
        # Create masks for each interaction type
        self.mask = torch.zeros(
            (self.node_num, self.node_num),
            requires_grad=False).type_as(next(self.parameters()))
            
        for adj_val, func in self.adj_val2get_param_func.items():
            mask = func(self.W * self.adj_val2mask[adj_val].type_as(next(self.parameters())))
            self.mask = self.mask + mask

4. Easy Experimental Configuration

The framework supports:

• Hyperparameter scheduling across training stages
• Custom loss function definitions within user-defined ODE modules
• Automatic network weight saving at specified epochs
• Performance tracking and evaluation

Example of implementing a custom ODE function with hyperparameter scheduling:

class Example_Cellbox(GeneralODE):
    def __init__(self, config, master):
        super().__init__(config, master)
        
        # Initialize network model with structural information
        self.net_model = StructInfoLayer(config, master=self)
        self.net_model.init_params()
        self.net_model.load_mask()
        
        # Register learnable parameters
        self.register_parameter(
            'epsilon', nn.parameter.Parameter(
                F.softplus(torch.ones((config.config["n_x"]),
                           dtype=torch.float32, requires_grad=True)))
        )
        
        self.register_parameter(
            'alpha', nn.parameter.Parameter(
                F.softplus(torch.ones((config.config["n_x"]),
                           dtype=torch.float32, requires_grad=True)))
        )
        
        # Setup hyperparameters
        self.hyper_params = {
            'l1': 1e-4,
            'l2': 1e-4
        }
        
        self.mse = torch.nn.MSELoss()
        self.stages = config.config['stages']
        
    def forward(self, t0, x, arg_dict):
        '''Differential equation implementation'''
        y = self.net_model(x) + arg_dict['mu']
        y = torch.tanh(y)
        y = self.epsilon * y
        y = y - self.alpha * x
        return y
        
    # Custom loss function
    def loss_fn(self, output, target):
        l1 = torch.sum(torch.abs(self.net_model.W))
        l2 = torch.sum(torch.pow(self.net_model.W, 2))
        
        mse_loss = self.mse(torch.squeeze(output), torch.squeeze(target))
        
        total_loss = self.hyper_params['l1'] * l1 + self.hyper_params['l2'] * l2 + mse_loss
        return total_loss
        
    # Hyperparameter scheduling based on training stage
    def training_epoch_end(self, outputs):
        cur_epoch = self.master[0].master[0].current_epoch
        if not self.stages:
            return
        else:
            idx_stage, change_hype = self.epoch2stage_idx(cur_epoch)
            stage = self.stages[idx_stage]
            if change_hype:
                # Update hyperparameters based on stage configuration
                lr = stage['lr_val'] if 'lr_val' in stage else self.config.config['default']['lr_val']
                l1 = stage['l1'] if 'l1' in stage else self.config.config['default']['l1']
                l2 = stage['l2'] if 'l2' in stage else self.config.config['default']['l2']
                nt = stage['nT'] if 'nT' in stage else self.config.config['default']['nT']
                
                print('Changing Hyperparameters')
                print(f'lr {lr}, l1 {l1}, l2 {l2}, nt {nt}')
                
                # Apply new hyperparameters
                self.master[0].master[0].lr = lr
                self.hyper_params['l1'] = l1
                self.hyper_params['l2'] = l2
                self.master[0].nt = nt
                
                self.master[0].master[0].configure_optimizers()
                self.save_net_weight(cur_epoch)

Implementation Details

Core Components

1. LitSfeno Class: Wraps the model with PyTorch Lightning functionality, handling training, validation, and testing steps while maintaining device compatibility.

2. GeneralModel Class: Manages the ODE solving process and coordinates between the Lightning module and the user-defined ODE equations.

class GeneralModel_v2(nn.Module):
    def __init__(self, config, dt, ud_dxdt, master):
        super().__init__()
        # Initialize user defined module
        self.master = [master]
        self.dxdt = ud_dxdt(config, master=master)
        
        self.ode_solver = None
        self.dt = dt
        self.nt = 200
        self.config = config
        
    def forward(self, t, y0, arg_dict):
        if self.nt == 0:
            raise Exception('Number of iteration nt on model is zero')
            
        batch_size = y0.shape[0]
        
        # Solve the ODE system
        ys = self.ode_solver.general_solve(
            ode=self.dxdt,
            dt=self.dt,
            nt=self.nt,
            t0=t,
            x=y0,
            arg_dict=arg_dict
        )
        
        # Return the final state
        yhat = ys[-1].reshape(batch_size, -1, 1)
        return yhat

1. Example_Cellbox ODE Class: Demonstrates how to implement custom ODE equations, including parameters such as epsilon and alpha, as well as custom loss functions.

2. StructInfoLayer: Handles network structural constraints through sophisticated masking techniques.

Hardware Compatibility

SFENO achieves hardware agnosticism by following key design principles:

• Avoiding explicit device specifications (.cuda() or .to())
• Using PyTorch Lightning’s device management system
• Properly registering parameters and buffers for automatic device placement

For instance, when creating new tensors, SFENO uses this approach to maintain device compatibility:

# Create a device-agnostic tensor
self.mask = torch.zeros(
    (self.node_num, self.node_num),
    requires_grad=False).type_as(next(self.parameters()))

Example of ODE solver implementation that works across different devices:

class HeunSolver:
    """Heun's ODE solver"""
    def __init__(self):
        pass

    def general_solve(self, ode, dt, nt, t0, x, arg_dict):
        batch_s = x.shape[0]
        xs = []
        
        for i in range(nt):
            t = i*dt
            dxdt_current = ode(t, x, arg_dict)
            dxdt_next = ode(t + dt, x + dt * dxdt_current, arg_dict)
            x = x + dt * 0.5 * (dxdt_current + dxdt_next)
            xs.append(x)

        xs = torch.stack(xs, dim=0)
        return xs

Performance Benchmarks

The system was tested on multiple biological datasets:

• Borisov 2009 (Cyanobacteria photosynthesis)
• Molinelli 2013 (Immunology disease treatment)
• Nelander 2008 (Brain tumor research)
• Pezze 2012 (Metabolic network analysis)
• Schliemann 2011 (Protein interaction networks)
• Korkut 2015 (Cancer biology data)

Example of running experiments across multiple datasets:

experiment_path_ls = [
    'sfeno/datasets/borisov_2009/sfeno_data',
    'sfeno/datasets/molinelli_2013/sfeno_data',
    'sfeno/datasets/nelander_2008/sfeno_data',
    'sfeno/datasets/pezze_2012/sfeno_data',
    'sfeno/datasets/schliemann_2011/sfeno_data',
    'sfeno/datasets/korkut_2015a/sfeno_data',
]

for net_info in [False, True]:
    for data_path in experiment_path_ls:
        config.config['data_path'] = data_path
        config.load_paths_n_info()
        
        # Configure the model based on parameters
        config.config['batchsize'] = config.config['batchsize'] if pargs.num_batch==0 else pargs.num_batch
        config.config['max_epoch'] = config.config['max_epoch'] if pargs.num_epoch==0 else pargs.num_epoch
        
        # Setup data loaders
        trainloader, validloader, testloader = get_dataloaders(config)
        lit_datamodule = DataModule(trainloader, validloader, testloader)
        
        # Create model with appropriate ODE implementation
        if pargs.nn_ode:
            lit_sfeno = LitSfeno(config=config, args=args, ud_class_nm=general_model.NN_dxdt)
        else:
            lit_sfeno = LitSfeno(config=config, args=args, ud_class_nm=general_model.Example_Cellbox)
            
        # Train and evaluate
        trainer.fit(lit_sfeno, datamodule=lit_datamodule)
        pred_ls = trainer.predict(lit_sfeno, datamodule=lit_datamodule)

Initial performance scaling tests showed promising results:

• Single GPU: 21 hours for 3000 epochs with 1000 nodes
• 4-GPU DDP: 12 hours for the same workload (approximately 1.75x speedup)

While these results are preliminary and will require more extensive testing across different hardware configurations and model sizes, they suggest that SFENO can effectively utilize distributed computing resources.

Conclusion

SFENO offers a more flexible alternative to the original Cellbox model by providing a modular, hardware-agnostic framework for modeling biological networks. The separation of ODE equations from the core model architecture, along with PyTorch Lightning integration and the enhanced masking system, simplifies experimentation with different mathematical approaches.

The framework aims to reduce the technical barriers to modeling complex biological systems, allowing researchers to focus more on the biological questions rather than implementation details. While SFENO is still in development and has been tested on a limited number of datasets, early results suggest it could be a useful tool for systems biology research.

Researchers interested in customizing ODE models can create a class that inherits from GeneralODE, implement the forward method to define their differential equations, and specify a custom loss_fn method for training. This design provides flexibility without requiring modifications to the core architecture, potentially making it easier to explore different modeling approaches in systems biology.

github links:
Cellbox
SFENO

1. Mandelbrot-Planet
2. https://github.com/RubADuckDuck/Mandelbrot-Planet/tree/main/src/Network
3. https://github.com/RubADuckDuck/Mandelbrot-Planet/tree/main/include/Network

Project Overview

This project implements a networked cooperative game with a focus on robust client-server communication and game state management.

Technical Architecture

The game uses a dual-protocol networking approach:

• TCP for reliable, ordered communication
• UDP for real-time, low-latency game updates

Key Components

1. Network Message System The system translates network messages into game commands through a multi-step process:
   • Network messages represent raw game events
   • These messages are converted to game commands
   • Commands modify the game state

   Example of message processing:

   std::unique_ptr<IGameCommand> GameMessageProcessor::ProcessMessage(const INetworkMessage& message) {
       switch (message.GetType()) {
       case MessageType::PLAYER_INPUT:
           return std::make_unique<PlayerInputCommand>(
               message.playerDirection, 
               message.playerID
           );
       // Other message types handled similarly
       }
   }
   

2. Game State Management The GameState class manages the entire game world, including:
   • Object creation and registration
   • Dynamic object type handling
   • Grid-based positioning
   • Object lifecycle management
3. Connection Workflow The connection process involves several stages:
   • TCP handshake
   • Authentication
   • UDP verification
   • Game state synchronization

Technical Implementations

Object Creation

void GameState::CreateAndRegisterGameObjectWithID(uint32_t id, uint8_t typeId, bool fromNetwork) {
    std::unique_ptr<GameObject> newGameObject;
    newGameObject = factoryRegistry[typeId]->Create();
    newGameObject->SetID(id);
    gameObjects[id] = std::move(newGameObject);
    objectsByType.insert({ typeId, id });
}

Message Handling

void NetworkCodec::HandleNetworkData(const std::vector<uint8_t>& data, GameState& gameState) {
    try {
        std::unique_ptr<INetworkMessage> message = Decode(data);
        std::unique_ptr<IGameCommand> command = GameMessageProcessor::GetInstance().ProcessMessage(*message);
        command->Execute(gameState);
    }
    catch (const std::exception& e) {
        std::cerr << "Error processing message: " << e.what() << std::endl;
    }
} 

Technical Challenges Addressed

• Consistent game state synchronization
• Secure connection verification
• Real-time input handling
• Flexible object management

Technologies and Tools

• C++
• ASIO Networking Library
• UDP/TCP Protocols
• Object-Oriented Design Patterns

Implementation Highlights

• Secure authentication mechanism
• Cryptographically secure verification codes
• Dynamic object creation and management
• Flexible communication protocol
• Efficient state synchronization

Want to see the full code or learn more? Check out the GitHub repository:

1. Mandelbrot-Planet
2. https://github.com/RubADuckDuck/Mandelbrot-Planet/tree/main/src/Network
3. https://github.com/RubADuckDuck/Mandelbrot-Planet/tree/main/include/Network

1. Mandelbrot-Planet
2. https://github.com/RubADuckDuck/Mandelbrot-Planet/blob/main/include/Core/GridManager.h
3. https://github.com/RubADuckDuck/Mandelbrot-Planet/blob/main/src/Core/GridManager.cpp

Project Overview

This post documents my implementation of non-Euclidean geometry in a cooperative game environment. The project uses concepts from differential geometry, particularly parallel transport, to create a general framework for representing various non-Euclidean spaces in a grid-based game world.

Parallel Transport Implementation

The core mathematical concept I implemented is parallel transport, which describes how vectors change when moved along paths in curved spaces. In the context of my game, this determines how a player’s orientation changes when moving through different regions of space.

The implementation is centered around the ParallelTransporter class:

class ParallelTransporter {
public:
    // Conversion between Direction enum and integer representations
    static int Direction2Int(Direction xDir);
    static Direction Int2Direction(int x);
    
    // Functions that implement different rotations
    static int TurnLeft(int x);
    static int TurnRight(int x);
    static int Turn180(int x);
    static int StayStraight(int x);
    
    // Calculates resulting direction after movement
    Direction CalculateDireciton(Direction facingWhere, Direction goingWhere);
    
    // Map of rotation functions
    Int2RotationOnInt directionInt2RotationInt;
};

This class encodes how directions transform when moving through space. For each grid cell, the directionInt2RotationInt map defines what happens to a player’s orientation when moving in each of the four cardinal directions. This allows for precise control over the local geometry at each point in the game world.

Movement System

The movement system is implemented in the MovementManager class, which coordinates how entities navigate the non-Euclidean space:

class MovementManager {
public:
    std::vector<std::vector<Action2Coord2d*>> grid2Transporter;
    std::vector<std::vector<ParallelTransporter*>> grid2ParallelTransporter;
    
    NavigationInfo Move(Coord2d position, Direction movingDirection, Direction facingDirection);
};

The Move method calculates both the new position and new orientation when moving in a particular direction. It returns a NavigationInfo struct containing the updated position, direction, and orientation change.

Cube World Implementation

As a concrete example, I implemented a cube world where a flat grid wraps around the six faces of a cube. This requires careful handling of the connections between faces, particularly at the edges:

void MovementManager::InitTransporters(int startY, int startX, int size) {
    // Code that sets up connections between cube faces
    // Defines how direction changes when crossing between faces
}

When a player moves across an edge of the cube, both their position and orientation update according to the parallel transport rules. For example, moving “up” from the front face of the cube places the player on the top face, but now “up” means a different direction in the global coordinate system.

Visual Representation

To visualize the non-Euclidean space, I implemented the GridTransformManager class, which defines 3D positions for each grid cell:

void GridTransformManager::InitTransforms(int startY, int startX, int size) {
    // Maps logical grid positions to 3D coordinates and orientations
}

This creates a coherent visual representation where the connections defined by parallel transport are reflected in the game’s graphics.

Extensibility

The framework is designed to be general, allowing for various non-Euclidean spaces to be implemented:

1. Torus worlds: The InitializeTori method sets up a simple wrap-around world

1. Custom manifolds: By defining appropriate parallel transport rules, any manifold can be represented

Technical Details

The implementation uses several key data structures:

• Action2Coord2d: Maps a movement direction to the resulting coordinate
• Int2RotationOnInt: Maps direction integers to rotation functions
• Coord2dWithDirection: Represents position and orientation

The system discretizes continuous geometric concepts to work in a grid-based environment, but preserves the essential mathematical properties of parallel transport.

Applications

This technical framework enables several gameplay possibilities:

1. Spatial puzzles that leverage non-Euclidean geometry
2. Environments that are more complex than their memory footprint would suggest
3. Navigation challenges where understanding the geometry is key

Want to see the full code or learn more? Check out the GitHub repository:

1. Mandelbrot-Planet
2. https://github.com/RubADuckDuck/Mandelbrot-Planet/blob/main/include/Core/GridManager.h
3. https://github.com/RubADuckDuck/Mandelbrot-Planet/blob/main/src/Core/GridManager.cpp

1. Python-side
2. Unity-side

I built a system that tracks where you are in 3D space using two webcams and adjusts a display to look 3D from your viewpoint. Here’s how it works.

How We Find Your Position in 3D

The system uses two webcams to figure out where you are. Think of each webcam as shooting out an invisible line toward you - where these lines meet is your position. Here’s the specific process:

1. First, we find where you appear in each camera’s view:

def img_2d_to_3d(img_coord, fx, fy, wfov, hfov): 
    # Convert 2D image coordinates to 3D direction vector
    u = img_coord[0]  # x position in image
    v = img_coord[1]  # y position in image

    # Convert to normalized coordinates using camera parameters
    x = u / fx - math.tan(wfov / 2)
    y = v / fy - math.tan(hfov / 2) 
    z = 1 

    return np.array([x, y, z])

1. Then we find the rays from each camera to you:

# Camera positions in 3D space
camera1_global_coord = np.array([-1, 0, 0])  # Left camera
camera2_global_coord = np.array([1, 0, 0])   # Right camera

# Get 2D positions from each camera
position_cam1 = find_color_coordinates_hsv(cur_img1)  # Find you in left image
position_cam2 = find_color_coordinates_hsv(cur_img2)  # Find you in right image

# Convert to 3D directions
coord_3d_cam1 = img_2d_to_3d(position_cam1, fx_focal_length, fy_focal_length, 
                            wfov_rad, hfov_rad)
coord_3d_cam2 = img_2d_to_3d(position_cam2, fx_focal_length, fy_focal_length, 
                            wfov_rad, hfov_rad)

1. Finally, we find where these rays intersect (your position):

# Find intersection of the two rays
intersection_point = calculate_intersection_of_ray(
    camera1_global_coord,    # Position of first camera
    coord_3d_cam1,          # Direction from first camera
    camera2_global_coord,    # Position of second camera
    coord_3d_cam2           # Direction from second camera
)

Sending Position Data to Unity

Once we have your position, we send it to Unity through shared memory:

with mmap.mmap(-1, shm_size, tagname='Local\\memfile', access=mmap.ACCESS_WRITE) as mm:
    data = struct.pack('fff', intersection_point[0], 
                            intersection_point[1], 
                            intersection_point[2]) 
    mm.seek(0) 
    mm.write(data) 
    mm.flush()

Results

This shows how the view changes based on viewer position.

Initial tracking was jittery.

Kalman filtering made the tracking much smoother.

Want to see the full code or learn more? Check out the GitHub repository:

1. Python-side
2. Unity-side