Skip to content

NevarokML: Under the Hood

NevarokML is a powerful plugin designed to bring machine learning capabilities to Unreal Engine. It operates as a client-server application, where Unreal Engine serves as the server, and a Python library acts as the client. This setup enables seamless integration of machine learning functionalities into Unreal Engine projects. Here's an overview of how NevarokML works:

sequenceDiagram
  autonumber
  participant Unreal Engine
  participant NevarokML Plugin
  participant Python Library

  Unreal Engine ->> NevarokML Plugin: Send game state and observations
  NevarokML Plugin ->> Python Library: Receive game state and observations
  Python Library ->> NevarokML Plugin: Perform machine learning computations
  NevarokML Plugin ->> Unreal Engine: Send computed results (actions, model parameters)

Client-Server Architecture:

  • NevarokML follows a client-server architecture, with Unreal Engine as the server and the Python library as the client. This architecture facilitates the integration of machine learning capabilities into Unreal Engine projects.

Communication via Sockets and JSON:

  • NevarokML utilizes sockets and exchanges data in JSON format to enable communication between Unreal Engine and the Python client. This approach ensures efficient and reliable transfer of information between the server and the client.

Training Models:

  • NevarokML supports training machine learning models using reinforcement learning algorithms. It leverages the stable-baselines3 library, a popular and reliable reinforcement learning library. With NevarokML, you can use algorithms such as Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), Deep Deterministic Policy Gradient (DDPG), Deep Q-Network (DQN), Soft Actor-Critic (SAC), and Twin Delayed DDPG (TD3) to train intelligent agents.

Integration with Unreal Engine Neural Network Engine (NNE):

  • NevarokML seamlessly integrates with Unreal Engine's Neural Network Engine (NNE). It allows you to import trained models into Unreal Engine in the form of NNEModelData. These models can enhance the capabilities of Unreal Engine projects, enabling AI-driven experiences.

Note

It's important to note that the models are trained in the stable-baselines3 environment and are automatically converted to ONNX models upon saving. Unreal Engine supports ONNX models, ensuring compatibility and ease of use.

Platform Compatibility:

  • While the training of models is currently supported only on Windows machines, the trained models can be imported and used on any platform supported by Unreal Engine.

By leveraging the capabilities of NevarokML, Unreal Engine developers can harness the power of machine learning algorithms and create immersive and intelligent experiences within their projects.