Guides
C++ APIC APIDownloadDiscordGitHub

Integrating With Games

Suggest Edits

Ultralight offers developers the ability to display fast, modern HTML/CSS/JS within games and other GPU-based apps.

Let's dive into the main things you need to know to integrate Ultralight into an existing game or game engine.

Two Ways to Integrate

Ultralight can either:

  • Render via the CPU to a pixel buffer (using the Surface API).
  • Render via the GPU to a texture (using the GPUDriver API)

The pixel-buffer method is easiest to get started and gives great performance across a range of use-cases. In a game, you would simply upload the pixel buffer to a texture each time it changes.

You can always switch to the GPU renderer later if you are animating lots of paths / images (GPU is superior for these types of workloads) and are comfortable working with low-level GPU driver commands.

👍

We'll be using the CPU Renderer (Surface API) in this article.

If you'd like to use the GPU Renderer instead, please see Using a Custom GPUDriver

Configuring the Library

The first thing to do is to set up our Config-- you can use this to customize many low-level rendering, layout, performance, and styling aspects of the engine.

#include <Ultralight/Ultralight.h>
  
using namespace ultralight;

void Init() {
  Config config;
  
  ///
  /// Let's set some custom global CSS to make our background
  /// purple by default.
  ///
  config.user_stylesheet = "body { background: purple; }";
  
  ///
  /// Pass our configuration to the Platform singleton so that
  /// the library can use it.
  ///
  Platform::instance().set_config(config);
}

📘

Check out the Config API

There's more options to choose from, see the full Config API here.

Defining Platform Handlers

Ultralight was designed to be as platform-agnostic as possible to allow maximum portability and customizability.

The Platform singleton can be used to provide custom handlers for things like file loading, font loading, clipboard access, and other low-level OS tasks.

All platform handlers are optional except for FileSystem and FontLoader.

Using the Default Platform Handlers

To make it easier to get started, AppCore provides default implementations for FileSystem, FontLoader, and Logger for all major desktop platforms.

You can use these default implementations in your code via the following (just make sure to #include <AppCore/Platform.h>, it's not included in the main library header):

#include <Ultralight/Ultralight.h>
#include <AppCore/Platform.h>
  
using namespace ultralight;

void InitPlatform() {
  ///
  /// Use the OS's native font loader
  ///
  Platform::instance().set_font_loader(GetPlatformFontLoader());

  ///
  /// Use the OS's native file loader, with a base directory of "."
  /// All file:/// URLs will load relative to this base directory.
  ///
  Platform::instance().set_file_system(GetPlatformFileSystem("."));

  ///
  /// Use the default logger (writes to a log file)
  ///
  Platform::instance().set_logger(GetDefaultLogger("ultralight.log"));
}

Using a Custom FileSystem

When integrating into a game, you may want to provide your own FileSystem implementation to load encrypted / compressed HTML/JS/CSS assets from your game's native data loader.

To do this you would simply subclass the FileSystem interface, override the callbacks, and provide an instance of the class to Platform::instance().set_file_system(). (This instance should outlive the Renderer).

Creating the Renderer

The Renderer singleton maintains the lifetime of the library, orchestrates all painting and updating, and is required before creating any Views.

To create the Renderer simply call Renderer::Create().

RefPtr<Renderer> renderer;

void CreateRenderer() {
  ///
  /// Create our Renderer (call this only once per application).
  /// 
  /// The Renderer singleton maintains the lifetime of the library
  /// and is required before creating any Views.
  ///
  /// You should set up the Platform handlers before this.
  ///
  renderer = Renderer::Create();
}

Creating Views

Views are sized containers for displaying web-page content, similar to a tab in a browser. You can create one via Renderer::CreateView():

RefPtr<View> view;

void CreateView() {
  ///
  /// Configure our View, make sure it uses the CPU renderer by
  /// disabling acceleration.
  ///
  ViewConfig view_config;
  view_config.is_accelerated = false;
  
  ///
  /// Create an offscreen web view, 500 by 500 pixels large.
  ///
  view = renderer->CreateView(500, 500, view_config, nullptr);
  
  ///
  /// Load raw HTML asynchronously into the view.
  ///
  view->LoadHTML("<h1>Hello World!</h1>");
}

Using the Surface

Each View is painted to an underlying pixel buffer Surface when using the CPU renderer.

You can get the Surface for a View via View::surface():

///
/// Get the pixel-buffer Surface for a View.
///
Surface* surface = view->surface();

The default Surface implementation is BitmapSurface, which is backed by a Bitmap.

You can access it via the following:

///
/// Get the pixel-buffer Surface for a View.
///
Surface* surface = view->surface();

///
/// Cast it to a BitmapSurface.
///
BitmapSurface* bitmap_surface = (BitmapSurface*)surface;

///
/// Get the underlying bitmap.
///
RefPtr<Bitmap> bitmap = bitmap_surface->bitmap();

Once you have the Bitmap, you can lock it to access the pixel buffer and upload it to a texture.

void CopyBitmapToTexture(RefPtr<Bitmap> bitmap) {
  ///
  /// Lock the Bitmap to retrieve the raw pixels.
  /// The format is BGRA, 8-bpp, premultiplied alpha.
  ///
  void* pixels = bitmap->LockPixels();

  ///
  /// Get the bitmap dimensions.
  ///
  uint32_t width = bitmap->width();
  uint32_t height = bitmap->height();
  uint32_t stride = bitmap->row_bytes();
 
  ///
  /// Psuedo-code to upload our pixels to a GPU texture.
  ///
  CopyPixelsToTexture(pixels, width, height, stride);

  ///
  /// Unlock the Bitmap when we are done.
  ///
  bitmap->UnlockPixels();
}

Updating the Renderer

You should call Renderer::Update() as often as possible to give the library a chance to process background tasks, JavaScript callbacks, and dispatch event listeners.

void UpdateLogic() {
  ///
  /// Give the library a chance to handle any pending tasks and timers.
  ///
  ///
  renderer->Update();
}

Rendering Views

You should generally call Renderer::Render() once per frame to update the Surface of each View.

Views only re-render if they actually need painting, you can check if a Surface has changed by checking if Surface::dirty_bounds() is non-empty.

You'll also need to call Renderer::RefreshDisplay(0) to update any animations (ideally you would call this whenever the physical display refreshes but once per frame is close enough).

void RenderOneFrame() {
  ///
  /// Notify the renderer that the physical display has refreshed and
  /// any active animations should be updated.
  ///
  renderer->RefreshDisplay(0);
  
  ///
  /// Render all active Views (this updates the Surface for each View).
  ///
  renderer->Render();
  
  ///
  /// Psuedo-code to loop through all active Views.
  ///
  for (auto view : view_list) {
    ///
    /// Get the Surface as a BitmapSurface (the default implementation).
    ///
    BitmapSurface* surface = (BitmapSurface*)(view->surface());
    
    ///
    /// Check if our Surface is dirty (pixels have changed).
    ///
    if (!surface->dirty_bounds().IsEmpty()) {
      ///
      /// Psuedo-code to upload Surface's bitmap to GPU texture.
      ///
      CopyBitmapToTexture(surface->bitmap());
      
      ///
      /// Clear the dirty bounds.
      ///
      surface->ClearDirtyBounds();
    }
  }
}

Passing Mouse Input

Passing mouse input to a View is pretty straightforward-- just create a MouseEvent and pass it to View::FireMouseEvent():

MouseEvent evt;
evt.type = MouseEvent::kType_MouseMoved;
evt.x = 100;
evt.y = 100;
evt.button = MouseEvent::kButton_None;

view->FireMouseEvent(evt);

📘

Make sure coordinates are localized to the View

All mouse coordinates should be localized to the View's position in screen-space (top-left is 0,0)

Also, units should be in logical units, not pixels. (Remember to scale coordinates by whatever device scale you've set in ViewConfig)

Passing Keyboard Input

Keyboard events are broken down into three major types:

  • KeyEvent::kType_RawKeyDown -- Physical key press
  • KeyEvent::kType_KeyUp -- Physical key release
  • KeyEvent::kType_Char -- Text generated from a key press. This is typically only a single character but can be more if generated by an IME (Input Method Editor).

Key Down Events

📘

RawKeyDown

You should almost always use KeyEvent::kType_RawKeyDown for key presses since it lets WebCore translate these events properly.

Whenever a key is pressed on the keyboard, you should create a KeyEvent with type KeyEvent::kType_RawKeyDown and pass it to View::FireKeyEvent():

// Synthesize a key down event for the 'Right Arrow' key
KeyEvent evt;
evt.type = KeyEvent::kType_RawKeyDown;
evt.virtual_key_code = KeyCodes::GK_RIGHT;
evt.native_key_code = 0;
evt.modifiers = 0;

// You'll need to generate a key identifier from the virtual key code
// when synthesizing events. This function is provided in KeyEvent.h
GetKeyIdentifierFromVirtualKeyCode(evt.virtual_key_code, evt.key_identifier);

view->FireKeyEvent(evt);

Key Up Events

Whenever a key is released on the keyboard, you should create a KeyEvent with type KeyEvent::kType_KeyUp and pass it to View::FireKeyEvent()

// Synthesize a key up event for the 'Right Arrow' key
KeyEvent evt;
evt.type = KeyEvent::kType_KeyUp;
evt.virtual_key_code = KeyCodes::GK_RIGHT;
evt.native_key_code = 0;
evt.modifiers = 0;

// You'll need to generate a key identifier from the virtual key code
// when synthesizing events. This function is provided in KeyEvent.h
GetKeyIdentifierFromVirtualKeyCode(evt.virtual_key_code, evt.key_identifier);

view->FireKeyEvent(evt);

Text Input

In addition to key presses / key releases, you'll need to pass in the actual text generated. (For example, pressing the A key should generate the character a).

// Synthesize an event for text generated from pressing the 'A' key
KeyEvent evt;
evt.type = KeyEvent::kType_Char;
evt.text = "a";
evt.unmodified_text = "a"; // If not available, set to same as evt.text

view->FireKeyEvent(evt);

Enter Keys

The library expects the Enter key to generate text input containing the \n control character. Certain pages will not respond properly to the enter key if you fail to do so.

Debugging Keyboard Input

To help determine if keyboard input is mapped properly, we recommend loading the following page both into your favorite browser and your Viewand comparing the output side-by-side as keys are pressed.

Javascript Key Event Test Script: https://unixpapa.com/js/testkey.html

Handling Keyboard Focus

When dispatching keyboard events in your game, you may only want the View to consume keyboard events when an input element has keyboard focus (usually indicated by a blinking caret).

You can check if a view has input focus via View::HasInputFocus():

if (view->HasInputFocus()) {
  ///
  /// The View has an input element with visible keyboard focus (blinking caret).
  /// Dispatch the keyboard event to the view and consume it.
  ///
  DispatchAndConsumeKeyboardEvent(view, evt);
}

📘

You should call View::Focus() and View::Unfocus() whenever the View gains or loses focus to give visual indication that the View has active input focus (eg, blinking carets in textfields).

Multithreading

The Ultralight API is not thread-safe-- calling the API from multiple threads is not supported and will lead to subtle issues / application instability.

The library does not need to run on the main thread though-- you can create the Renderer on another thread and make all calls to the API on that thread.

Updated 7 months ago