Unreal Engine digital twin

Built during my internship at Mojexa, a startup building real-time digital twins that stream IoT sensor data into 3D visualizations. The deliverable was a working proof-of-concept: sensors in the real world, actors moving in an Unreal Engine 5 scene, with sub-second latency and the ability to replay the last 24 hours of activity.

Plugin source · Sample project

Scope#

Two halves, both designed together:

1. A real-time backend that ingests IoT events from devices in the field and fans them out to subscribed game clients.
2. A UE5 C++ plugin that subscribes to entity feeds, transforms GPS coordinates into the game world, and drives in-game actors from the resulting stream.

Either half on its own is a system; the interesting work was making them meet cleanly in the middle.

Architecture#

IoT sensors
     │
     ▼
API Gateway ──► Kinesis Data Streams        (durable, ordered buffer)
                     │
       ┌─────────────┴──────────────┐
       ▼                            ▼
WebSocket servers          Lambda ──► DynamoDB        (historical store)
(KCL consumers)                          │
       │                                 ▼
       ▼                          Replay API (on demand)
  Redis PubSub                            │
       │                                 ▼
       ▼                          WebSocket servers
WebSocket servers                         │
       │                                 ▼
       ▼                          UE5 clients (plugin)
UE5 clients (plugin)

Live tracking and historical replay share the same delivery channel (WebSocket → plugin) but read from different sources upstream. The client doesn't need to know which mode it's in.

Design Decisions#

A few design decisions worth calling out:

Kinesis as the durable buffer, not the streaming layer. Kinesis holds an ordered, replayable record of every event but it doesn't push to clients. The actual streaming layer is WebSocket; Kinesis just guarantees we never lose an event and can replay from a position. This is a distinction I see people get wrong — they reach for Kinesis and then wonder why their clients aren't getting updates.

Kinesis over Kafka. Fully managed, no broker operations, native AWS integration. At Mojexa's scale this was the right trade — Kafka would have been a second platform to learn and run.

Kinesis over EventBridge. EventBridge is an event router for discrete events; it has no ordered shards, no position cursor, no replay. Wrong tool for a continuous high-throughput stream.

Redis PubSub for fan-out across WebSocket servers. WebSocket connections are sticky to a single server, but a given entity's subscribers can be on any server. Redis PubSub on entity-specific channels (e.g. entity:{id}) lets any KCL consumer publish updates and have them reach the right subscribers regardless of which server holds the socket.

Lambda → DynamoDB for the replay path. A separate Lambda tails the same Kinesis stream and writes events into DynamoDB partitioned by entity. Replay is just a range query and a paced re-publish over WebSocket. Keeping it on a separate path means a slow replay can't backpressure live tracking.

The UE5 plugin#

SpacesMarkerManager is a Game Instance that owns the WebSocket connection, the entity subscriptions, and the spawning logic for in-game markers. The plugin's job per tick is small: pull any new messages off the WebSocket, route them to the correct marker by entity ID (spawning a marker if it's new), advance any DynamicMarker queues that have data, and tick down TemporaryMarker TTLs.

Marker classes#

The plugin exposes three Actor subclasses, each modelling a different kind of real-world entity:

• StaticMarker — pinned in space and time. Once spawned it stays put until explicitly removed. Used for fixed assets like buildings or known waypoints.
• TemporaryMarker — pinned in space but with a time-to-live counter. It shrinks visually as it ages and self-destructs when the counter hits zero. Used for short-lived observations like an alert or a one-off ping.
• DynamicMarker — holds a priority queue of (coordinate, timestamp) data points and walks through them in chronological order. When the queue drains it falls back to TTL-decay and eventually self-destructs. Used for moving entities — a vehicle, a person, a drone — where the next position depends on what arrives next on the stream.

Coordinate transforms#

UE5 thinks in Unreal Units (centimetres, by convention) in a local Cartesian frame. The sensors send WGS84 latitude and longitude. The plugin keeps a configurable origin — a lat/lng anchor that maps to UE5 (0, 0, 0) — and converts incoming coordinates into local metres using equirectangular projection scaled by the cosine of the origin latitude:

x = R · (lng - lng₀) · cos(lat₀)
y = R · (lat - lat₀)

This is accurate enough for the scale of a single site (a stadium, a warehouse, a city block). For larger areas you'd switch to a proper map projection per region, but for the digital-twin use case the linearised form is the right level of complexity.

Game Instance ownership#

UGameInstance is the natural home for state that has to outlive any single level or actor — the WebSocket connection, the entity registry, the AWS clients. Unreal guarantees exactly one instance per game, created and destroyed by the engine itself, which is what I wanted.

The caveat I'd flag to anyone copying this approach: a project can only have one UGameInstance. If you already have one, you either fold the marker logic into it or move this logic into a UGameInstanceSubsystem. The subsystem is the cleaner path for a reusable plugin and is what I'd reach for now.

Stack#

• AWS — Kinesis Data Streams, Lambda, DynamoDB, API Gateway, S3
• Terraform — infrastructure as code for the AWS side
• WebSocket — live delivery to UE5 clients (KCL consumers act as servers)
• Redis PubSub — fan-out across WebSocket server instances
• C++ / Unreal Engine 5 — the plugin and sample project
• DynamoDB — historical store for the replay path

Reflection#

This was the first time I designed an end-to-end streaming system rather than touching pieces of one someone else had built. The pieces I keep coming back to are the boring ones — the distinction between a durable log and an actual streaming layer, why fan-out belongs in PubSub rather than the queue, and why the replay path should be a sibling of live tracking rather than layered on top. Those decisions outlast the specific technologies; the same shape works equally well in Kafka, Pulsar, or whatever comes next.

Plugin targets Unreal Engine 5.0.0 or newer.