VR Jester API - Gesture Recognition API for Minecraft VR

Do you play Vivecraft or QuestCraft and love mods? Want mods to work better in virtual reality? This solution is designed for both developers and players!

This API allows Minecraft mod developers to easily add gesture support for VR compatibility. Create any custom gestures that can be bound to any keys and trigger events. No more binding controller buttons to keyboard keys - with this API, your body becomes the controller.

Demonstrations:

• Video demonstration 1
• Video demonstration 2
• Example gallery 1
• Example gallery 2
• Example gallery 3

Installation:

1. Install required dependencies. The Questbind mod is only required for use with QuestCraft.
2. Download the VR Jester API mod and place it in the mods folder.
3. Launch Minecraft. Necessary configuration files will be created automatically during initialization.

Usage:

1. In Minecraft VR, bind one of your controller buttons to "Gesture Listener Trigger". It's recommended to use a grip button. For Vivecraft, this is done through Steam VR's controller bindings menu; for QuestCraft - through QuestBind.

2. There are two ways to create gestures:

   • Using the /gesture command
   • Directly editing the config/gesture_store.json file where gestures are stored.

   Example gesture store:

"GESTURE 1": {
 "RIGHT_CONTROLLER|LEFT_CONTROLLER": [
   {
     "vrDevice": "RIGHT_CONTROLLER|LEFT_CONTROLLER",
     "movement": "forward",
     "elapsedTime": 0,
     "speed": 0.0,
     "direction": {
       "x": 0.0,
       "y": 0.0,
       "z": 0.0
     },
     "devicesInProximity": {}
   }
 ]
},
...

3. After creating gestures, you can bind them to keys in the config/VRJesterAPI.cfg file by creating key-value objects in the "GESTURE_KEY_MAPPINGS" field. Key binding names can be found in the .minecraft/options.txt file. The "KEY_ACTION" field determines whether the recognized gesture triggers a single click or a hold key press. Hold is triggered when you maintain the gesture pose after recognition.

"GESTURE_KEY_MAPPINGS": {
"GESTURE 1": {
   "KEY_MAPPING": "examplemod.key.ability_1",
   "KEY_ACTION": "click"
},
"GESTURE 2": {
   "KEY_MAPPING": "key.hotbar.2,key.sneak",
   "KEY_ACTION": "hold"
},
"GESTURE 3": {
   "KEY_MAPPING": "key.keyboard.j",
   "KEY_ACTION": "click"
}
}

Information:

• Each object within square brackets [ ] represents a part of the gesture. You can add multiple such GestureComponent objects. This API recognizes gestures of any complexity! However, the more complex the gesture, the harder it is to perform correctly to match the stored gesture.

• Available values for "movement": forward, back, left, right, up, down. Movement direction is relative to your facing direction when you press the Jester Trigger that initiates the gesture listener. A simple example: punching in the same direction you're facing would be recognized as "forward".

• Available values for "vrDevice": RIGHT_CONTROLLER, LEFT_CONTROLLER, HEAD_MOUNTED_DISPLAY. To recognize a gesture on multiple VR devices, use logical OR via the | symbol, e.g.: RIGHT_CONTROLLER|LEFT_CONTROLLER|HEAD_MOUNTED_DISPLAY.

• The "elapsedTime" field is specified in milliseconds. A value of 2000 means that part of the gesture lasts 2 seconds, suitable for charge-up moves.

• The "speed" field represents the velocity of GestureComponent as a floating-point number. For recognizing punch speed, a suitable threshold would be 1500.0-2000.0 - experiment with values.

• The "direction" field accepts x,y,z floating-point values creating a normalized 3D vector. For example, {0.0, 1.0, 0.0} recognizes when the player's hand is facing upward. There's a certain threshold, so exact matching isn't required.

• The "devicesInProximity" field is formatted as:

"devicesInProximity": {
    "LEFT_CONTROLLER": 20
}

This means the vrDevice of this GestureComponent must be in proximity to the left controller for 20 ticks.

• To handle events triggered by recognized gestures, subscribe to the event bus (Forge) or register in the event callback interface (Fabric).

Forge:

@SubscribeEvent
public void onGestureEvent(GestureEvent event) {
    // gesture handling code here -> event.getGestureName()
}

Fabric:

public static void init() {
    GestureEventCallback.EVENT.register((gestureEvent) -> {
        // gesture handling code here -> gestureEvent.getGestureName()
    });
}

Development Plans:

Future plans include adding continuous gesture recognition, more configuration options, and optimizations.

Update 3/27/2025 - Work on the mod continues despite temporary lack of updates. The developer is busy with other projects but plans to return to this mod.