28. Advanced input system and interactivity

Goal

Why this matters

Prerequisites

1. How Avalonia routes input

Avalonia turns OS-specific events into a three-stage pipeline (InputManager.ProcessInput).

  1. Raw input arrives as RawInputEventArgs (mouse, touch, pen, keyboard, gamepad). Each IRenderRoot has devices that call Device.ProcessRawEvent.
  2. Pre-process observers (InputManager.Instance?.PreProcess) can inspect or cancel before routing. Use this sparingly for diagnostics, not business logic.
  3. Device routing converts raw data into routed events (PointerPressedEvent, KeyDownEvent, TextInputMethodClientRequestedEvent).
  4. Process/PostProcess observers see events after routing—handy for analytics or global shortcuts.

Because the input manager lives in AvaloniaLocator, you can temporarily subscribe:

using IDisposable? sub = InputManager.Instance?
    .PreProcess.Subscribe(raw => _log.Debug("Raw input {Device} {Type}", raw.Device, raw.RoutedEvent));

Remember to dispose subscriptions; the pipeline never terminates while the app runs.

2. Pointer fundamentals and event order

InputElement exposes pointer events (bubble strategy by default).

Event Trigger Key data
PointerEntered / PointerExited Pointer crosses hit-test boundary Pointer.Type, KeyModifiers, Pointer.IsPrimary
PointerPressed Button/contact press PointerUpdateKind, PointerPointProperties, ClickCount in PointerPressedEventArgs
PointerMoved Pointer moves while inside or captured GetPosition, GetIntermediatePoints
PointerWheelChanged Mouse wheel / precision scroll Vector delta, PointerPoint.Properties
PointerReleased Button/contact release Pointer.IsPrimary, Pointer.Captured
PointerCaptureLost Capture re-routed, element removed, or pointer disposed PointerCaptureLostEventArgs.Pointer

Event routing is tunable:

protected override void OnInitialized()
{
    base.OnInitialized();
    AddHandler(PointerPressedEvent, OnPreviewPressed, handledEventsToo: true);
    AddHandler(PointerPressedEvent, OnPressed, routingStrategies: RoutingStrategies.Tunnel | RoutingStrategies.Bubble);
}

Use tunnel handlers (RoutingStrategies.Tunnel) for global shortcuts (e.g., closing flyouts). Keep bubbling logic per control.

Working with pointer positions

3. Pointer capture and lifetime handling

Capturing sends subsequent input to an element regardless of pointer location—vital for drags.

protected override void OnPointerPressed(PointerPressedEventArgs e)
{
    if (e.Pointer.Type == PointerType.Touch)
    {
        e.Pointer.Capture(this);
        _dragStart = e.GetPosition(this);
        e.Handled = true;
    }
}

protected override void OnPointerReleased(PointerReleasedEventArgs e)
{
    if (ReferenceEquals(e.Pointer.Captured, this))
    {
        e.Pointer.Capture(null);
        CompleteDrag(e.GetPosition(this));
        e.Handled = true;
    }
}

Key rules:

4. Multi-touch, pen, and high-precision data

Avalonia assigns unique IDs per contact (Pointer.Id) and marks a primary contact (Pointer.IsPrimary). Keep per-pointer state in a dictionary:

private readonly Dictionary<int, PointerTracker> _active = new();

protected override void OnPointerPressed(PointerPressedEventArgs e)
{
    _active[e.Pointer.Id] = new PointerTracker(e.Pointer.Type, e.GetPosition(this));
    UpdateManipulation();
}

protected override void OnPointerReleased(PointerReleasedEventArgs e)
{
    _active.Remove(e.Pointer.Id);
    UpdateManipulation();
}

Pen-specific data lives in PointerPoint.Properties:

var sample = e.GetCurrentPoint(this);
float pressure = sample.Properties.Pressure; // 0-1
bool isEraser = sample.Properties.IsEraser;

Touch sends a contact rectangle (ContactRect) you can use for palm rejection or handle-size aware UI.

5. Gesture recognizers in depth

Two gesture models coexist:

  1. Predefined routed events in Avalonia.Input.Gestures (Tapped, DoubleTapped, RightTapped). Attach with Gestures.AddDoubleTappedHandler or AddHandler.
  2. Composable recognizers (InputElement.GestureRecognizers) for continuous gestures (pinch, pull-to-refresh, scroll).

To attach built-in recognizers:

GestureRecognizers.Add(new PinchGestureRecognizer
{
    // Your subclasses can expose properties via styled setters
});

Creating your own recognizer lets you coordinate multiple pointers and maintain internal state:

public class PressAndHoldRecognizer : GestureRecognizer
{
    public static readonly RoutedEvent<RoutedEventArgs> PressAndHoldEvent =
        RoutedEvent.Register<InputElement, RoutedEventArgs>(
            nameof(PressAndHoldEvent), RoutingStrategies.Bubble);

    public TimeSpan Threshold { get; set; } = TimeSpan.FromMilliseconds(600);

    private CancellationTokenSource? _hold;
    private Point _pressOrigin;

    protected override async void PointerPressed(PointerPressedEventArgs e)
    {
        if (Target is not Visual visual)
            return;

        _pressOrigin = e.GetPosition(visual);
        Capture(e.Pointer);

        _hold = new CancellationTokenSource();
        try
        {
            await Task.Delay(Threshold, _hold.Token);
            Target?.RaiseEvent(new RoutedEventArgs(PressAndHoldEvent));
        }
        catch (TaskCanceledException)
        {
            // Swallow cancellation when pointer moves or releases early.
        }
    }

    protected override void PointerMoved(PointerEventArgs e)
    {
        if (Target is not Visual visual || _hold is null || _hold.IsCancellationRequested)
            return;

        var current = e.GetPosition(visual);
        if ((current - _pressOrigin).Length > 8)
            _hold.Cancel();
    }

    protected override void PointerReleased(PointerReleasedEventArgs e) => _hold?.Cancel();
    protected override void PointerCaptureLost(IPointer pointer) => _hold?.Cancel();
}

Register the routed event (PressAndHoldEvent) on your control and listen just like other events. Note the call to Capture(e.Pointer) which also calls PreventGestureRecognition() to stop competing recognizers.

Manipulation gestures and inertia

Avalonia exposes higher-level manipulation data through gesture recognizers so you do not have to rebuild velocity tracking yourself.

Attach event handlers directly on the recognizer when you need raw data:

var scroll = new ScrollGestureRecognizer();
scroll.Scroll += (_, e) => _viewport += e.DeltaTranslation;
scroll.Inertia += (_, e) => StartInertiaAnimation(e.Velocity);
GestureRecognizers.Add(scroll);

Manipulation events coexist with pointer events. Mark the gesture event as handled when you consume it so the default scroll viewer does not fight your logic. For custom behaviors (elastic edges, snap points), tune ScrollGestureRecognizer.IsContinuous, ScrollGestureRecognizer.CanHorizontallyScroll, and ScrollGestureRecognizer.CanVerticallyScroll to match your layout.

6. Designing complex pointer experiences

Strategies for common scenarios:

Platform differences worth noting:

7. Keyboard navigation, focus, and shortcuts

Avalonia's focus engine is pluggable.

<StackPanel
    input:XYFocus.Up="{Binding ElementName=SearchBox}"
    input:XYFocus.NavigationModes="Keyboard,Gamepad" />

Key bindings complement commands without requiring specific controls:

KeyBindings.Add(new KeyBinding
{
    Gesture = new KeyGesture(Key.N, KeyModifiers.Control | KeyModifiers.Shift),
    Command = ViewModel.NewNoteCommand
});

HotKeyManager subscribes globally:

HotKeyManager.SetHotKey(this, KeyGesture.Parse("F2"));

Ensure the target control implements ICommandSource or IClickableControl; Avalonia wires the gesture into the containing TopLevel and executes the command or raises Click.

Ensure focus cues remain visible: call NavigationMethod.Tab when moving focus programmatically so keyboard users see an adorner.

8. Gamepad, remote, and spatial focus

When Avalonia detects non-keyboard key devices, it sets KeyDeviceType on key events. Use FocusManager.GetFocusManager(this)?.Focus(elem, NavigationMethod.Directional, modifiers) to respect D-Pad navigation.

Configure XY focus per visual:

Property Purpose
XYFocus.Up/Down/Left/Right Explicit neighbours when layout is irregular
XYFocus.NavigationModes Enable keyboard, gamepad, remote individually
XYFocus.LeftNavigationStrategy Choose default algorithm (closest edge, projection, navigation axis)

For dense grids (e.g., TV apps), set XYFocus.NavigationModes="Gamepad,Remote" and assign explicit neighbours to avoid diagonal jumps. Pair with KeyBindings for shortcuts like Back or Menu buttons on controllers (map gamepad keys via key modifiers on the key event).

Where hardware exposes haptic feedback (mobile, TV remotes), query the platform implementation with TopLevel.PlatformImpl?.TryGetFeature<TFeature>(). Some backends surface rumble/vibration helpers; when none are available, fall back gracefully so keyboard-only users are not blocked.

9. Text input services and IME integration

Text input flows through InputMethod, TextInputMethodClient, and TextInputOptions.

Set options in XAML:

<TextBox
    Text=""
    input:TextInputOptions.ContentType="Email"
    input:TextInputOptions.ReturnKeyType="Send"
    input:TextInputOptions.ShowSuggestions="True"
    input:TextInputOptions.IsSensitive="False" />

When you implement custom text surfaces (code editors, chat bubbles):

  1. Implement TextInputMethodClient to expose text range, caret rect, and surrounding text.
  2. Handle TextInputMethodClientRequested in your control to supply the client.
  3. Call InputMethod.SetIsInputMethodEnabled(this, true) and update the client's TextViewVisual so IME windows track the caret.
  4. On geometry changes, raise TextInputMethodClient.CursorRectangleChanged so the backend updates composition windows.

Remember to honor TextInputOptions.IsSensitive—set it when editing secrets so onboard keyboards hide predictions.

10. Accessibility and multi-modal parity

Advanced interactions must fall back to keyboard and automation:

11. Multi-modal input lab (practice)

Create a playground that exercises every surface:

  1. Project setup: scaffold dotnet new avalonia.mvvm -n InputLab. Add a CanvasView control hosting drawing, a side panel for logs, and a bottom toolbar.
  2. Pointer canvas: capture touch/pen input, buffer points per pointer ID, and render trails using DrawingContext.DrawGeometry. Display pressure as stroke thickness.
  3. Custom gesture: add the PressAndHoldRecognizer (above) to show context commands after 600 ms. Hook the resulting routed event to toggle a radial menu.
  4. Pinch & scroll: attach PinchGestureRecognizer and ScrollGestureRecognizer to pan/zoom the canvas. Update a MatrixTransform as gesture delta arrives.
  5. Keyboard navigation: define KeyBindings for Ctrl+Z, Ctrl+Shift+Z, and arrow-key panning. Update XYFocus properties so D-Pad moves between toolbar buttons.
  6. Gamepad test: using a controller or emulator, verify focus flows across the UI. Log KeyDeviceType in KeyDown to confirm Avalonia recognises it as Gamepad.
  7. IME sandbox: place a chat-style TextBox with TextInputOptions.ReturnKeyType="Send", plus a custom MentionTextBox implementing TextInputMethodClient to surface inline completions.
  8. Accessibility pass: ensure every action has a keyboard alternative, set automation names on dynamically created controls, and test the capture cycle with screen reader cursor.
  9. Diagnostics: subscribe to InputManager.Instance?.Process and log pointer ID, update kind, and capture target into a side list for debugging.

Document findings in README (which gestures compete, how capture behaves on focus loss) so the team can adjust default UX.

12. Troubleshooting & best practices

Look under the hood (source bookmarks)

Check yourself

What's next