Graphical User Interface#

The movement graphical user interface (GUI), powered by our custom plugin for napari, makes it easy to view and explore movement motion tracks. Currently, you can use it to visualise 2D poses datasets as points overlaid on video frames.

Warning

The GUI is still in early stages of development but we are working on ironing out the kinks. Please get in touch if you find any bugs or have suggestions for improvements!

The napari plugin is shipped with the movement package starting from version 0.1.0. To use it, you need to install the package via a method that includes the napari dependency.

Launch the GUI#

To launch the movement GUI, type the following command in your terminal:

movement launch

This is equivalent to running napari -w movement and will open the napari window with the movement widget docked on the right-hand side, as in the screenshot below.

In napari, data is typically loaded into layers, which can be reordered and toggled for visibility in the layers list panel. For example, keypoint data can be added as a points layer, while image stacks (including videos) can be added as image layers. Below, we’ll explain how to do this.

Load a background layer#

Though this is not strictly necessary, it is usually informative to view the keypoints overlaid on a background that provides some spatial context. You can either load the video corresponding to the poses dataset, or a single image, e.g., a still frame derived from that video. You can do this by dragging and dropping the corresponding file onto the napari window or by using the File > Open File(s) menu option. Please read the following sections for detailed information and some important considerations.

Load a video#

When trying to load a video file into napari, you will be prompted via a pop-up dialog to select the reader. Choose the video reader—corresponding to the napari-video plugin—and click OK. You can optionally select to remember this reader for all files with the same extension.

napari-video will load the video as an image stack with a slider at the bottom that you can use to navigate through frames. You may also use the left and right arrow keys to navigate frame-by-frame.

Clicking on the play button will start the video playback at a default rate of 10 frames per second. You can adjust that by right-clicking on the play button or by opening the napari > Preferences menu (File > Preferences on Windows) and changing the Playback frames per second setting.

Video playback limitations

  • The video playback may freeze or stutter if you click on the slider to jump to a specific frame. We recommended pausing the playback before such jumps.

  • napari-video may struggle to play videos at a high frame rate, depending on your hardware, the video resolution and codec. If you experience performance issues, such as the video freezing or skipping frames, try reducing the playback frames per second or fall back to using a single image as a background.

Load an image#

This usually means using a still frame extracted from the video, but in theory you could use any image that’s in the same coordinate system as the tracking data. For example, you could use a schematic diagram of the arena, as long as it has the same width and height as the video and is properly aligned with the tracking data.

Extracting a still frame from a video

You can use the command line tool ffmpeg to extract a still frame from a video.

To extract the first frame of a video:

ffmpeg -i video.mp4 -frames:v 1 first-frame.png

To extract a frame at a specific time stamp (e.g. at 2 seconds):

ffmpeg -i video.mp4 -ss 00:00:02 -frames:v 1 frame-2sec.png

Dragging and dropping the image file onto the napari window (or opening it via the File menu) will load the image as a single 2D frame without a slider.

Load the poses dataset#

Now you are ready to load some pose tracks over your chosen background layer.

On the right-hand side of the window you should see an expanded Load poses menu. To load pose data in napari:

  1. Select the source software from the dropdown menu.

  2. Set the fps (frames per second) of the video the pose data refers to. Note this will only affect the units of the time variable shown when hovering over a keypoint. If the fps is not known, you can set it to 1, which will effectively make the time variable equal to the frame number.

  3. Select the file containing the predicted poses. The path can be directly pasted or you can use the file browser button.

  4. Click Load.

The data should be loaded into the viewer as a points layer. By default, it is added at the top of the layer list.

Note

See supported formats for more information on the expected software and file formats.

You will see a view similar to the one below:

napari widget with poses dataset loaded

The keypoints are represented as points, colour-coded by keypoint ID for single-individual datasets, or by individual ID for multi-individual datasets. These IDs can be also displayed as text next to the points by enabling the display text option from the layer controls panel.

Hovering with your mouse over a point (with the points layer selected) will bring up a tooltip containing the names of the individual and keypoint, the point-wise confidence score (provided by the source software), and the time in seconds (calculated based on the frame number and the fps value you provided).

Using the slider at the bottom of the window, you can move through the frames of the dataset, and the points and video will update in sync.

Stay tuned

Though the display style of the points layer is currently fixed, we are working on adding more customisation options in future releases, such as enabling you to change the point size, colour, or shape.

We are also working on enabling the visualisation of bounding boxes datasets in the plugin.