Note
Go to the end to download the full example code. or to run this example in your browser via Binder
Drop outliers and interpolate#
Filter out points with low confidence scores and interpolate over missing values.
Imports#
from movement import sample_data
Load a sample dataset#
ds = sample_data.fetch_dataset("DLC_single-wasp.predictions.h5")
print(ds)
<xarray.Dataset> Size: 61kB
Dimensions: (time: 1085, individuals: 1, keypoints: 2, space: 2)
Coordinates:
* time (time) float64 9kB 0.0 0.025 0.05 0.075 ... 27.05 27.07 27.1
* individuals (individuals) <U12 48B 'individual_0'
* keypoints (keypoints) <U7 56B 'head' 'stinger'
* space (space) <U1 8B 'x' 'y'
Data variables:
position (time, individuals, keypoints, space) float64 35kB 1.086e+03...
confidence (time, individuals, keypoints) float64 17kB 0.05305 ... 0.0
Attributes:
fps: 40.0
time_unit: seconds
source_software: DeepLabCut
source_file: /home/runner/.movement/data/poses/DLC_single-wasp.predi...
ds_type: poses
frame_path: /home/runner/.movement/data/frames/single-wasp_frame-10...
video_path: None
We see that the dataset contains the 2D pose tracks and confidence scores for a single wasp, generated with DeepLabCut. The wasp is tracked at two keypoints: “head” and “stinger” in a video that was recorded at 40 fps and lasts for approximately 27 seconds.
Visualise the pose tracks#
Since the data contains only a single wasp, we use
xarray.DataArray.squeeze() to remove
the dimension of length 1 from the data (the individuals dimension).
ds.position.squeeze().plot.line(
x="time", row="keypoints", hue="space", aspect=2, size=2.5
)
<xarray.plot.facetgrid.FacetGrid object at 0x7fbc07fa3550>
We can see that the pose tracks contain some implausible “jumps”, such as the big shift in the final second, and the “spikes” of the stinger near the 14th second. Perhaps we can get rid of those based on the model’s reported confidence scores?
Visualise confidence scores#
The confidence scores are stored in the confidence data variable.
Since the predicted poses in this example have been generated by DeepLabCut,
the confidence scores should be likelihood values between 0 and 1.
That said, confidence scores are not standardised across pose
estimation frameworks, and their ranges can vary. Therefore,
it’s always a good idea to inspect the actual confidence values in the data.
Let’s first look at a histogram of the confidence scores. As before, we use
xarray.DataArray.squeeze() to remove the individuals dimension
from the data.
ds.confidence.squeeze().plot.hist(bins=20)
(array([ 61., 13., 16., 10., 10., 8., 21., 11., 14.,
11., 26., 13., 28., 19., 39., 44., 79., 84.,
149., 1514.]), array([0. , 0.04999823, 0.09999646, 0.14999469, 0.19999292,
0.24999115, 0.29998938, 0.34998761, 0.39998584, 0.44998407,
0.4999823 , 0.54998053, 0.59997876, 0.64997699, 0.69997522,
0.74997345, 0.79997168, 0.84996991, 0.89996814, 0.94996637,
0.99996459]), <BarContainer object of 20 artists>)
Based on the above histogram, we can confirm that the confidence scores indeed range between 0 and 1, with most values closer to 1. Now let’s see how they evolve over time.
ds.confidence.squeeze().plot.line(
x="time", row="keypoints", aspect=2, size=2.5
)
<xarray.plot.facetgrid.FacetGrid object at 0x7fbc07614150>
Encouragingly, some of the drops in confidence scores do seem to correspond to the implausible jumps and spikes we had seen in the position. We can use that to our advantage.
Filter out points with low confidence#
Using the
filter_by_confidence()
method of the move accessor,
we can filter out points with confidence scores below a certain threshold.
The default threshold=0.6 will be used when threshold is not
provided.
This method will also report the number of NaN values in the dataset before
and after the filtering operation by default (print_report=True).
We will use xarray.Dataset.update() to update ds in-place
with the filtered position.
Missing points (marked as NaN) in input
Individual: individual_0
head: 0/1085 (0.0%)
stinger: 0/1085 (0.0%)
Missing points (marked as NaN) in output
Individual: individual_0
head: 121/1085 (11.2%)
stinger: 93/1085 (8.6%)
Note
The move accessor filter_by_confidence()
method is a convenience method that applies
movement.filtering.filter_by_confidence(),
which takes position and confidence as arguments.
The equivalent function call using the
movement.filtering module would be:
from movement.filtering import filter_by_confidence
ds.update({"position": filter_by_confidence(position, confidence)})
We can see that the filtering operation has introduced NaN values in the
position data variable. Let’s visualise the filtered data.
ds.position.squeeze().plot.line(
x="time", row="keypoints", hue="space", aspect=2, size=2.5
)
<xarray.plot.facetgrid.FacetGrid object at 0x7fbc0539c110>
Here we can see that gaps (consecutive NaNs) have appeared in the pose tracks, some of which are over the implausible jumps and spikes we had seen earlier. Moreover, most gaps seem to be brief, lasting < 1 second (or 40 frames).
Interpolate over missing values#
Using the
interpolate_over_time()
method of the move accessor,
we can interpolate over the gaps we’ve introduced in the pose tracks.
Here we use the default linear interpolation method (method=linear)
and interpolate over gaps of 40 frames or less (max_gap=40).
The default max_gap=None would interpolate over all gaps, regardless of
their length, but this should be used with caution as it can introduce
spurious data. The print_report argument acts as described above.
Missing points (marked as NaN) in input
Individual: individual_0
head: 121/1085 (11.2%)
stinger: 93/1085 (8.6%)
Missing points (marked as NaN) in output
Individual: individual_0
head: 0/1085 (0.0%)
stinger: 0/1085 (0.0%)
Note
The move accessor interpolate_over_time()
is also a convenience method that applies
movement.filtering.interpolate_over_time()
to the position data variable.
The equivalent function call using the
movement.filtering module would be:
from movement.filtering import interpolate_over_time
ds.update({"position": interpolate_over_time(
position_filtered, max_gap=40
)})
We see that all NaN values have disappeared, meaning that all gaps were indeed shorter than 40 frames. Let’s visualise the interpolated pose tracks.
ds.position.squeeze().plot.line(
x="time", row="keypoints", hue="space", aspect=2, size=2.5
)
<xarray.plot.facetgrid.FacetGrid object at 0x7fbc053fad50>
Log of processing steps#
So, far we’ve processed the pose tracks first by filtering out points with
low confidence scores, and then by interpolating over missing values.
The order of these operations and the parameters with which they were
performed are saved in the log attribute of the position data array.
This is useful for keeping track of the processing steps that have been
applied to the data. Let’s inspect the log entries.
for log_entry in ds.position.log:
print(log_entry)
{'operation': 'filter_by_confidence', 'datetime': '2024-09-06 12:30:55.519032', 'confidence': <xarray.DataArray 'confidence' (time: 1085, individuals: 1, keypoints: 2)> Size: 17kB
0.05305 0.07366 0.03532 0.03293 0.01707 0.01022 ... 0.0 0.0 0.0 0.0 0.0 0.0
Coordinates:
* time (time) float64 9kB 0.0 0.025 0.05 0.075 ... 27.05 27.07 27.1
* individuals (individuals) <U12 48B 'individual_0'
* keypoints (keypoints) <U7 56B 'head' 'stinger'}
{'operation': 'interpolate_over_time', 'datetime': '2024-09-06 12:30:56.977758', 'max_gap': 40}
Filtering multiple data variables#
All movement.filtering functions are available via the
move accessor. These move accessor methods operate on the
position data variable in the dataset ds by default.
There is also an additional argument data_vars that allows us to
specify which data variables in ds to filter.
When multiple data variable names are specified in data_vars,
the method will return a dictionary with the data variable names as keys
and the filtered DataArrays as values, otherwise it will return a single
DataArray that is the filtered data.
This is useful when we want to apply the same filtering operation to
multiple data variables in ds at the same time.
For instance, to filter both position and velocity data variables
in ds, based on the confidence scores, we can specify
data_vars=["position", "velocity"] in the method call.
As the filtered data variables are returned as a dictionary, we can once
again use xarray.Dataset.update() to update ds in-place
with the filtered data variables.
Missing points (marked as NaN) in input
Individual: individual_0
head: 0/1085 (0.0%)
stinger: 0/1085 (0.0%)
Missing points (marked as NaN) in output
Individual: individual_0
head: 121/1085 (11.2%)
stinger: 93/1085 (8.6%)
Missing points (marked as NaN) in input
Individual: individual_0
head: 0/1085 (0.0%)
stinger: 0/1085 (0.0%)
Missing points (marked as NaN) in output
Individual: individual_0
head: 121/1085 (11.2%)
stinger: 93/1085 (8.6%)
Total running time of the script: (0 minutes 2.401 seconds)
