Note
Go to the end to download the full example code. or to run this example in your browser via Binder
Drop outliers and interpolate#
Filter out points with low confidence scores and interpolate over missing values.
Imports#
from movement import sample_data
from movement.filtering import filter_by_confidence, interpolate_over_time
from movement.kinematics import compute_velocity
Load a sample dataset#
ds = sample_data.fetch_dataset("DLC_single-wasp.predictions.h5")
print(ds)
<xarray.Dataset> Size: 61kB
Dimensions: (time: 1085, space: 2, keypoints: 2, individuals: 1)
Coordinates:
* time (time) float64 9kB 0.0 0.025 0.05 0.075 ... 27.05 27.07 27.1
* space (space) <U1 8B 'x' 'y'
* keypoints (keypoints) <U7 56B 'head' 'stinger'
* individuals (individuals) <U12 48B 'individual_0'
Data variables:
position (time, space, keypoints, individuals) float64 35kB 1.086e+03...
confidence (time, keypoints, individuals) float64 17kB 0.05305 ... 0.0
Attributes:
fps: 40.0
time_unit: seconds
source_software: DeepLabCut
source_file: /home/runner/.movement/data/poses/DLC_single-wasp.predi...
ds_type: poses
frame_path: /home/runner/.movement/data/frames/single-wasp_frame-10...
video_path: None
We see that the dataset contains the 2D pose tracks and confidence scores for a single wasp, generated with DeepLabCut. The wasp is tracked at two keypoints: “head” and “stinger” in a video that was recorded at 40 fps and lasts for approximately 27 seconds.
Visualise the pose tracks#
Since the data contains only a single wasp, we use
xarray.DataArray.squeeze() to remove
the dimension of length 1 from the data (the individuals dimension).
ds.position.squeeze().plot.line(
x="time", row="keypoints", hue="space", aspect=2, size=2.5
)
<xarray.plot.facetgrid.FacetGrid object at 0x7fe0a1e22590>
We can see that the pose tracks contain some implausible “jumps”, such as the big shift in the final second, and the “spikes” of the stinger near the 14th second. Perhaps we can get rid of those based on the model’s reported confidence scores?
Visualise confidence scores#
The confidence scores are stored in the confidence data variable.
Since the predicted poses in this example have been generated by DeepLabCut,
the confidence scores should be likelihood values between 0 and 1.
That said, confidence scores are not standardised across pose
estimation frameworks, and their ranges can vary. Therefore,
it’s always a good idea to inspect the actual confidence values in the data.
Let’s first look at a histogram of the confidence scores. As before, we use
xarray.DataArray.squeeze() to remove the individuals dimension
from the data.
ds.confidence.squeeze().plot.hist(bins=20)
(array([ 61., 13., 16., 10., 10., 8., 21., 11., 14.,
11., 26., 13., 28., 19., 39., 44., 79., 84.,
149., 1514.]), array([0. , 0.04999823, 0.09999646, 0.14999469, 0.19999292,
0.24999115, 0.29998938, 0.34998761, 0.39998584, 0.44998407,
0.4999823 , 0.54998053, 0.59997876, 0.64997699, 0.69997522,
0.74997345, 0.79997168, 0.84996991, 0.89996814, 0.94996637,
0.99996459]), <BarContainer object of 20 artists>)
Based on the above histogram, we can confirm that the confidence scores indeed range between 0 and 1, with most values closer to 1. Now let’s see how they evolve over time.
ds.confidence.squeeze().plot.line(
x="time", row="keypoints", aspect=2, size=2.5
)
<xarray.plot.facetgrid.FacetGrid object at 0x7fe0a43c6dd0>
Encouragingly, some of the drops in confidence scores do seem to correspond to the implausible jumps and spikes we had seen in the position. We can use that to our advantage.
Filter out points with low confidence#
Using the movement.filtering.filter_by_confidence() function from the
movement.filtering module, we can filter out points with confidence
scores below a certain threshold. This function takes position and
confidence as required arguments, and accepts an optional threshold
parameter, which defaults to threshold=0.6 unless specified otherwise.
The function will also report the number of NaN values in the dataset before
and after the filtering operation by default, but you can disable this
by passing print_report=False.
We will use xarray.Dataset.update() to update ds in-place
with the filtered position.
ds.update({"position": filter_by_confidence(ds.position, ds.confidence)})
Missing points (marked as NaN) in input
Individual: individual_0
head: 0/1085 (0.0%)
stinger: 0/1085 (0.0%)
Missing points (marked as NaN) in output
Individual: individual_0
head: 121/1085 (11.2%)
stinger: 93/1085 (8.6%)
We can see that the filtering operation has introduced NaN values in the
position data variable. Let’s visualise the filtered data.
ds.position.squeeze().plot.line(
x="time", row="keypoints", hue="space", aspect=2, size=2.5
)
<xarray.plot.facetgrid.FacetGrid object at 0x7fe0a1e969d0>
Here we can see that gaps (consecutive NaNs) have appeared in the pose tracks, some of which are over the implausible jumps and spikes we had seen earlier. Moreover, most gaps seem to be brief, lasting < 1 second (or 40 frames).
Interpolate over missing values#
Using the movement.filtering.interpolate_over_time() function from the
movement.filtering module, we can interpolate over gaps
we’ve introduced in the pose tracks.
Here we use the default linear interpolation method (method=linear)
and interpolate over gaps of 40 frames or less (max_gap=40).
The default max_gap=None would interpolate over all gaps, regardless of
their length, but this should be used with caution as it can introduce
spurious data. The print_report argument acts as described above.
ds.update({"position": interpolate_over_time(ds.position, max_gap=40)})
Missing points (marked as NaN) in input
Individual: individual_0
head: 121/1085 (11.2%)
stinger: 93/1085 (8.6%)
Missing points (marked as NaN) in output
Individual: individual_0
head: 0/1085 (0.0%)
stinger: 0/1085 (0.0%)
We see that all NaN values have disappeared, meaning that all gaps were indeed shorter than 40 frames. Let’s visualise the interpolated pose tracks.
ds.position.squeeze().plot.line(
x="time", row="keypoints", hue="space", aspect=2, size=2.5
)
<xarray.plot.facetgrid.FacetGrid object at 0x7fe0997085d0>
Log of processing steps#
So, far we’ve processed the pose tracks first by filtering out points with
low confidence scores, and then by interpolating over missing values.
The order of these operations and the parameters with which they were
performed are saved in the log attribute of the position data array.
This is useful for keeping track of the processing steps that have been
applied to the data. Let’s inspect the log entries.
for log_entry in ds.position.log:
print(log_entry)
{'operation': 'filter_by_confidence', 'datetime': '2024-12-09 16:46:35.015246', 'arg_1': <xarray.DataArray 'confidence' (time: 1085, keypoints: 2, individuals: 1)> Size: 17kB
0.05305 0.07366 0.03532 0.03293 0.01707 0.01022 ... 0.0 0.0 0.0 0.0 0.0 0.0
Coordinates:
* time (time) float64 9kB 0.0 0.025 0.05 0.075 ... 27.05 27.07 27.1
* keypoints (keypoints) <U7 56B 'head' 'stinger'
* individuals (individuals) <U12 48B 'individual_0'}
{'operation': 'interpolate_over_time', 'datetime': '2024-12-09 16:46:36.468583', 'max_gap': 40}
Filtering multiple data variables#
We can also apply the same filtering operation to
multiple data variables in ds at the same time.
For instance, to filter both position and velocity data variables
in ds, based on the confidence scores, we can specify a dictionary
with the data variable names as keys and the corresponding filtered
DataArrays as values. Then we can once again use
xarray.Dataset.update() to update ds in-place
with the filtered data variables.
# Add velocity data variable to the dataset
ds["velocity"] = compute_velocity(ds.position)
# Create a dictionary mapping data variable names to filtered DataArrays
# We disable report printing for brevity
update_dict = {
var: filter_by_confidence(ds[var], ds.confidence, print_report=False)
for var in ["position", "velocity"]
}
# Use the dictionary to update the dataset in-place
ds.update(update_dict)
Total running time of the script: (0 minutes 2.250 seconds)
