Retinal curl as a functional signal for heading estimation beyond the focus of expansion
- Vision and Control of Action Group, Department of Cognition, Development and Psychology of Education and Institute of Neurosciences, Universitat de Barcelona, Spain
Figures
Figure 1 with 2 supplements
Ground texture, trajectories, and retinal curl distributions across conditions.
(A) Snapshot of the ground texture based on simplex noise. Yellow lines indicate optic flow vectors computed using the Farnebäck algorithm. A clear rotational component (curl) is visible, consistent with the observer looking at a point (yellow dot) located to the left of the simulated path. The Focus of Expansion is shifted in the direction of the gaze. (B) Schematic of the experimental trajectories. Position (0, 0) represents the starting point of simulated locomotion. The red arrow indicates the initial heading θ0, which was aligned longitudinally with the 3D scene (world coordinates). Participants were instructed to report their perceived heading within this world-centered reference frame. The five colored dots mark the fixation points in world coordinates at the beginning of each trial (20 m ahead of the observer). (C–E) Distributions of mean retinal curl across trials for left-, center-, and right-gaze conditions, respectively. Dark filled bars indicate the unaltered curl condition, while lighter bars and outlined steps represent the cancelled curl condition. The vertical gray line denotes zero curl.
Figure 1—figure supplement 1
Speed profile for a representative trial for different components.
(A) Translational or forward speed. (B) Bounce (vertical component) and sway (lateral component).
Figure 1—figure supplement 2
Time course of rotation rate (head/eye).
Figure 2 with 1 supplement
Perceived heading Reported instant directions are plotted when fixating eccentric points on the ground.
Rows: fixation 2 m (top) and 4 m (bottom) to the side. Columns: different physical path conditions (center column = straight path). Colors code for initial gaze direction (left/center/right). Thick colored lines denote the mean across observers. Thin colored lines denote individual observers. Dark gray denotes physical paths. Axes show lateral position (x-axis) versus depth position (y-axis).
Figure 2—figure supplement 1
Time course of raw responses for one representative participant in trials simulating straight ahead movement.
Positive angles denote left responses.
Figure 3
Average perceived heading for each gaze condition (columns) and each physical path curvature (rows), separately for the three retinal–flow manipulation conditions: unaltered curl (green), cancelled curl (cyan), and over-cancelled curl (purple).
Shaded envelopes indicate between-observer variability (95% CI). Note the heading biases in the gaze center condition/middle row. Unexpected positive or negative curl was added in some trials to maintain a full factorial design (see main text for more details). The right axis applies to the last column and illustrates the mean 2D displacement between observed and physical paths. This measurement indicates the mean displacement that is required for the observed path to align with the physical one. The displacement is shown for the different flow manipulations (color-coded). The error bars indicate between-observer variability (95% CI).
Figure 4 with 13 supplements
Separate fits of the controller.
Average perceived heading across participants for each gaze condition (columns) and each physical path curvature (rows), separately for the three retinal–flow manipulation conditions (thinner solid lines): unaltered curl (green), cancelled curl (cyan), and over-cancelled curl (purple). The different red thicker solid lines denote the best fit of the controller. The numbers in each panel indicate the average lateral deviation per step between the fit and the observed heading. Note that for the centered gaze and straight path, only the unaltered flow condition was fitted.
Figure 4—figure supplement 1
Joint fits of the controller.
Average perceived heading across participants for each gaze condition (columns) and each physical path curvature (rows), separately for the three retinal–flow manipulation conditions (thinner solid lines): unaltered curl (yellow-green), cancelled curl (cyan), and over-cancelled curl (purple). The different red thicker solid lines denote the best fit of the controller. The numbers in each panel indicate the average lateral deviation per step between the fit and the observed heading. Note that for the centered gaze and straight path, only the unaltered flow condition is shown.
Figure 4—figure supplement 2
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 3
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 4
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 5
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 6
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 7
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 8
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 9
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 10
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 11
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 12
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 4—figure supplement 13
Observed mean heading directions (solid lines) and model fits (dotted lines) are shown for two participants (identified by id#).
The plots are organized by heading (columns) and gaze eccentricity (2 m/4 m) and retinal flow condition (rows within the facets). The color of the trajectories indicates the direction of gaze. The number displayed within each panel represents the mean lateral deviation between the fitted trajectory and the observed path.
Figure 5 with 2 supplements
Neural network model of gaze-contingent heading bias.
The parameters used to produce these panels are: I0=0.03, Kp=0.4, σp=0.18, σk=0.12. (A) Ring attractor connectivity showing synaptic weight (W) as a function of relative preferred heading (Δφ) in pixels, featuring a central excitatory peak and asymmetric inhibitory surround. While we present results using asymmetric connectivity, no significant differences were observed between asymmetric and symmetric configurations. (B) Heatmap of neural activity across neuron preferred headings (y-axis) over the frame index (x-axis), with a dashed red line indicating the decoded population heading estimate for the trial in which gaze directed 4 m to the left. (C) Mechanism of sensory-prior competition plotting normalized amplitude against ring position in pixels, illustrating the spatial alignment of the straight-ahead prior (blue dotted line), gaze-centered inhibition (red dashed line), and the resulting neural activity bump (black solid line). (D) Phase portrait showing the change in heading (dθ/dt) versus the heading estimate (θ) in pixels, with a horizontal line at zero marking the convergence of the trajectory toward a stable fixed point. The color bar denotes frame number.
Figure 5—figure supplement 1
Neural model.
First column: The heatmap shows the neural activity across neuron preferred headings (y-axis) over the frame index (x-axis), with a dashed white line indicating the decoded population heading estimate. The color bar denotes the activity level. (middle column) The mechanism of sensory-prior competition plotting normalized amplitude against ring position in pixels, illustrating the spatial alignment of the straight-ahead prior (blue dotted line), gaze-centered inhibition (red dashed line), and the resulting neural activity bump (black solid line). Last column: Phase portrait showing the change in heading (dθ/dt) versus the heading estimate (θ) in pixels. The color bar denotes frame number. Different gaze eccentricities are shown row-wise (A) gaze to the left –2 m-, (B) gaze centered, (C) gaze to the right –2 m- and (D) gaze to the right –4 m-.
Figure 5—figure supplement 2
Reproduced 3D paths from the neural model decoding.
Appendix 3—figure 1
Reproduction of steering paths towards targets.
Response of the controller model (red lines) to the conditions introduced in Experiment 3 of Wilkie and Wann, 2003. Targets were placed 60 m ahead at 10, 14, and 18 to the left and right (gray circles). The gray lines denote an approximation of the observer paths reported in Wilkie and Wann, 2003. The simulated speed in their study was 8 m/s.
Videos
Video 1
Retinal flow pattern during forward translation with fixation on a target (white dot) located to the left of the heading.
For optimal viewing, the display should be centered relative to the observer with a field of view exceeding 50° (e.g. a 60 cm screen viewed from a distance of 60 cm).
Video 2
The same as Video 1, retinal flow pattern during forward translation and fixating a target (white dot) located to the left of the current path, but, the rotational component of the flow has been counteracted, so there is very little curl around the fovea.
For optimal viewing, the display should be centered relative to the observer with a field of view exceeding 50° (e.g. a 60 cm screen viewed from a distance of 60 cm).
Video 3
Controller illustration: how heading is corrected to match gaze by applying Equation 5 in the main text.
Tables
Appendix 2—table 1
Parameter values for reproducing experimental biases.
σ values are expressed in fractions of azimuth range in pixels, which was set to 360 in our simulations.
| Parameter | Value range | Description |
|---|---|---|
| Kp | 0.4–0.8 | Inhibitory gain |
| I0 | 0.03–0.24 | Prior strength |
| σp | 0.18–0.25 | Prior width |
| σk | 0.02–0.12 | Inhibition width |
| τ | 60ms | Membrane time constant |
| N | 181 | Ring population size |
| Nsamples | 400 | Gaze-centered flow samples |
-
The specific parameter values used to produce Figure 5 - figure supplement 1 are: I0=0.03, Kp=0.4, σp=0.18, σk=0.12.
Appendix 3—table 1
Performance for the two fitting approaches (separate and join: models) in the different retinal flow conditions.
Cost is the average 2D deviation per step between the fit and the observed heading.
| Model type | Flow | Normalized cost | Negative log-likelihood | AIC |
|---|---|---|---|---|
| Separate (30 parameters) | Unaltered | 0.080 | –3.54 | 67.08 |
| Separate (28 parameters) | Canceled | 0.066 | –3.41 | 62.82 |
| Separate (28 parameters) | Over-canceled | 0.077 | –3.49 | 62.98 |
| Join (2 parameters) | Unaltered | 0.379 | –4.32 | 12.65 |
| Join (2 parameters) | Canceled | 0.252 | –4.08 | 12.17 |
| Join (2 parameters) | Over-canceled | 0.330 | –4.22 | 12.44 |
Appendix 3—table 2
Parameters of the controller to successfully steer to the 6 targets shown in Appendix 3—figure 1 .
The controller speed was very close to the simulated speed (8 m/s) in Wilkie and Wann, 2003. The final lateral position always ended within the target dimensions (2 m width).
| Target initial eccentricity (°) | Kp | Vel (m/s) | Final x (m) |
|---|---|---|---|
| –18° | 3.54 | 8.00 | –19.9 |
| –14° | 3.57 | 8.03 | –16.4 |
| –10° | 3.21 | 7.7 | –10.5 |
| 10° | 3.38 | 7.8 | 11.2 |
| 14° | 3.69 | 7.82 | 16.4 |
| 18° | 3.68 | 7.90 | 20.4 |
Additional files
-
MDAR checklist
- https://cdn.elifesciences.org/articles/110770/elife-110770-mdarchecklist1-v1.pdf
-
Source code 1
3D_stimulus_generator.py: Python custom code including OpenGL shader to create 3D experimental stimuli.
- https://cdn.elifesciences.org/articles/110770/elife-110770-code1-v1.zip
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Retinal curl as a functional signal for heading estimation beyond the focus of expansion
eLife 15:RP110770.
https://doi.org/10.7554/eLife.110770.3
