A hardware system for real-time decoding of in vivo calcium imaging data

DeCalcion is designed for conducting real-time calcium imaging experiments in freely behaving animals. Although the animal species, targeted brain regions, and behavioral tasks will necessarily differ depending on the goals of a given experiment, real-time imaging sessions conducted with DeCalciOn will generally be carried out by performing a sequence of five common steps (Figure 1B).

Step 1 is to mount the miniscope on the subject’s head and collect 1000 frames of data to derive a reference alignment image for online motion correction. It is preferable for the subject to be in a quiescent state during this initial data collection period, to minimize brain motion while the reference image data are being stored. Imaging and behavioral data are transmitted via Ethernet from the Ultra96 for storage on the host PC. After 1000 frames of data have been stored to the host PC (~50 s after the start of acquisition at the ~20 Hz frame rates used here), imaging acquisition automatically pauses and the user is prompted to select a 128 × 128 pixel subregion of the image to use for motion correction. The host PC derives a reference image for motion correction by computing the mean image within the selected area over the 1000 frames of stored data. The reference image is then uploaded to the Ultra96 and data collection resumes with online motion correction enabled. Collecting the alignment data and uploading the reference frame takes ~3 min.

Step 2 is to collect an initial set of motion-corrected data for training the decoder. Online motion correction vectors are derived in real time for each frame and uploaded to the host PC for storage along with imaging and behavioral data. The time required to collect initial training data depends upon how much data is needed to train the decoder, which in turn depends upon the quality of calcium signals and what behaviors the decoder will be trained to predict. In performance tests presented here (see below), asymptotic accuracy for decoding a rat’s position on a linear track was achieved with as little as ~3 min of training data whereas asymptotic accuracy for classifying categorical behaviors during an instrumental task required 20–40 min of training data (~100 experimental trials). The size of the training dataset required for a given experiment can be determined in advance (prior to real-time imaging sessions) by storing offline imaging data from the brain ROI while subjects are engaged in the experimental task, then carrying out offline simulations of real-time decoder performance with training sets of varying size to determine the minimum amount of training data needed for accurate decoding.

Step 3 is to pause data acquisition so that the host PC can perform analyses to extract calcium traces from each frame of the initial training dataset. Calcium traces can be extracted by two methods: a contour-based method that requires 30–60 min of processing time on the host PC during Step 3 (for reasons explained below), or a contour-free method that requires no processing time during Step 3 because this method allows calcium traces to be extracted online and stored to the host PC concurrently with collection of the initial training dataset, thereby obviating the need to devote additional time or effort to calcium trace extraction during Step 3. It is preferable for the miniscope to remain attached to the animal’s head throughout Step 3 (and Step 4), because removing and remounting the miniscope can introduce shifts in the focus or alignment of the brain image and thereby degrade the accuracy of subsequent real-time decoding. Since the contour-based method of calcium trace extraction consumes considerable processing time to perform offline identification of cell contours, it also incurs the inconvenience of monitoring the subject during this extended time period while the miniscope remains attached to the head.

Step 4 is to use extracted calcium traces for training a decoder to predict behavior from vectors of calcium trace values. The time required for this depends upon the complexity of the user-defined decoder that is being trained (the decoder is trained on the host PC and runs in real time on the Ultra96 ARM core; see ‘Stage 4: population decoding’). Thus, a multilayer deep network decoder would take longer to train than a single-layer linear classifier of the kind used here for performance testing. Training consumed <60 s for the linear classifiers used in performance tests presented below. Trained decoder weights (along with derived cell contour pixel masks if contour-based decoding was used) are uploaded from the host PC to the Ultra96 (a process which takes a maximum of 5 min to complete). Once the decoder weights have been uploaded, DeCalciOn is fully configured for real-time decoding of calcium traces.

Step 5 is to resume the behavior experiment while DeCalciOn performs real-time imaging and decoding of calcium traces to generate feedback output at TTL pins.

Figure 2 shows a schematic diagram of DeCalciOn’s real-time image processing pipeline, which performs four sequential stages of image processing and calcium trace decoding: (1) online motion stabilization, (2) image enhancement via background removal, (3) calcium trace extraction, and (4) fluorescence vector decoding. As explained below, Stages 1–3 are performed by our custom ACTEV hardware accelerator (Chen et al., 2019; Chen et al., 2020; Chen et al., 2022a) running in the programmable logic fabric of the FPGA, whereas Stage 4 is performed by a C++ program running under the FreeRTOS operating system on the MPSoC’s embedded ARM core.

Figure 2

Download asset Open asset

Stage 1: motion stabilization

Incoming frames arriving to the Ultra96 from the Miniscope DAQ are cropped to a 512 × 512 subregion, manually selected to contain the richest area of fluorescing neurons in each animal. Pixel data from the cropped subregion are stored to a BRAM buffer on the Ultra96 where it can be accessed by ACTEV accelerator running in the FPGA logic fabric. To correct for translational movement of brain tissue, a 128 × 128 pixel area with distinct anatomical features is manually selected during Stage 1 (Figure 1B) from within the 512 × 512 imaging subwindow to serve as a motion stabilization window. ACTEV’s image stabilization algorithm (Chen et al., 2019) rigidly corrects for translational movement of brain tissue by convolving the 128 × 128 stabilization window in each frame with a 17 × 17 contrast filter kernel, and then applying a fast 2D FFT/IFFT-based algorithm to correlate the stabilization window contents with a stored reference template (derived by averaging 1000 contrast-filtered frames from the beginning of each experimental session) to obtain a 2D motion displacement vector for the current frame. Video 1 demonstrates online performance of ACTEV’s real-time motion stabilization algorithm.

Video 1

Download asset

Stage 2: background removal

After motion stabilization, ACTEV removes background fluorescence from the 512 × 512 image by performing a sequence of three enhancement operations (Chen et al., 2020): smoothing via convolution with a 3 × 3 mean filtering kernel, estimating the frame background via erosion and dilation with a 19 × 19 structuring element (Lu et al., 2018), and subtracting the estimated background from the smoothed image. These operations produce an enhanced image in which fluorescing neurons stand out in contrast against the background (see Video 2; ‘Enhanced image’ in Figure 2). Calcium traces are then extracted (by methods described under ‘Stage 3’) from the motion-corrected (Stage 1) and background-removed (Stage 2) image.

Video 2

Download asset

Results presented below were obtained using two different versions of the UCLA Miniscope: CA1 position decoding tests were conducted using the Large Field-of-View (Blair et al., 2021) (MiniLFOV) version featuring 1296 × 972 pixel resolution sampled at 22.8 fps, whereas OFC instrumental behavior decoding tests were conducted using the V4 version featuring 600 × 600 pixel resolution sampled at 19.8 fps. For performance tests, we used a virtual sensor (see Methods) that was capable of feeding stored image data to the ACTEV accelerator at exactly the same frame rate (19.8 fps for V4, 22.8 fps for MiniLFOV) as raw video data arriving in real time. This allowed different real-time image processing algorithm implementations (e.g., contour-based vs. contour-free decoding) to be compared and benchmarked on the same stored datasets. Calcium traces obtained with the virtual sensor were verified to be identical with those obtained during real-time imaging sessions. ACTEV accelerators for virtual sensor testing and for real-time (non-virtual) imaging with both versions of the UCLA Miniscope (MiniLFOV and V4) can be found at https://github.com/zhe-ch/ACTEV, (Chen, 2023 copy archived at swh:1:rev:aa6393d3bd2dd490aa5369e1f2677e85e8a64a82).

Position decoding from CA1 cells

For the first set of performance tests, the MiniLFOV scope was used to image neurons in the hippocampal CA1 region while Long-Evans rats (n = 12) ran back and forth on a 250-cm linear track (Figure 3A). Each linear track session was ~7 min in duration, yielding 8K–9K frames of calcium trace data per rat. CA1 pyramidal neurons are known to behave as ‘place cells’ that fire selectively when an animal traverses specific locations in space (O’Keefe and Dostrovsky, 1971), so a rodent’s position can be reliably decoded from CA1 population activity (Ziv et al., 2013; Cai et al., 2016; Wilson and McNaughton, 1993; Kinsky et al., 2018; Center for Brains Minds Machines, 2017). As expected, many of the neurons we imaged in rat CA1 were spatially tuned on the linear track (see example tuning curves in Figure 3C, D; summary data in panels D, E of Supplement to Figure 3). For real-time decoding, we did not define a decision boundary for classifying CA1 traces as coming from ‘place’ versus ‘non-place’ cells; calcium traces were decoded from all ROIs regardless of their spatial tuning properties.

Figure 3 with 1 supplement see all

Download asset Open asset

When hippocampal activity is analyzed or decoded offline (rather than online), it is common practice to perform speed filtering that omits time periods when the rat is sitting still. This is done because during stillness, the hippocampus enters a characteristic ‘sharp wave’ EEG state during which place cell activity is less reliably tuned for the animal’s current location (Buzsáki, 2015). Here, speed filtering was not implemented during online decoding of CA1 fluorescence because the linear classifier received only real-time calcium trace data and not position tracking data that would be needed for speed filtering. As shown below, DeCalciOn achieved accurate decoding of the rat’s position from calcium traces without speed filtering.

ACTEV can utilize a real-time spike inference engine to convert each frame’s calcium trace value into an inferred spike count per frame. However, when decoding information from spike events (e.g., from single-unit recordings of neural population activity), it is standard practice to perform a temporal integration step by counting spikes in time bins of a specified width. In calcium imaging experiments, the GCamp molecule itself serves as a temporal integrator of spike activity. The GCamp7s indicator used in our CA1 imaging experiments has a decay constant of several hundred ms (Dana et al., 2019), which is similar to the width of time bins typically used for counting spikes when decoding a rat’s position from single-unit recordings of place cells (Ziv et al., 2013; Cai et al., 2016; Wilson and McNaughton, 1993; Kinsky et al., 2018; Center for Brains Minds Machines, 2017). In accordance with prior studies of position decoding from CA1 calcium fluorescence (Tu et al., 2020), we found (see panel A of Supplement to Figure 3) that the linear classifier was more accurate at decoding the rat’s position from raw calcium trace amplitudes (which chemically time-integrated spikes on a time scale of the several hundred milliseconds, corresponding to the GCaMP7s decay rate) than from inferred spike counts (which computationally time-integrated spike counts on a much shorter time scale of ~50 ms, corresponding to the miniscope’s frame interval). Hence, we decoded the rat’s position from raw calcium traces rather than inferred spike counts in performance tests reported below.

The linear classifier was trained on calcium trace and position data from the first half of each session, then tested on data from the second half of the session. The classifier’s output vector consisted of 12 units that used a Gray coding scheme (Figure 3B) to represent 24 position bins (each ~20 cm wide) along a circularized representation of the track. After the linear classifier had been trained on data from the first portion of the session, learned weights were uploaded from the host PC to the Ultra96 and image data from the second half of the session (test data) was fed through the virtual sensor to mimic real-time decoding. To verify that decoders predicted position with far better than chance accuracy, error performance was compared for decoders trained on calcium traces that were aligned versus misaligned with position tracking data (panel C, Supplement to Figure 3). Position decoding performance was compared for different methods of extracting online fluorescence traces including contour-based (CB) versus contour-free (CF) methods as explained below.

CB trace decoding

CB trace extraction derived calcium traces by summing fluorescence within pixel masks defining ROIs overlapping with identified neurons (Figure 3C; Video 2). Neurons were identified during the intermission period by using the CaImAn pipeline (Giovannucci et al., 2019) to perform constrained non-negative matrix factorization (CNMF) (Giovannucci et al., 2019; Pnevmatikakis et al., 2016) on the training dataset and extract ROIs for demixed calcium traces. Running CaImAn on 4K–5K frames of training data required 30–60 min of computing time on the host PC, making this the slowest step performed during the intermission period (see Figure 1B). After contour ROIs had been identified, an additional ~5 min of computing time was required to feed all of the training data frames through a simulator to reconstruct online calcium traces that would have been extracted in real time during the first half of the session using the extracted contour masks. The simulated online traces were then aligned frame-by-frame with position data and used to train a linear classifier for decoding the rat’s position on the linear track from calcium activity; training the decoder required a maximum ~60 s of computing time.

Accurate position decoding requires that calcium trace inputs to the decoder must retain stable spatial tuning across the training and testing epochs of the session. To verify that this was the case, spatial tuning curves were derived for CB calcium traces during the first (training) versus second (testing) half of each session (Figure 3D, left). Analyses of spatial selectivity and within-session spatial tuning stability (panels D, E in Supplement to Figure 3) indicated that large proportions of CB traces in CA1 exhibited stable spatial tuning between the training and testing epochs, as required for accurate decoding. The real-time decoder was trained and tested on CB traces from all ROIs identified by CaImAn in each session; we did not restrict the decoder’s training set to a subset of traces that met a spatial selectivity criterion.

Two measures were used to quantify accuracy of position predictions decoded from calcium traces in each frame: Distance error ${\displaystyle \lambda (t)}$ was the absolute distance (in cm) between the true versus decoded position in frame $t$; the mean distance error over frames in the testing epoch shall be denoted ${\displaystyle \overline{\lambda }}$ . Hit rate $\rho$ was the percentage of all frames in the test epoch for which the predicted position was within ±30 cm of the true position. Figure 3E shows that when the decoder was trained and tested with DeCalciOn’s online (non-demixed) CB traces, there was no significant difference in mean decoding accuracy (${\displaystyle \overline{\lambda }}$ : t₁₁ = −1.32, p = 0.21; $\rho$: t₁₁ = 1.32, p = 0.21) from when the same decoder was trained on demixed offline (‘Off’) traces extracted by CaImAn from the same set of ROIs. Hence, decoding from online (real-time) CB traces versus demixed offline traces yielded similarly accurate position predictions. Unsurprisingly, the accuracy of decoding position from calcium traces was proportional to the population size of detected neurons (${\displaystyle \overline{\lambda }}$ : R = −0.6, p = 0.0375; $\rho$: R₁ = 0.59, p = 0.0426; Figure 4E, right). Hence, the accuracy of position predictions improves with the number of calcium traces from which predictions are generated.

Figure 4

Download asset Open asset

CF decoding

In contrast with CB extraction, CF extraction derived calcium traces by summing fluorescence within pixel masks that did not overlap with identified neurons. Instead, ROIs were obtained by partitioning the 512 × 512 image into a 32 × 32 sheet of tiles, each tile measuring 16 × 16 pixels in area (Video 3); 124 tiles bordering the frame (shaded light blue in middle plot of Figure 3C) were excluded to eliminate edge contamination by motion artifacts. The CF extraction method always yielded a total of 900 calcium traces per rat (one for each square tile). By contrast, the CB method (see above) yielded a variable number of traces depending upon how many neuron ROIs were detected in the training dataset by CaImAn.

Video 3

Download asset

Unlike CB trace extraction, CF extraction does not require any computing time for contour identification or trace simulation during the intermission period, because the decoder is trained directly on 900 traces of CF data stored to memory during the training epoch; this training consumed <60 s of computing time on the host PC for 5K frames of training data. Hence, CF trace extraction required much less computing time during the intermission period than the CB trace extraction (which requires 30–60 min of computing time during intermission to identify trace ROIs, as explained above). Since CF calcium traces were derived from square tiles that were not aligned with locations of individual neurons in the sensor image, each CF trace could be modulated by fluorescence from multiple neurons within the ROI of its assigned tile. CF traces are thus analogous to multiunit spike recordings in neurophysiology that combine signals from multiple neurons. Similarly, CB traces can be regarded as analogous with single-unit recordings, since they are better at isolating signals from single neurons.

Like CB traces, CF traces in CA1 also exhibited stable spatially selective tuning (Figure 3D, middle). However, individual CF traces showed less within-session stability and fired over wider regions of the track (panels D, E in Supplement to Figure 3) than individual CB traces, as should be expected if CF traces contain signals from multiple place cells and CB traces isolate signals from individual place cells. But at the population level, decoding the rat’s position from CF traces was as or more accurate than decoding from CB traces. As in the case of CB traces, the real-time decoder was trained and tested on all CF traces (N = 900 per rat in the tile mosaic) regardless of their spatial selectivity. When the accuracy of CF versus CB decoding was compared by performing paired t-tests on the ${\displaystyle \overline{\lambda }}$ and $\rho$ measures defined above (Figure 3F, left), CF decoding was more accurate than CB decoding (${\displaystyle \overline{\lambda }}$ : t₁₁ = −3.43, p = 0.0056; $\rho$: t₁₁ = 4.6, p = 7.49e⁻⁴). Analogous results have been reported in electrophysiology, where it has been found that decoding from multiunit spike trains can be more accurate than decoding from sorted spikes sourced to single units (Deng et al., 2015).

Since decoding accuracy for CB traces was proportional to population size (Figure 3E, right), it is possible that the reason CF traces provided better decoding accuracy was because they were greater in number (in all rats, there were more CF than CB traces). To test this, CaImAn’s sensitivity parameters were adjusted to detect more CB traces in each rat, yielding an expanded population (CB+) of CB traces that was larger than the original population of CB traces in each rat (Figure 3C, right). Figure 4E shows that CB+ decoding was more accurate than CB decoding (${\displaystyle \overline{\lambda }}$ : t₁₁ = −3.09, p = 0.0013; $\rho$: t₁₁ = 3.7, p = 0.0035), but did not differ significantly in accuracy from CF decoding (${\displaystyle \overline{\lambda }}$ : t₁₁ = −1.08, p = 0.3; $\rho$: t₁₁ = 0.83, p = 0.42). Hence, CF and CB+ traces yielded similar decoding accuracy, suggesting that the number of calcium trace predictors was a strong determinant of decoding accuracy. Deriving CB+ traces required about the same amount of computing time (30–60 min) during the intermission period as CB traces to identify contours and simulate online traces (see Figure 1B). Therefore, despite the similar accuracy of CF and CB+ traces, it was much more efficient to use CF traces since they did not require extra computing time to identify ROIs during the intermission period.

Duration of the training epoch

An initial dataset must be collected at the beginning of each session to train the decoder before real-time imaging begins. The training epoch should be long enough to obtain sufficient data for training an accurate decoder, but short enough to leave sufficient remaining time in the session for real-time imaging. To analyze how position decoding accuracy on the linear track depended upon the duration of the training epoch, we varied the length of the training epoch in increments of 500 frames (see Methods). As expected, position decoding accuracy (quantified by $\stackrel{-}{\lambda }$ and $\rho$) improved with the duration of the training epoch (Figure 3G). On average, asymptotic accuracy occurred when the training epoch reached ~3000 frames, beyond which increasing the duration of the training epoch yielded diminishing benefits for decoding accuracy.

Online motion correction

Online motion stabilization (Stage 1) is the most computationally expensive stage in the real-time image processing pipeline, consuming more MPSoC hardware resources than any other stage (Chen et al., 2022a). To quantify the benefits of these high resource costs, we compared decoder performance with (mc+) and without (mc−) motion correction using CB or CF methods for CA1 calcium trace extraction (Figure 4). For these analyses, the decoder was trained on mc+ or mc− traces from the first 3000 frames in each session, and then tested on the corresponding mc+ or mc− traces from the remainder of the session.

Effects of motion correction upon session-averaged distance error and hit rate were computed using the formulas Δ ${\displaystyle \overline{\lambda }}$ = ${\displaystyle \overline{\lambda }}$_mc−mc+ and $\Delta \rho$ = $\rho$_mc+$-\rho$_mc−, respectively (note that both formulas yield a positive result for cases where motion correction improves accuracy, and a negative result for cases where motion correction degrades accuracy). Paired t-tests found no difference between mc+ versus mc− conditions for Δ ${\displaystyle \overline{\lambda }}$ or $\Delta \rho$ (Figure 4A), but session-averaged decoding accuracy may not be the most appropriate measure for motion correction’s benefits because image stabilization is only beneficial when the image is translationally displaced from its reference alignment with the sensor, and this only occurs during a subset of frames within a session. To further assess the benefits of online motion correction, we analyzed how image stabilization reduced two distinct types of alignment error that can arise from translational displacement of the image on the sensor: shift error and jitter error.

Shift error

Shift error occurs when the image permanently changes its steady-state alignment with the miniscope sensor (either suddenly or gradually) during the course of a session. For example, this can occur if the miniscope shifts its seating in the baseplate during a session. Significant shift error occurred in only 1 of the 12 linear track sessions analyzed for our performance tests (Rat #1 in Figure 4C); this session is marked by filled circles in panels of Figure 4. For this one session (but no others), benefits of motion correction were apparent even in the session-averaged data (Figure 4A). Hence, online motion stabilization was effective at correcting shift errors in the one session where they occurred; but given that shift errors occurred so rarely, it might be argued that correcting these errors does not adequately justify resource costs of online motion stabilization, especially if shift errors can be effectively prevented by other means such as rigidly attaching the miniscope to the baseplate.

Jitter error

Jitter error occurs when the animal’s inertial head acceleration causes transient motion of the brain inside the skull cavity, resulting in phasic translation of the brain image across the sensor. By definition, jitter error is temporary and lasts only until the brain image returns to its original alignment with the sensor after inertial motion stops. Jitter error can occur when the animal jerks or angles its head in such a way that the brain moves within the skull. Motion of the brain inside the skull is a natural phenomenon during free behavior, and for this reason, jitter error cannot easily be prevented by means other than online motion stabilization. To analyze whether online stabilization was effective at correcting for jitter error, we used the distance formula to compute displacement of each frame’s non-motion-corrected image against the reference alignment: $d_t=\sqrt{\Delta x_t^2+\Delta y_t^2}$ . Two Pearson correlation coefficients (R_mc+ and R_mc−) were derived for each session by correlating the distance error vectors ($\lambda$_mc+ and $\lambda$_mc−) with the displacement vector d over all frames in the testing epoch for the session. Positive R values indicated that distance error was larger during frames that were misaligned on the sensor, as would be expected if decoding accuracy were impaired when jitter artifact corrupted the values of calcium traces. Conversely, negative R values indicated that decoding error was smaller during frames that were misaligned on the sensor, which might occur if jitter error was correlated with the rat’s position on the track in such a way that the decoder artifactually learned to predict the rat’s location from image displacement rather than from neural calcium activity. R values from each session were converted into motion artifact error scores using the formula Ω = −log₁₀ (P) × sign(R), where R is the correlation between distance error and image displacement and P is the significance level for R; the p < 0.01 significance level for the similarity score is S > 2, since log₁₀(0.01) = 2.

Figure 4B shows scatterplots of Ω_mc+ versus Ω_mc− derived from CB (top) or CF (bottom) traces for all 12 linear track sessions in the performance testing analysis. Sessions falling within the central gray square region (3/12 for CB traces, 6/12 for CF traces) had non-significant motion artifact error scores before as well as after online motion correction, indicating that for these sessions, there was little benefit from online motion stabilization. Sessions falling within the upper yellow region (6/12 for CB traces, 2/12 for CF traces) had significant positive motion artifact error scores in the mc− condition that resolved in the mc+ condition, indicating that in these sessions, online motion correction successfully prevented degradation of decoding accuracy by motion artifact. Sessions falling within the lower blue region (1/12 for CB traces, 0/12 for CF traces) had significant ‘negative’ motion artifact error scores in the mc− but not mc+ condition, suggesting that without motion stabilization, the decoder artifactually learned to predict position from location-dependent motion artifacts rather than from neural calcium activity; this ersatz decoding was reduced by online motion correction. Further examples of decoders learning to predict behavior from image motion are shown below (see ‘Decoding instrumental behavior from OFC neurons’).

In Figure 4B and a small number of plotted points fell outside any of the shaded regions indicating that significant motion artifact scores occurred in both the mc− and mc+ conditions. However, the mean distance of points from the vertical midline (D_V) was significantly smaller than their mean distance (D_H) from the horizontal midline in scatterplots of both CB (t₁₁ = 2.63, p = 0.0232) and CF (t₁₁ = 2.92, p = 0.0139) traces. This indicates that correlations (positive and negative) between distance error and motion artifact were significantly reduced by online motion correction of both CB and CF traces. Hence, online motion stabilization benefitted decoding accuracy by improving the accuracy of online predictions for frames with motion artifact.

Another set of performance tests was carried out using the V4 UCLA Miniscope to image neurons in OFC while Long-Evans rats (n = 2) performed a 2-choice instrumental touchscreen task (Figure 5A). Each trial of the task began with the appearance of a central fixation stimulus (circle) on the screen, which the rat was required to touch. After touching the center stimulus, two visually distinct choice stimuli (A and B) appeared on the left and right sides of the screen. Assignment of stimuli A/B to the left/right side of the screen was randomized on each trial, but the correct response was always to touch the stimulus on one side (left for JL66, right for JL63) regardless of its identity. A correct response immediately triggered audible release of a food pellet into a magazine located behind the rat; an incorrect response triggered a 5-s timeout period. After each rewarded or non-rewarded outcome, a 10-s intertrial interval was enforced before initiating the next trial. Figure 5B (top) summarizes behavioral performance during 45-m imaging sessions (one for each rat) conducted after training on the discrimination task. In both sessions, rats performed >150 trials with >95% correct choices.

Figure 5

Download asset Open asset

In both rats, real-time decoding of reward retrieval and correct choice events was similarly accurate (from both CB and CF traces) regardless of whether online motion correction was used (mc+ vs. mc−; Figure 5D–F). This suggests that online motion correction did not benefit decoding accuracy. However, in both rats, jitter error (quantified by $\left|d\right|$) was significantly greater during reward retrieval than during other behaviors (Figure 6A), indicating that retrieval of food from the magazine hopper caused the brain to move inside the skull. In rat JL63 (but not JL66), $\left|d\right|$ was also greater during correct choice events than during other behaviors (Figure 6B), indicating that pressing the touchscreen may also have sometimes caused the brain to move inside the skull. Given that brain motion was significantly correlated with some behaviors, it is important to consider the possibility that in the absence of motion correction, the decoder may have learned to predict behavior from motion artifacts rather than from calcium activity. If so, then even though prediction accuracy remains high without motion correction, it would still be necessary to perform motion correction to assure that the real-time decoder derives its predictions from calcium activity rather than motion artifacts.

Figure 6

Download asset Open asset

To investigate this, we isolated all frames that occurred during a given type of behavior event (reward retrieval or correct choice), and then performed rank sum tests to compare the median value of $\left|d\right|$ from frames in which the behavior event was correctly classified (‘predicted’) versus frames in which it was not (‘unpredicted’). Without motion correction, the median jitter error $\left|d\right|$ in both rats was significantly higher for frames in which CB traces correctly predicted reward retrieval (Figure 6C, top graphs) or correct choice (Figure 6D, top graphs); this effect was abolished after motion correction (Figure 6C, D; bottom graphs). Similarly, without motion correction the median $\left|d\right|$ in both rats was significantly higher for frames in which CF traces correctly predicted reward retrieval (Figure 6E, top graphs) or correct choice (Figure 6F, top graphs); this effect was also abolished after motion correction (Figure 6E, F; bottom graphs) in all but one case (reward retrieval predictions in JL63).

DeCalciOn’s real-time feedback latency, ${\tau }_F$ , is equal to the time that elapses between photons hitting the miniscope’s image sensor and the rising edge of the TTL output signal that is triggered when the decoder detects a target pattern of neural activity. The total feedback latency is the sum of three delays that accrue sequentially: ${\tau }_F$ = ${\tau }_L$ + ${\tau }_T$ + ${\tau }_I$ , where ${\tau }_L$ is the light-gathering delay, ${\tau }_T$ is the frame transmission delay, and ${\tau }_I$ is the image processing delay.

Light-gathering delay (${\tau }_L$)

The first delay in the real-time decoding sequence arises from the time it takes for the sensor to gather light. Each image frame is formed on the sensor by summing light from photons that arrive during an exposure time window that is ~1 ms shorter than the frame interval, I. For example, at a frame rate of F ≈ 20 Hz (as in experiments presented here) the frame interval is I ≈ 50 ms, so the exposure time would be (50-1) ≈ 49 ms, which is slightly shorter than I because the sensor’s circuits require ~1 ms to perform analog-to-digital (A/D) conversion of pixel values at the end of each exposure. Hence, at each end-of-frame, the captured image integrates the intensity of light that has hit the sensor during the preceding frame interval. This binned averaging of light intensity is tantamount to convolving the ‘true’ brain fluorescence signal with a boxcar kernel of width I, effectively shifting the sampled fluorescence signal rightward in time from the true fluorescence signal by a delay of ${\tau }_L$ = I/2 (Figure 7A). Thus, at the frame rate of 20 Hz used here, ${\tau }_L$ ≈ 50/2 = 25 ms.

Figure 7

Download asset Open asset

Most in vivo calcium imaging experiments utilize offline algorithms to analyze calcium trace data that has previously been collected and stored during a behavioral task. Popular offline analysis packages such as CaImAn (Giovannucci et al., 2019) and MIN1PIPE (Deng et al., 2015) employ sophisticated algorithms like CNMF (Pnevmatikakis et al., 2016) to demix crossover fluorescence and source calcium traces to single neurons. Accurate source identification is important for studies where investigating the tuning properties of individual neurons is a major aim of data analysis, but DeCalciOn is designed for real-time imaging experiments in which fast and accurate decoding of behavior from neural population activity takes precedence over characterizing single-neuron firing properties. For real-time decoding, the high computational costs of source identification may not yield proportional benefits to decoding accuracy. Supporting this, we found that linear classifiers trained on calcium traces extracted online (without demixing) predicted behavior just as accurately as those trained on traces extracted offline (with demixing) using the same set of ROIs (Figure 3E; Figure 5D–F). We also found that decoders trained on traces extracted from contour-free ROIs (which do not overlap with individual neurons) predicted behavior as or more accurately than traces extracted from contour-based ROIs that overlapped with identified neurons in the image frame (Figure 3F and G; Figure 4A and B; Figure 5D–F). Analogous results have been reported in electrophysiology, where ‘spike sorting’ algorithms for sourcing action potentials to single neurons incur high computational overhead costs that are comparable to the costs of sourcing calcium traces to single neurons with CNMF; decoding behavior from electrophysiological recordings of multiunit (unsorted) spikes is more computationally efficient than decoding from sorted spikes and often incurs no cost (or even a benefit) to decoding accuracy (Deng et al., 2015). Results presented here indicate that as in electrophysiology studies, the high computing cost of source identification in calcium imaging studies must be weighed against the benefits. For real-time imaging studies, contour-free decoding approaches can offer a strategy for optimizing computing efficiency without compromising decoding accuracy.

After source identification, the most computationally expensive stage in the image processing pipeline is motion stabilization. DeCalciOn reduces motion stabilization costs by using a motion correction subwindow (128 × 128 pixels, see Figure 2) that is considerably smaller than the 512 × 512 pixel imaging window (yet still large enough to achieve reliable motion correction). Even so, deriving each frame’s motion displacement vector at millisecond latencies consumes more MPSoC hardware resources than any other stage within the image processing pipeline (Chen et al., 2022a). To quantify the benefits of these high resource costs, we compared decoder performance with and without motion correction. We found that online motion stabilization was effective at preventing two problems that can arise from instability of the brain image on the sensor. First, when decoding a rat’s position on the linear track, online motion correction prevented decoding accuracy from becoming degraded during frames in which brain motion shifted the sensor image, as evidenced by the fact the positive correlations between decoding error and image displacement were reduced by online motion correction (Figure 4). Second, in the instrumental touch screen task where brain motion was significantly correlated with specific behaviors, the decoder learned to predict behavior from image motion (rather than calcium fluorescence) in the absence of motion correction; the decoder was prevented from learning these ersatz predictions when the online motion correction algorithm was applied to the decoder’s training and testing datasets (Figure 5D–F; Figure 6).

Motion correction is a transient phenomenon and is therefore not needed during every image frame. Illustrating this, session-averaged accuracy for decoding position on the linear track was not significantly improved by online motion correction (Figure 4A). However, for real-time feedback experiments, the session-averaged prediction accuracy is not the most important measure of decoder performance. What matters most is the accuracy of decoding during specific frames when it is important for feedback stimulation to be delivered or withheld. If such frames happen to overlap with motion artifact that interferes with decoding accuracy, then failure to perform motion correction can have disastrous consequences for feedback experiments, regardless of whether or not motion correction improves the ‘average’ decoding accuracy. Based on these considerations and the results of our performance tests, we conclude that the high resource costs of online motion correction are well worth the benefits they accrue by increasing the reliability of online decoding of neural population activity.

Several previous online calcium fluorescence trace extraction algorithms have been proposed; some have been validated only by offline experiments showing that they can generate online traces that rival the quality of offline traces, and have not yet been performance tested on hardware platforms to assess their performance at real-time imaging or decoding of calcium traces (Friedrich et al., 2017; Friedrich et al., 2021; Taniguchi et al., 2021). Other systems have been performance tested in real time, as we have done here for DeCalciOn. For example, Liu et al., 2021 introduced a system for real-time decoding with the UCLA miniscope that, unlike DeCalciOn, requires no additional hardware components because it is implemented entirely by software running on the miniscope’s host PC. The system does not perform online motion stabilization, which is a significant drawback for reasons discussed above (see ‘Online motion correction’). Liu et al., 2021 showed that their system can implement a single binary classifier to decode calcium traces from 10 ROIs in mouse cortex at latencies <3 ms; this latency measurement appears to only include the decoder’s execution time for generating predictions from calcium vectors (equivalent to Step 4 of our image processing pipeline) but not the light-gathering delay time (Figure 7A) or data buffering time. Liu et al., 2021 did not report how their system’s latency scales with the number of classifiers or the size/number of calcium ROIs, but under the assumption that serial processing time on the host PC scales linearly with these variables, decoding latencies of several seconds or more would be incurred for the decoding stage alone if 12–24 classifiers were used to predict behavior from 1204 calcium traces, which is much longer then the millisecond latencies achieved by DeCalciOn in performance tests presented here (see Figure 7C, D).

Zhang et al., 2018 performed 2-photon calcium imaging experiments that incorporated real-time image processing on a GPU to detect neural activity and trigger closed-loop optical feedback stimulation in mouse cortex. The time required for calcium trace extraction increased linearly with a slope of ~0.05 ms per ROI (from a y-intercept of 0.5 ms; see panel C of Supplementary Fig. 6 in Zhang et al., 2018,), implying a total trace extraction time of ~52 ms for 1204 ROIs. Summing this latency with the light-gathering delay (~15 ms at the system’s default 30-Hz frame rate), data buffering delay (reported to be ~2.5 ms), online motion correction delay (reported to be ~3.5 ms), and decoder execution time (~2 ms), it can be estimated that when decoding 1024 traces, Zhang et al., 2018 system would incur a delay of ~75 ms between light arriving at the sensor and triggering of closed-loop feedback. This latency is only 25 ms (~50%) longer than DeCalciOn’s 47.5-ms decoding latency under similar conditions (Figure 7), and could possibly be improved using parallelization or other efficiency strategies. Although latency performance is similar for DeCalciOn’s FPGA-based pipeline and Zhang et al., 2018 GPU-based pipeline, the two systems have different hardware compatibilities: DeCalciOn is designed for compatibility with head-free miniscopes that use the UCLA DAQ interface, whereas Zhang et al., 2018 system is designed for compatibility with hardware and software interface standards used in benchtop multiphoton imaging systems.

DeCalciOn’s low-cost, ease of use, and latency performance compare favorably against other real-time imaging systems proposed in the literature. We hope that by making DeCalciOn widely available to the research community, we can provide a platform for real-time decoding of neural population activity that will facilitate novel closed-loop experiments and accelerate discovery in neuroscience and neuroengineering. All of DeCalciOn’s hardware, software, and firmware are openly available through miniscope.org.

A total of 14 Long-Evans rats (6F, 6M for CA1 experiments, 2M for OFC experiments) were acquired from Charles River at 3 months of age. Subjects were singly housed within a temperature and humidity controlled vivarium on a 12-hr reverse light cycle. Surgical procedures began after a 1-week acclimation period in the vivarium, and recordings and behavioral experiments began around 5 months of age. All experimental protocols were approved by the Chancellor’s Animal Research Committee of the University of California, Los Angeles, in accordance with the US National Institutes of Health (NIH) guidelines.

After rats had been baseplated, they were placed on food restriction to reach a goal weight of 85% ad lib weight and then began behavioral training. Time between the beginning of the surgical procedures and the start of behavior training was typically 6–8 weeks. Rats earned 20 mg chocolate pellets by alternating between two rewarded ends of a linear track (250 cm) during 15-min recordings beginning 5 days after baseplating. After receiving a reward at one end, the next reward had to be earned at the other end by crossing the center of the track. A webcam mounted in the behavior room tracked a red LED located on the top of the Miniscope and this video was saved alongside the calcium imaging via the Miniscope DAQ software with synchronized frame timestamps. These behavior video files were initially processed by custom python code, where all the session videos were concatenated together into one tiff stack, downsampled to the video imaging frame rate, the median of the stack was subtracted from each image, and finally they were all rescaled to the original 8-bit range to yield the same maximum and minimum values before subtraction. Background subtracted behavior videos were then processed in MATLAB. The rat’s position in each frame was determined using the location of the red LED on the camera. Extracted positions were then rescaled to remove the camera distortion and convert the pixel position to centimeters according to the maze size. Positional information was then interpolated to the timestamps of the calcium imaging video using a custom MATLAB script.

Before pretraining, rats were acclimated to a gentle hold on the lap of the experimenter sitting adjacent to the test chamber. This acclimation took place over at least 1 week prior to an imaging session wherein the rat was acclimated to unscrewing the cover of the baseplate on the rat’s head, then placing the miniscope on the baseplate while on and imaging. After checking placement, the miniscope was screwed onto the baseplate. The rat was then placed in the operant chamber and the miniscope was plugged into the commutator. Pretraining schedules were then initiated that culminated in independent nosepokes to the touchscreen (Harris et al., 2021). Following criterion of 60 committed trials or more during a 45-min session, rats were advanced to the main instrumental task (Figure 5A). Rats were required to initiate a trial by touching the white graphic stimulus in the center screen (displayed for 40 s), and after initiation rats would be presented with two identical stimuli (i.e., fan or marble) on the left and right side of the screen (displayed for 60 s), that they were required to nosepoke as either the correct spatial side (p_R(B) = 1; rewarded with one sucrose pellet) or incorrect spatial side (p_R(W) = 0). Thus, rats were required to ignore the properties of the stimuli and determine the better-rewarded side.

At the end of the experiment, rats were anesthetized with isoflurane, intraperitoneally injected with 1 ml of pentobarbital, then transcardially perfused with 100 ml of 0.01 M phosphate-buffered saline (PBS) followed by 200 ml of 4% paraformaldehyde in 0.01 M PBS to fix the brain tissue. Brains were sectioned at 40-µm thickness on a cryostat (Leica), mounted on gelatin prepared slides, then imaged on a confocal microscope (Zeiss) to confirm GFP expression and GRIN lens placement.

The DeCalciOn system is designed for use with UCLA Miniscope devices (Cai et al., 2016) and other miniscopes (Scott et al., 2018; de Groot et al., 2020; Blair et al., 2021) that use the UCLA DAQ interface. The DAQ hardware requires modification by soldering flywires to the PCB (see Figure 1); instructions for doing this are available at https://github.com/zhe-ch/ACTEV (Chen, 2023 copy archived at swh:1:rev:aa6393d3bd2dd490aa5369e1f2677e85e8a64a82) and pre-modified DAQ boards are available at miniscope.org. DeCalciOn is implemented on an Avnet Ultra96 development board (available from the manufacturer at Avnet.com), which must be mated to our custom interface board available at miniscope.org (the interface can also be ordered from a PCB manufacturer with design files available at https://github.com/zhe-ch/ACTEV) (Chen, 2023 copy archived at swh:1:rev:aa6393d3bd2dd490aa5369e1f2677e85e8a64a82). ACTEV compiled hardware (for online image stabilization, image enhancement, calcium trace extraction) and embedded software (for real-time decoding and ethernet communication, which runs under the FreeRTOS operating system) can be programmed onto the Ultra96 board by copying a bootable bitstream file to a MicroSD card and then powering up the board with the card inserted. The V4 and MiniLFOV versions of the bitstream used for experiments reported here are available at https://github.com/zhe-ch/ACTEV., (Chen, 2023 copy archived at swh:1:rev:aa6393d3bd2dd490aa5369e1f2677e85e8a64a82) Bitstream files for other miniscope models and Vivado HLS C source code for generating new custom bitstream files are available on request.

Real-time imaging and decoding are controlled by custom RTI software (available at https://github.com/zhe-ch/ACTEV) (Chen, 2023 copy archived at swh:1:rev:aa6393d3bd2dd490aa5369e1f2677e85e8a64a82) which runs on the host PC alongside standard DAQ software (available at miniscope.org) for adjusting the gain and focus of the Miniscope device. The DAQ software’s default operation mode is to receive and display raw Miniscope video data from the DAQ via a USB 3.0 port (and store these data if the storage option has been selected), and also to receive and display raw behavior tracking video from a webcam through a separate USB port. At the start of a real experimental session, data acquisition by both programs (DAQ and RTI software) is initiated simultaneously with a single button click in the RTI user interface, so that Miniscope video storage by the RTI software and behavior video storage by the Miniscope DAQ software are synchronized to begin at exactly the same time. This allows behavioral data stored by the DAQ software to be aligned with Miniscope video and calcium trace data stored by the RTI software. During the intermission period between initial data acquisition and real-time inference, data stored by DAQ and RTI software are used to train the linear classifier on the host PC. Trained classifier weights are then uploaded to the Ultra96 for real-time decoding.

Pixel masks for contour-based trace extraction were derived using the CaImAn (Giovannucci et al., 2019) pipeline (implemented in python) to analyze motion-corrected sensor images from the training dataset (as noted in the main text, this took 30–60 min of computing time on the host PC). CaImAn is an offline algorithm that uses CNMF to isolate single neurons by demixing their calcium traces. The CNMF method is acausal so it cannot be used to extract traces in real time. But during offline trace extraction, CaImAn generates a set of spatial contours identifying pixel regions from which each demixed trace was derived, which ACTEV then uses as pixel masks for online extraction of contour-based traces. Once pixel masks were identified from the training data, motion-correct miniscope video from the training period was passed through an offline simulator that used ACTEV’s causal algorithm for extracting calcium traces from contour pixel masks. This yielded a set of calcium traces identical to those that would have been extracted from the training data in real time by ACTEV. These simulated traces were then used as input vectors to train the linear classifier. Since the online traces are not generated by CNMF (and are thus not demixed from one another), they are susceptible to contamination from fluorescence originating outside of contour boundaries. However, this crosstalk fluorescence did not impair decoding since similar accuracy results were obtained by training the linear classifier on online or offline contour-based traces (Figure 4F, G).

To implement contour-free trace extraction, we simply partitioned the 512 × 512 image frame into a 32 × 32 sheet of tiles, each measuring a square of 16 × 16 pixels (Figure 4B). No traces were extracted from 124 tiles bordering the edge of the frame, to avoid noise artifacts that might arise from edge effects in the motion stabilization algorithm. Hence, a total of 1024–124 = 900 pixel mask tiles were used for contour-free calcium trace extraction. These traces were derived in real time and stored to the host PC throughout the initial data acquisition period, so they were immediately available for training the linear classifier at the start of the intermission. Consequently, an advantage of contour-free trace extraction is that the intermission period between training and testing is shortened to just a few minutes, because the lengthy process of contour identification is no longer required. A disadvantage of contour-free trace extraction is that contour tiles do not align with individual neurons in the sensor image. As reported in the main text, this lack of alignment between neurons and pixel masks did not impair (and often enhanced) position decoding; however, contour-free decoding does place limits upon what can be inferred about how single neurons represent information in imaged brain regions.

After pixel masks were created using one of the two methods (contour-based or contour-free) and uploaded to the FPGA, calculations for extracting calcium traces from the masks were the same. Each mask specified a set of pixels over which grayscale intensities were summed to obtain the fluorescence value of a single calcium trace:

where T(f) is the summed trace intensity for frame f, and p_i(f) is the intensity of the ith pixel in the mask for frame f, and P is the number of pixels in the mask. Each contour-free mask was a square tile containing 16 × 16 = 256 pixels (Figure 4B), whereas the size of each contour-based mask depended upon where CaImAn identified neurons in the image (Figure 4A).

The mean size of contour-based masks was 20–100 pixels (depending on the rat), which was an order of magnitude smaller than contour-free tiles. In a few rats, a small amount of jitter sometimes penetrated the online motion correction filter, causing the stabilized sensor image to slip by 1–2 pixels against stationary contour masks during 1–2 frames (see Video S1, line graph at lower right). This slippage misaligned pixels at the edges of each contour mask, producing intermittent noise in the calcium trace that was proportional to the fraction of misaligned pixels in the contour, which in turn was inversely proportional to the size of the contour, so that motion jitter caused more noise in traces derived from small contour masks than large contour masks. To filter out this occasional motion jitter noise from calcium traces, a drop filter was applied to traces derived from contour masks that contained fewer than 50 pixels. The drop filter exploited the fact that genetic calcium indicators have slow decay times, and therefore, sudden drops in trace fluorescence can be reliably attributed to jitter noise. For example, the GCamp7s indicator used here has a half decay time O’Keefe and Dostrovsky, 1971 of about 0.7 s, so at a frame rate of ~20 Hz, a fluorescence reduction of more than 5% between frames can only arise from jitter artifact. The drop filter defines a maximum permissible reduction in fluorescence between successive frames as follows:

where G is the maximum possible fluorescence intensity for any single pixel (255 for 8-bit grayscale depth), C is the number of pixels in the contour mask for the trace, and q is a user-specified sensitivity threshold, which was set to 0.9 for all results presented here. The drop-filtered calcium trace value for frame f was given by:

It should be noted that while drop filtering protects against artifactual decreases in trace values, it offers no protection against artifactual increases that might masquerade as neural activity events. However, motion jitter almost never produced artifactual increases in fluorescence for the small contours to which drop filtering was applied, because small contours were ‘difficult targets’ for patches of stray fluorescence to wander into during jitter events that rarely exceeded 1–2 pixels of slippage. Larger contours were slightly more likely to experience artifactual fluorescence increases during jitter events, but in such cases, the artifact was diluted down to an inconsequential size because it only affected a tiny fraction of the large contour’s pixels. In summary, jitter artifact was highly asymmetric, producing artifactual decreases but not increases for small contours, and producing negligible artifact of any kind for large contours.

Once trace values have been extracted (and drop filtered if necessary), ACTEV can apply a real-time spike inference engine to convert raw calcium trace values into inferred spike counts for each frame. Briefly, the spike inference engine measures how much the trace value in the current frame has changed from the prior frame, T(f) − T(f − 1), and compares this against a threshold value, Φ, which is equal to 2.5 times the standard deviation of difference values between successive frames of the same trace in the training dataset. For the CA1 data reported here, we found that the linear classifier was more accurate at decoding raw (unconverted) calcium traces than inferred spike counts (panel A of Supplementary to Figure 3). Decoding from raw calcium traces is not only more accurate but also more computationally efficient; therefore, the main text reports results obtained by decoding raw calcium trace values.

To train the linear classifier, data stored during the initial acquisition period by the DAQ and RTI software were analyzed during the intermission period by custom MATLAB scripts, available at https://github.com/zhe-ch/ACTEV, (Chen, 2023 copy archived at swh:1:rev:aa6393d3bd2dd490aa5369e1f2677e85e8a64a82). Different linear classifier architectures were used for position versus behavioral decoding, as explained below. Training vectors and target outputs were passed to MATLAB’s fitceoc function (from the machine learning toolbox) to compute the trained linear classifier weights, which were then uploaded from the host PC to the Ultra96 via ethernet, so that the real-time classifier running under FreeRTOS in the ARM core could decode calcium traces in real time.

Once classifier weights had been uploaded to the Ultra96 from the host PC, it was then possible for the embedded ARM core in the Ultra96 MPSoC to read in calcium trace values from the DRAM in real time as each frame of data arrived from the Miniscope, and present these values as inputs to the linear classifier. The ARM core was programmed in C++ to perform calculations equivalent to MATLAB’s predict function (from the machine learning toolbox) and thereby generate the linear classifier output vector. For predicting the rat’s position on the track from CA1 activity, the decoded position bin was taken to be that represented by the Gray code vector that most closely matched the classifier output vector. For categorizing the rat’s behavior from OFC activity, the predicted behavior label was taken to be the label that had been most often selected by the decoder during a five-frame window that included the current and previous four frames. OFC decoding performance was assessed by computing sensitivity (the overall percentage of behavior events during the test epoch that were correctly predicted) and F-score (a standard measure of binary classification accuracy computed as 2PS/(P + S), where S is the sensitivity and P is the precision defined as the number of correct positive results divided by the number of all positive results). Decoder predictions were sent back to the host PC via ethernet for storage, and were also sent to a programmable logic mapper running on the ARM core, which converted the linear classifer’s output vector into a pattern of square pulses generated at on of 8 TTL pinouts from the Ultra96, capable of driving closed-loop neural or sensory feedback via external devices such as lasers, electrical stimulators, or audiovisual display equipment.

To generate spatial tuning curves for fluorescence traces, each trace was first referenced to zero by subtracting its minimum observed fluorescence value (over all frames) from the value in every frame: $Z\left(f\right)=T\left(f\right)-min\left(T\right)$, where Z(f) is the zero-referenced trace value for frame f. The 250-cm linear track was subdivided into 24 evenly spaced spatial bins (12 each in the left-to-right and right-to-left directions; see Figure 2A). Fluorescence activity, A, in each bin b was averaged by

where B is the number of visits to bin b and f_i is the ith frame during which the rat was visiting bin b. The tuning curve vector [A₁, A₂, …, A₂₄] was then normalized by dividing each of its elements by the value of the maximum element. Normalized tuning curve vectors were plotted as heatmaps in Figure 2C, D. Spatial similarity scores (S) were computed as described in the main text; S distributions for each rat are shown in Extended Data Figure 3.

Two Pearson correlation coefficients (R_mc+ and R_mc−) were derived for each session by correlating the distance error vectors ($\lambda$_mc+ and $\lambda$_mc−) in each frame with the displacement vector d = {d₁, d₂, ..., d_F}, where F is the total number of frames in the session. A necessary condition for computing reliable R values was that $\lambda$ and $d$ had to be sufficiently dispersed along their respective axes to detect a significant linear relationship between the variables. However, within any given session, there were often some position bins in which motion artifact either never occurred or was so small in magnitude that it was impossible to assess the impact of motion artifact upon decoding accuracy at that position. For this reason, the motion-accuracy correlation analysis was performed only on frames from selected position bins in which the maximum value of $d$ exceeded a threshold of 4 pixels, which was a large enough displacement to alter the values of CB trace values extracted from images without motion correction. All frames from position bins meeting this criterion (and no frames from position bins that did not) were included in the correlation analysis. Hence, analysis was restricted to those locations on the track where trace values (and thus distance error of decoded positions) could potentially be affected by motion artifact and excluded locations where no influence was possible because of a lack of motion artifact.

The funders had no role in study design, data collection, and interpretation, or the decision to submit the work for publication.

The authors thank Xilinx Corporation for donation of the Ultra96 board. We also thank Jeffrey Johnson for technical assistance in designing the custom interface board, and Shiyun Wang and Ryan Grgurich for helpful comments on the manuscript. This work was supported by NSF NeuroNex 1704708 (HTB, PG, DA, JC), 1UF1NS107668 (DA), and MH122800 (AI, HTB).

This study was performed in strict accordance with the recommendations in the Guide for the Care and Use of Laboratory Animals of the National Institutes of Health. All of the animals were handled according to approved Institutional Animal Care and Use Committee (IACUC) protocols (#2017-038) of the University of California Los Angeles. The protocol was approved by the Committee on the Ethics of Animal Experiments of UCLA. All surgery was performed under deep isoflurane anesthesia, and every effort was made to minimize suffering, including administration of pre- and post-surgical analgesia.

1. Preprint posted: February 2, 2022 (view preprint)
2. Received: March 3, 2022
3. Accepted: January 23, 2023
4. Accepted Manuscript published: January 24, 2023 (version 1)
5. Version of Record published: February 8, 2023 (version 2)

© 2023, Chen et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.