How To Increase Data Card Speed?

Can the application of a  data compression algorithm to analytical data increase the speed in which it is processed?

• Data compression is essentially a form of feature detection. Can compressing the data beforehand (as it is incorporated) and decompressing it after offer improved speed and insight?

• Answer:

  It depends on how loose your definition of a data compression algorithm is, but there are a variety of algorithms that can take a noisy time series and chunk the series into groups. Roughly speaking, what one does is find a finite number of states that a noisy time series (e.g. financial data, MD trajectories) is sampling and then clusters the time-series into these states. One usually tries to fit a Markov process that can generate a compressed version of the time-series. More precisely, if my set of states is {S1,…,Sn}{S1,…,Sn} \{ S_1, \ldots, S_n\}, then each time series element xtxtx_t is associated to some SiSi S_i . Now we compress our data by simply saying that the value associated to SiSiS_i is the mean of all the time points that correspond to that state, E[Si]:=E[{xt:xt∈Si}]E[Si]:=E[{xt:xt∈Si}]\mathsf{E}[S_i] := \mathsf{E} [ \{x_t : x_t \in S_i \} ] . This compression gives us a bunch of step functions for our time series, as opposed to a bunch of noisy data. The whole goal is to see if the system is really generated by a simple process (e.g. the Markov process that generates to the mean of each state and the transitions) and some noise.  After one trains such a compressed model, he/she can take in raw data streams and then classify them into the states of the aforementioned Markov model in order to get things like (smoothed out) autocovariance functions. Moreover, one can attempt to predict transitions between states (which is of crucial importance in finance and Statistical Physics).