In order to stabilize the orientation, a smoothing algorithm is used. This is in essence a low pass filter applied to the 3D orientation. To illustrate how this works, let's take one of the simplest low pass filters in 1D: the exponential filter, also known as exponential moving average or exponentially weighted moving average depending on the field of study (see the wikipedia article for more info). If \(\alpha\) be a number between 0 and 1 and the \(Y_t\) being the t-th input sample. The output, \(S_t\) of this low pass filter is then:
This can be seen as a weighted average between the previous output and the newest input sample. Alternatively this can be seen as the output moving a fractional amount towards to input for each sample. The result is a smoothed 1D signal, but introduces a delay in the signal. By applying the same filter twice in opposite directions, this delay is eliminated.
Now, how can this be expanded to 3D? A three-dimensional orientation needs to be represented by at least three numbers, so one obvious way of filtering the orientation is to apply the low pass filter to each axis. Although this works, the result might not be the smoothing you expect. For instance, if the orientation is represented by an Euler angle, applying a low pass filter to each axis leads to non-linear behavior in the resulting orientation. If instead the orientations are represented by quaternions, the Slerp (Spherical Linear Interpolation, wiki article) method invented by Ken Shoemake can be used. Slerp can compute the intermediate orientation (quaternion) between two original quaternions corresponding to a smooth trajectory between the two. This is exactly what's needed for the simple exponential smoothing, since the weighted sum is equivalent to starting at \(Y_{t}\) and moving a fractional distance of \(\alpha\) towards \(S_{t-1}\). This filter can thus be translated to an orientation filter as follows:
Once again, this is applied in both directions to counteract the introduced delay. The difference between the input quaternions (estimated camera motion) and output quaternions (smoothed virtual camera) is then used for the image stabilization process. Due to the simplicity, this orientation filter was the first one implemented in the Gyroflow project, but has a number of limitations, mainly during large camera shakes and the lack of smoothness control for each rotational axis. Nevertheless, you should now have an overview of how the general smoothing algorithm works (Input orientation quaternions -> Smoothing algo -> Smoothed orientation quaternions) which might give you ideas for your own smoothing algorithm.