Joint Selective State Space Model and Detrending for Robust Time Series Anomaly Detection
Abstract
Deep learning-based sequence models are extensively employed in Time Series Anomaly Detection (TSAD) tasks due to their effective sequential modeling capabilities. However, the ability of TSAD is limited by two key challenges: (i) the ability to model long-range dependency and (ii) the generalization issue in the presence of non-stationary data. To tackle these challenges, an anomaly detector that leverages the selective state space model known for its proficiency in capturing long-term dependencies across various domains is proposed. Additionally, a multi-stage detrending mechanism is introduced to mitigate the prominent trend component in non-stationary data to address the generalization issue. Extensive experiments conducted on real-world public datasets demonstrate that the proposed methods surpass all compared baseline methods.
Index Terms:
Time series anomaly detection, Selective state space model, Time series detrendingI Introduction
As the volume of data generated continues to grow exponentially, Time Series Anomaly Detection (TSAD) has garnered significant attention due to its increasing demand in various real-world applications such as intrusion detection, disaster warning, and medical diagnosis [1, 2, 3, 4, 5]. TSAD aims to identify irregular points or subsequences collectively referred to as anomalies. Typically, samples with high reconstruction errors of deep neural networks (DNNs) trained exclusively on normal data are detected as anomalies. Given the temporal correlations inherent in time series [6], DNN-based sequence models are considered the most suitable approach for TSAD tasks [7, 8, 9, 10, 11, 12, 13, 14]. However, two key challenges remain in these sequence model-based TSAD: (i) the ability to model long-range dependencies and (ii) the generalization issue for non-stationary data.
Normal behavior in time-series data typically involves long-term dependencies[1]. To capture these dependencies and enhance the modeling of normal behavior, various sequence models including Temporal Convolutional Networks (TCNs) [15], Recurrent Neural Networks (RNNs) [8, 9] and Transformers [11, 12] have been explored for TSAD. However, existing methods still encounter difficulties due to their intrinsic characteristics, such as limited context window, high memory costs, or unstable gradient flow. Recent research on the selective state space model (S6) [16] has addressed the shortcomings of the aforementioned sequence models and demonstrated its excellent long-term modeling capabilities in other domains. Moreover, the selective nature of the S6 model allows it to discard abnormal information and generate reliable reconstructed output for TSAD. Despite offering a potential solution to model long-range dependencies, there has been no research exploring its application in TSAD.
The generalization issue for non-stationary data stems from trends in non-stationary data. The erratic distribution of trends can result in significant fluctuations in data magnitude, leading to erroneously high reconstruction errors in regions with previously unseen trend patterns in the training set, ultimately causing false alarms [14]. Traditional time series decomposition methods such as Seasonal-Trend decomposition using LOESS (STL) [17] and Hodrick–Prescott (HP) trend filter [18] are frequently used to mitigate the impact of these trends. However, achieving optimal decomposed results may not yield the best detection performance. To address this, several approaches have tried to integrate decomposition methods into detectors for joint optimization. Nevertheless, most of them rely on Moving Average (MA) with a fixed kernel size [19, 20, 14], lacking the flexibility for broader scenarios. Other methods rely on DNN models for detrending [21, 22]. Though they allow end-to-end optimization, their dependence on training data reintroduces generalization issues.
To tackle the challenges outlined above, an innovative detector constructed using the S6 model and integrated with a multi-stage detrending mechanism is proposed. The contributions of this paper is as follows: (i) Introduce a novel detector that utilizes the S6 model to effectively model long-range dependencies in TSAD. (ii) Propose a multi-stage detrending mechanism that can generate reliable decomposed results. (iii) Evaluate the proposed method on three benchmark datasets and demonstrate the superiority of the proposed method over recent State-of-the-art (SOTA) methods.
II Related Work
Sequence models such as TCN, RNN, and Transformer [23, 24, 25] had greatly improved TSAD tasks. However, these methods struggled with long-range dependency modeling, limiting detection performance. Fig. 1 illustrates how different sequence models gather context information. As shown in Fig. 1(a), TCN’s context was constrained by the kernel size , which made it unsuitable for capturing long-term dependencies. Conversely, the Transformer could capture long context using the Self-Attention (SA) mechanism as shown in Fig. 1(b). However, SA was memory and time-intensive during training and inference because it did not compress context information, which made it impractical for modeling long dependencies. Although RNN could efficiently compress context information using a finite state shown in Fig. 1(c), it suffered from unstable gradient issues with long sequences [26]. The State Space Model (SSM), similar to RNN, resolves gradient issues by employing solely linear transformations, as shown in Fig. 1(d).
The Structured State Space Sequence model (S4) [27] was a well-known SSM that combines the benefits of RNN with the stability of linear transformations. This allows S4 to compress context states into a finite size while ensuring stable gradients, making it suitable for modeling long-range dependencies. However, its parameters remain constant through time, forming a Linear Time Invariance (LTI) model, limiting its effectiveness in TSAD tasks. To remove the LTI constraint, Gu et al. [16] proposed a selective SSM called S6 by introducing input-dependent parameters. S6 exhibited superior performance in modeling long-range dependencies in other domains, including natural language processing [16], computer vision [28], and speech [29]. Additionally, its selection mechanism has the potential to enhance TSAD performance by adaptively selecting the appropriate context information for producing reliable anomaly scores. Despite its effectiveness across various domains, its application in TSAD remains to be unexplored.
![Refer to caption](x1.png)
III Preliminaries
III-A Problem Statement
Given a multivariate time series with length for training, where each observation contains features. TSAD aims to predict anomalous labels for unseen test time series of length , where . This prediction is based on anomaly scores generated by the detector and a predefined threshold , where if , otherwise .
III-B S6 Model
A typical SSM model maps a univariate sequence to the output through a hidden state with four parameters , , where denotes the -th time-step and is the number of states.
To remove the LTI constraint, three of the above parameters in S6 model were adjusted to vary with time, namely , , where denotes the number of time-steps. The operation of the S6 model could be represented by a general formula applicable to multivariate time series data :
(1) | ||||
where denotes the -th feature, represents the Hadamard product, and . Here, and represent discretization rules for and . It is suggested that followed the zero-order hold rule, while follows the Euler rule [16], expressed as:
(2) | ||||
III-C Time Series Detrending
A common time series model with trend and seasonality could be formulated as where denotes the observation at the -th time-step, is the trend component, is the seasonal component with a period of , and represents the residual component. In this paper, emphasis was placed on trend removal and HP trend filter was adopted as the detrending method, expressed as:
(3) |
where is a hyper-parameter controlling the smoothness.
IV Method
Fig. 2 illustrates the overall framework of the proposed method, comprising a HP trend filter, multiple Decomposition-based Mamba (DMamba) blocks, and a final output module. Initially, input data was processed by the trend filter to extract initial trends and seasonality components. The initial seasonality was refined through DMamba blocks, while the initial trend was merged with trends from DMamba blocks. Finally, the fused trend and refined seasonality were summed to form the reconstructed output.
![Refer to caption](x2.png)
IV-A HP Trend Filter
The input data was initially segmented into subsequences using sliding windows of length . Each input window underwent detrending using HP trend filter. To ensure the extraction of global trends, the current input window was first combined with historical windows, and then the HP-based trend component was computed via Eq. (3). The HP-based seasonality was further derived as .
IV-B DMamba Blocks
![Refer to caption](x3.png)
The HP-based seasonality was first projected to a -dimensional space by an embedding network , and then it underwent DMamba blocks. Each block comprised a Mamba block [16] and an Adaptive MA (AMA) module. The input of each block was the output of the previous block, denoted as with . For clarity, the superscript was omitted in the following content. The input first entered the Mamba block shown in Fig. 3 and could be formulated by Eq. (4):
(4) | ||||
where are trainable projection matrices, represents a convolution layer with a kernel size of , denotes the SiLU[30] activation function, and denotes the S6 operation detailed in algorithm 1. The three parameters in Algorithm 1 could be computed as:
(5) | ||||
(6) | ||||
(7) |
where denotes the Softplus activation function, and are trainable projection matrices.
The AMA module then took as input and performed trend removal through average pooling:
(8) |
with a kernel size of . The kernel size is adaptively estimated by , where denotes the floor function, indicates the peak position of the power density function and is the Fourier transformation of . The output of the current block is then obtained by .
IV-C Final Reconstructed Output
The output of the final DMamba block served as the reconstructed seasonal component i.e., , where is a trainable projection matrix. The final trend is a combination of all extracted trends:
(9) |
where is a CNN layer with an input channel of , an output channel of , and a kernel size of . In the training phase, the objective was to minimize the Mean Square Error (MSE) between the original input and the reconstructed output:
(10) |
During inference, the MSE serves as the anomaly score.
V Experiments
V-A Setup
V-A1 Datasets
Dataset | Train | Test | Dimensions | Contamination (%) |
---|---|---|---|---|
NASA | ||||
SMD | ||||
SWaT |
Experiments were conducted on three publicly available datasets. The NASA dataset [8] contained sensor metrics recorded by Soil Moisture Active Passive satellite (SMAP) and Mars Science Laboratory rover (MSL). Three non-trivial subsets were utilized, namely A-4, T-1, and C-2. as suggested in [12]. The Server Machine Dataset (SMD) [9] includes multiple server machine metrics from a large Internet company. Emphasis was placed on non-trivial subsets 1-1, 1-6, 2-1, 3-2, and 3-7, as suggested [12]. Lastly, the Secure Water Treatment (SWaT) [38] was a dataset that spans 11 days of continuous operation. In the experiments, SWaT was downsampled to the minute level and binary features were discarded. Detailed information is listed in Tab. I.
V-A2 Baselines
The proposed method was compared to traditional and DNN-based methods. Traditional methods include LOF [31], CBLOF [32], OCSVM [33], IForest [34], HBOS [35], LODA [36] and ECOD [37]. DNN-based methods comprised two RNN-based methods: Omnianomaly [9] and FGANomaly [10], two Transformer-based methods: AnomalyTrans [11] and DCdetector [13], and a time series decomposition-based method: D3R [14].
V-A3 Implementation details
All baselines were implemented based on PyOD [39] or their publicly available codes. We utilized Peak-Over-Threshold (POT) [40] for threshold selection. The smoothness parameter was set to . The block size was , the model size was , the state size was and the window size was . The training was conducted for epochs with a batch size of , using the AdamW [41] optimizer with a learning rate of . Given that point-wise metrics can be influenced by lengthy anomalies and point-adjustment strategy [9] could lead to misguided rankings [42], the recently proposed affiliation-based precision (P-AF), recall (R-AF), and F1-score (F1-AF) [43] were adopted as evaluation metrics. These metrics offer evaluations at the level of abnormal events rather than individual points, making them suitable for TSAD. Since F1-AF represents the harmonic mean of P-AF and R-AF, it was employed to evaluate the overall performance, a common practice in the TSAD community.
V-B Results
The proposed method was compared to baselines, as listed in Tab. II. Generally, sequence model-based methods outperformed traditional methods due to their ability to model temporal dependencies. The proposed method surpassed all compared methods on the three datasets, achieving a relative improvement in average F1-AF compared to the best baseline method. This demonstrated that the proposed S6 model-based detector was suitable for TSAD tasks and superior to previous SOTA detectors with RNN or Transformer backbones. Meanwhile, for the SWaT and SMD datasets containing significant trend variation, D3R and the proposed method achieved notable performance improvement by incorporating time series decomposition approaches compared to other methods. Overall, the experimental results showcased the robustness of the proposed method in non-stationary scenarios and its ability to detect complex time series anomalies.
Modules | F1-AF | ||||
---|---|---|---|---|---|
HP Trend Filter | AMA | NASA | SWaT | SMD | Avg. |
✗ | ✗ | ||||
✗ | ✓ | ||||
✓ | ✗ | ||||
✓ | ✓ |
To further verify the necessity of the HP trend filter and the AMA mechanism, an ablation analysis was conducted and the results were shown in Tab. III. The results demonstrated that applying only one might not bring improvement. For instance, solely applying the HP trend filter in the NASA dataset or only utilizing AMA in the SWaT dataset led to a performance drop. However, when both were applied, the HP trend filter provided stable input for AMA’s period estimation, where AMA and the detector could symbiotically optimize performance and capture more complex patterns, resulting in the best performance over all datasets. Additionally, a visualization in Fig. 4 clearly showcased the effectiveness of the proposed detrending mechanisms.
![Refer to caption](x4.png)
VI Conclusion
This paper proposed solutions to effectively tackle two critical challenges in TSAD: modeling long-range dependency and generalizing to non-stationary data, which are bottlenecks for sequence models in TSAD. Our solutions involve leveraging a recently published S6 model to capture long-term context and introducing a novel multi-stage detrending mechanism to provide stable input for the sequence model. Experimental results underscored the suitability of the S6 model for TSAD and highlighted the importance of the proposed detrending mechanism. Furthermore, this study laid the foundation for developing more advanced S6 model-based detectors for TSAD.
References
- [1] Z. Z. Darban, G. I. Webb, S. Pan, C. C. Aggarwal, and M. Salehi, “Deep learning for time series anomaly detection: A survey,” arXiv preprint arXiv:2211.05244, 2022.
- [2] J. Yang, G. I. Choudhary, S. Rahardja, and P. Fränti, “Classification of interbeat interval time-series using attention entropy,” IEEE Transactions on Affective Computing, vol. 14, no. 1, pp. 321–330, 2020.
- [3] J. Yang, X. Tan, and S. Rahardja, “Mipo: How to detect trajectory outliers with tabular outlier detectors,” Remote sensing, vol. 14, no. 21, p. 5394, 2022.
- [4] N. LaRosa, J. Farber, P. Venkitasubramaniam, R. Blum, and A. Al Rashdan, “Separating sensor anomalies from process anomalies in data-driven anomaly detection,” IEEE Signal Processing Letters, vol. 29, pp. 1704–1708, 2022.
- [5] Z. Wang, Y. Zhang, G. Wang, and P. Xie, “Main-auxiliary aggregation strategy for video anomaly detection,” IEEE Signal Processing Letters, vol. 28, pp. 1794–1798, 2021.
- [6] K.-H. Lai, D. Zha, J. Xu, Y. Zhao, G. Wang, and X. Hu, “Revisiting time series outlier detection: Definitions and benchmarks,” in Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 1), 2021.
- [7] K. Cheng, Y. Liu, and X. Zeng, “Learning graph enhanced spatial-temporal coherence for video anomaly detection,” IEEE Signal Processing Letters, vol. 30, pp. 314–318, 2023.
- [8] K. Hundman, V. Constantinou, C. Laporte, I. Colwell, and T. Soderstrom, “Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding,” in Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp. 387–395.
- [9] Y. Su, Y. Zhao, C. Niu, R. Liu, W. Sun, and D. Pei, “Robust anomaly detection for multivariate time series through stochastic recurrent neural network,” in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 2828–2837.
- [10] B. Du, X. Sun, J. Ye, K. Cheng, J. Wang, and L. Sun, “Gan-based anomaly detection for multivariate time series using polluted training set,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 12, pp. 12 208–12 219, 2021.
- [11] J. Xu, H. Wu, J. Wang, and M. Long, “Anomaly transformer: Time series anomaly detection with association discrepancy,” in International Conference on Learning Representations, 2021.
- [12] S. Tuli, G. Casale, and N. R. Jennings, “Tranad: deep transformer networks for anomaly detection in multivariate time series data,” Proceedings of the VLDB Endowment, vol. 15, no. 6, pp. 1201–1214, 2022.
- [13] Y. Yang, C. Zhang, T. Zhou, Q. Wen, and L. Sun, “Dcdetector: Dual attention contrastive representation learning for time series anomaly detection,” in Proceedings of the 29th ACM SIGKDD international conference on knowledge discovery & data mining, 2023, pp. 3033–3045.
- [14] C. Wang, Z. Zhuang, Q. Qi, J. Wang, X. Wang, H. Sun, and J. Liao, “Drift doesn’t matter: Dynamic decomposition with diffusion reconstruction for unstable multivariate time series anomaly detection,” Advances in Neural Information Processing Systems, vol. 36, 2024.
- [15] Y. He and J. Zhao, “Temporal convolutional networks for anomaly detection in time series,” in Journal of Physics: Conference Series, vol. 1213, no. 4. IOP Publishing, 2019, p. 042050.
- [16] A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023.
- [17] R. B. Cleveland, W. S. Cleveland, J. E. McRae, I. Terpenning et al., “Stl: A seasonal-trend decomposition,” J. Off. Stat, vol. 6, no. 1, pp. 3–73, 1990.
- [18] R. J. Hodrick and E. C. Prescott, “Postwar us business cycles: an empirical investigation,” Journal of Money, credit, and Banking, pp. 1–16, 1997.
- [19] H. Wu, J. Xu, J. Wang, and M. Long, “Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting,” Advances in neural information processing systems, vol. 34, pp. 22 419–22 430, 2021.
- [20] T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, “Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting,” in International conference on machine learning. PMLR, 2022, pp. 27 268–27 286.
- [21] Z. Zhang, R. Wang, R. Ding, and Y. Gu, “Unravel anomalies: an end-to-end seasonal-trend decomposition approach for time series anomaly detection,” in 2024 International Conference on Acoustics, Speech and Signal Processing. IEEE, 2024, pp. 5415–5419.
- [22] S. Qin, J. Zhu, D. Wang, L. Ou, H. Gui, and G. Tao, “Decomposed transformer with frequency attention for multivariate time series anomaly detection,” in 2022 International Conference on Big Data. IEEE, 2022, pp. 1090–1098.
- [23] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” arXiv preprint arXiv:1803.01271, 2018.
- [24] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
- [25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
- [26] R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks,” in International conference on machine learning. PMLR, 2013, pp. 1310–1318.
- [27] A. Gu, K. Goel, and C. Re, “Efficiently modeling long sequences with structured state spaces,” in International Conference on Learning Representations, 2021.
- [28] Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, and Y. Liu, “Vmamba: Visual state space model,” arXiv preprint arXiv:2401.10166, 2024.
- [29] X. Jiang, C. Han, and N. Mesgarani, “Dual-path mamba: Short and long-term bidirectional selective structured state space models for speech separation,” arXiv preprint arXiv:2403.18257, 2024.
- [30] S. Elfwing, E. Uchibe, and K. Doya, “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning,” Neural networks, vol. 107, pp. 3–11, 2018.
- [31] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “Lof: identifying density-based local outliers,” in Proceedings of the 2000 ACM SIGMOD international conference on Management of data, 2000, pp. 93–104.
- [32] Z. He, X. Xu, and S. Deng, “Discovering cluster-based local outliers,” Pattern recognition letters, vol. 24, no. 9-10, pp. 1641–1650, 2003.
- [33] B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, “Estimating the support of a high-dimensional distribution,” Neural computation, vol. 13, no. 7, pp. 1443–1471, 2001.
- [34] F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation forest,” in 2008 eighth ieee international conference on data mining. IEEE, 2008, pp. 413–422.
- [35] M. Goldstein and A. Dengel, “Histogram-based outlier score (hbos): A fast unsupervised anomaly detection algorithm,” KI-2012: poster and demo track, vol. 1, pp. 59–63, 2012.
- [36] T. Pevnỳ, “Loda: Lightweight on-line detector of anomalies,” Machine Learning, vol. 102, pp. 275–304, 2016.
- [37] Z. Li, Y. Zhao, X. Hu, N. Botta, C. Ionescu, and G. H. Chen, “Ecod: Unsupervised outlier detection using empirical cumulative distribution functions,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 12, pp. 12 181–12 193, 2022.
- [38] A. P. Mathur and N. O. Tippenhauer, “Swat: A water treatment testbed for research and training on ics security,” in 2016 international workshop on cyber-physical systems for smart water networks (CySWater). IEEE, 2016, pp. 31–36.
- [39] Y. Zhao, Z. Nasrullah, and Z. Li, “Pyod: A python toolbox for scalable outlier detection,” Journal of machine learning research, vol. 20, no. 96, pp. 1–7, 2019.
- [40] A. Siffer, P.-A. Fouque, A. Termier, and C. Largouet, “Anomaly detection in streams with extreme value theory,” in Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery & data mining, 2017, pp. 1067–1075.
- [41] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.
- [42] S. Kim, K. Choi, H.-S. Choi, B. Lee, and S. Yoon, “Towards a rigorous evaluation of time-series anomaly detection,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 7, 2022, pp. 7194–7201.
- [43] A. Huet, J. M. Navarro, and D. Rossi, “Local evaluation of time series anomaly detection algorithms,” in Proceedings of the 28th ACM SIGKDD international conference on knowledge discovery & data mining, 2022, pp. 635–645.