1. Introduction

2. Related Work

3. Trace Sampling Based on Type, Duration, and Errors

3.3. Hybrid Approach Based on Types and Durations

The hybrid approach takes into account both trace type and duration information. This algorithm performs accurate sampling if both durations and types are important. Assume that traces are already grouped into trace types. We generate the histogram of durations for each group and apply the procedure described in the previous subsection. We also consider the probability of a trace type. Common types will be sampled more aggressively.

Let

$$H^{\left(k\right)}=\{n_1^{\left(k\right)},\cdots ,n_m^{\left(k\right)}\},k=1,\cdots ,M,$$

be the histogram of durations of the k-th trace type and $n_s^{\left(k\right)}$ be the number of traces in the s-th bin of the k-th type. As above, M is the number of different types, and m is the number of bins in the histogram. Let $N_k$ be the number of traces in the k-th trace type:

$$N_k=\sum _{s=1}^m{n}_s^{\left(k\right)}.$$

Let N be the total number of traces:

$$N=\sum _{k=1}^M{N}_k.$$

Let

$$P^{\left(k\right)}=\{p_1^{\left(k\right)},\cdots ,p_m^{\left(k\right)}\},p_s^{\left(k\right)}=\frac{n_s^{\left(k\right)}}{N_k},k=1,\cdots ,M,$$

and

$$P=\{p_1,\cdots ,p_M\},p_k=\frac{N_k}{N}.$$

We denote the sampling rate of a trace from the s-th bin of the k-th trace type by $r_s^{\left(k\right)}$. It shows the fraction of traces that should be stored. We compute it by the following formula as inversely proportional to the corresponding probabilities:

$$r_s^{\left(k\right)}=1-{\left(p_k\right)}^{\alpha }{\left(p_s^{\left(k\right)}\right)}^{\beta },$$

where $\alpha ,\beta \ge 0$ are some unknown parameters to be tuned to meet the requirement for the final sampling rate r.

Let us show how it could be done. Let $N_k^{*}$ be the number of traces in the k-th trace type after the sampling. Let ${n_s^{\left(k\right)}}^{*}$ be the number of traces in the s-th bin and the k-th trace type after the sampling:

$$N_k^{*}=\sum _{s=1}^m{n_s^{\left(k\right)}}^{*}=\sum _{s=1}^m{n}_s^{\left(k\right)}{r}_s^{\left(k\right)}=N_k\sum _{s=1}^m{p}_s^{\left(k\right)}\left(1-{\left(p_k\right)}^{\alpha }{\left(p_s^{\left(k\right)}\right)}^{\beta }\right).$$

Let $N^{*}$ be the total number of traces after the sampling across all types and durations:

$$N^{*}=\sum _{k=1}^M{N}_k^{*}=\sum _{k=1}^M{N}_k\sum _{s=1}^m{p}_s^{\left(k\right)}\left(1-{\left(p_k\right)}^{\alpha }{\left(p_s^{\left(k\right)}\right)}^{\beta }\right)=NG(\alpha ,\beta ),$$

where

$$r=\frac{N^{*}}{N}=G(\alpha ,\beta )=\sum _{k=1}^M\sum _{s=1}^m{p}_k{p}_s^{\left(k\right)}\left(1-{\left(p_k\right)}^{\alpha }{\left(p_s^{\left(k\right)}\right)}^{\beta }\right)$$

is the required final sampling rate that can be accomplished by appropriately selecting parameters $\alpha$ and $\beta$.

First, we will illustrate this procedure when $\alpha =\beta$. Figure 9 reveals the procedure for a specific trace type containing 3996 traces. The left figure shows the scatter plot of durations. The right figure shows the histogram of durations with 7 bins before and after the samplings. The value $\beta =0.1$ corresponds to the sampling that stores 1104 traces. The next value $\beta =0.05$ stores 599 traces. The value $\beta =0.025$ corresponds to the sampling with 315 preserved traces. Finally, the value $\beta =0.01$ stores 131 traces. The last one corresponds to the $3.3\%$ sampling rate. By the way, the probability of this trace type is $0.2$, which also impacts the sampling rates with $\alpha =\beta$ setting.

Figure 9. The hybrid approach for a specific trace type (N2 in Figure 11). The left figure shows the plot of durations. The right figure shows the counts of traces in different bins before and after the samplings corresponding to different values of $\alpha =\beta$.

Figure 9. The hybrid approach for a specific trace type (N2 in Figure 11). The left figure shows the plot of durations. The right figure shows the counts of traces in different bins before and after the samplings corresponding to different values of $\alpha =\beta$.

Figure 10. The hybrid approach for a specific trace type (N17 in Figure 11). The left figure shows the plot of durations. The right figure shows the counts of traces for $\alpha =\beta$.

Figure 10. The hybrid approach for a specific trace type (N17 in Figure 11). The left figure shows the plot of durations. The right figure shows the counts of traces for $\alpha =\beta$.

Figure 11. The hybrid sampling approach for $\alpha =\beta$.

Figure 11. The hybrid sampling approach for $\alpha =\beta$.

Figure 10 shows the hybrid procedure for another trace type with the probability $0.0048$. It contains only 98 traces and is a rare type compared to the previous example. It also corresponds to the setting $\alpha =\beta$. The right figure shows the sampling results for different values of $\beta$. For example, the value $\beta =0.01$ corresponds to the sampling that stores only 7 traces of this specific type. It corresponds to the $7.1\%$ sampling rate. The value $\beta =0.025$ corresponds to the $18.4\%$ sampling rate. The value $\beta =0.05$ stores 31 traces and corresponds to the $31.6\%$ sampling rate. Finally, the value $\beta =0.1$ corresponds to the sampling rate $52\%$.

Figure 11 shows the result of the hybrid approach across different trace types when $\alpha =\beta$. The total sampling rate is $4\%$ for $\alpha =\beta =0.01$. The values $\alpha =\beta =0.1$ correspond to the sampling rate $31\%$. It means that the values of $\alpha$ and $\beta$ can control the sampling rates in a wide range of values. If we need to accomplish a strict requirement, we can turn to the $G(\alpha ,\beta )$ values corresponding to the sampling rates. Figure 12 shows the values of $G(\beta ,\beta )$, where the red cross corresponds to the sampling rate of $10\%$ (the exact value of G is $0.099$) with $\beta =0.029$. We can also tune the values of $\alpha$ and $\beta$ independently. Figure 13 shows the surface of sampling rates corresponding to different values. For example, the total sampling rate of $10\%$ can be accomplished by $\alpha =0.03,\beta =0.022$, or $\alpha =0.026,\beta =0.029$, or $\alpha =0.011,\beta =0.056$, or $\alpha =0.023,\beta =0.035$, or $\alpha =0.01,\beta =0.058$, etc.

Parameter optimization can be performed without the utilization of long historical data. We can take an initially random value for the parameters and, after each hour, verify the actual compression ratio. Then, by increasing or decreasing the values, we achieve the required sampling. This will work especially for dynamic applications when relying on available historical information is impossible.

How, in practice, can we sample a trace? When a specified trace (with known type and duration) is detected with a known sampling rate r, a random variable from $Bernoulli\left(r\right)$ distribution should be generated. This random variable has a boolean outcome with the value 1 that has probability r and value 0 with the probability $1-r$. If the outcome is 1, we store the specified trace; otherwise, we ignore it. This will allow us to store the traces with the required sampling rates in the long run.

Returning to the problem of histogram construction, we refer to a powerful approach known as the t-digest algorithm (see[71]), which addresses several problems in histogram construction. The first concern corresponds to storing time series data with trace durations. For each trace type, we need to store the corresponding time series of durations for further histogram construction, and we need those time series with sufficient statistics. For 28 trace types, as in our experiments, we need to store the 28 time series. The second concern corresponds to the process of histogram construction. Each time when we need those histograms, sorting should be applied to the corresponding data of durations, and the procedure must be repeated each time in case of some new arrivals. The concern is the impact of outliers. An efficient solution to the mentioned problems is the t-digest algorithm. Instead of storing the entire time series data, it stores only the result of data cluster centroids and data counts in each cluster. The efficient merging approach allows for combining different t-digest histograms, making the entire process streaming. The t-digest is very precise while estimating extreme data quantiles (close to 0 and 1), making the procedure robust to outliers.

4. Troubleshooting of Applications

5. Conclusions

6. Patents

Abbreviations

References

1. Parker, A.; Spoonhower, D.; Mace, J.; Sigelman, B.; Isaacs, R. Distributed Tracing in Practice: Instrumenting, Analyzing, and Debugging Microservices; O’Reilly Media, Incorporated, 2020.
2. Shkuro, Y. Mastering Distributed Tracing: Analyzing performance in microservices and complex systems; Packt Publishing, 2019.
3. Opentracing. What is distributed tracing? https://opentracing.io/docs/overview/what-is-tracing/, 2019. Accessed: 2021-01-26.
4. Cai, Z.; Li, W.; Zhu, W.; Liu, L.; Yang, B. A real-time trace-level toot-cause diagnosis system in Alibaba datacenters. IEEE Access 2019, 7, 142692–142702.
5. Liu, D.; He, C.; Peng, X.; Lin, F.; Zhang, C.; Gong, S.; Li, Z.; Ou, J.; Wu, Z. MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems. 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2021, pp. 338–347.
6. Poghosyan, A.V.; Harutyunyan, A.N.; Grigoryan, N.M.; Pang, C. Root Cause Analysis of Application Performance Degradations via Distributed Tracing. Third CODASSCA Workshop, Yerevan, Armenia: Collaborative Technologies and Data Science in Artificial Intelligence Applications; Hajian, A.; Baloian, N.; Inoue, T.; Luther, W., Eds.; Logos Verlag: Berlin, 2022; pp. 27–31.
7. Poghosyan, A.; Harutyunyan, A.; Grigoryan, N.; Pang, C. Distributed Tracing for Troubleshooting of Native Cloud Applications via Rule-Induction Systems. JUCS - Journal of Universal Computer Science 2023, 29, 1274–1297.
8. Distributed Tracing - Past, Present and Future.https://www.zerok.ai/post/distributed-tracing-past-present-future, 2023.
9. Young, T.; Parker, A. Learning OpenTelemetry; O’Reilly Media, 2024.
10. Cotroneo, D.; De Simone, L.; Liguori, P.; Natella, R. Run-time failure detection via non-intrusive event analysis in a large-scale cloud computing platform. Journal of Systems and Software 2023, 198, 111611.
11. Zhang, X.; Lin, Q.; Xu, Y.; Qin, S.; Zhang, H.; Qiao, B.; Dang, Y.; Yang, X.; Cheng, Q.; Chintalapati, M.; Wu, Y.; Hsieh, K.; Sui, K.; Meng, X.; Xu, Y.; Zhang, W.; Shen, F.; Zhang, D. Cross-dataset Time Series Anomaly Detection for Cloud Systems. 2019 USENIX Annual Technical Conference (USENIX ATC 19); USENIX Association: Renton, WA, 2019; pp. 1063–1076.
12. Poghosyan, A.; Harutyunyan, A.; Grigoryan, N.; Pang, C.; Oganesyan, G.; Ghazaryan, S.; Hovhannisyan, N. An Enterprise Time Series Forecasting System for Cloud Applications Using Transfer Learning. Sensors 2021, 21.
13. Abad, C.; Taylor, J.; Sengul, C.; Yurcik, W.; Zhou, Y.; Rowe, K. Log correlation for intrusion detection: A proof of concept. 19th Annual Computer Security Applications Conference, 2003. Proceedings., 2003, pp. 255–264.
14. Suriadi, S.; Ouyang, C.; van der Aalst, W.; ter Hofstede, A. Root cause analysis with enriched process logs. Business Process Management Workshops, International Workshop on Business Process Intelligence (BPI 2012). Volume 132 of Lecture Notes in Business Information Processing; Rosa, M.L.; Soffer, P., Eds.; Springer-Verlag: Berlin, 2013; pp. 174–186.
15. BigPanda. Incident Management. https://docs.bigpanda.io/docs/incident-management, 2020. Accessed: 2021-01-26.
16. Josefsson, T. Root-cause analysis through machine learning in the cloud; Uppsala Universitet, 2017.
17. Tak, B.; Tao, S.; Yang, L.; Zhu, C.; Ruan, Y. LOGAN: Problem diagnosis in the cloud using log-based reference models. 2016 IEEE International Conference on Cloud Engineering (IC2E) 2016, pp. 62–67.
18. Mi, H.; Wang, H.; Zhou, Y.; Lyu, M.R.; Cai, H. Localizing root causes of performance anomalies in cloud computing systems by analyzing request trace logs. SCIENCE CHINA Information Sciences 2012, 55, 2757–2773.
19. Barredo Arrieta, A.; Díaz-Rodríguez, N.; Del Ser, J.; Bennetot, A.; Tabik, S.; Barbado, A.; Garcia, S.; Gil-Lopez, S.; Molina, D.; Benjamins, R.; Chatila, R.; Herrera, F. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 2020, 58, 82 – 115.
20. Cohen, W.W. Fast Effective Rule Induction. Proceedings of the Twelfth International Conference on Machine Learning. Morgan Kaufmann, 1995, pp. 115–123.
21. Quinlan, J.R. C4.5: Programs for machine learning; Elsevier, 2014.
22. Las-Casas, P.; Papakerashvili, G.; Anand, V.; Mace, J. Sifter: Scalable Sampling for Distributed Traces, without Feature Engineering. Proceedings of the ACM Symposium on Cloud Computing; Association for Computing Machinery: New York, NY, USA, 2019; SoCC ’19, p. 312–324.
23. Thereska, E.; Salmon, B.; Strunk, J.; Wachs, M.; Abd-El-Malek, M.; Lopez, J.; Ganger, G.R. Stardust: Tracking activity in a distributed storage system. Proceedings of Joint International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS’06); ACM, , 2006.
24. Sambasivan, R.R.; Zheng, A.X.; Rosa, M.D.; Krevat, E.; Whitman, S.; Stroucken, M.; Wang, W.; Xu, L.; Ganger, G.R. Diagnosing performance changes by comparing request flows. 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2011.
25. Fonseca, R.; Porter, G.; Katz, R.H.; Shenker, S.; Stoica, I. X-trace: A pervasive network tracing framework. Proceedings of the 4th USENIX Conference on Networked Systems Design & Implementation; USENIX Association: USA, 2007; NSDI’07, p. 20.
26. Sigelman, B.H.; Barroso, L.A.; Burrows, M.; Stephenson, P.; Plakal, M.; Beaver, D.; Jaspan, S.; Shanbhag, C. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Technical report, Google, Inc., 2010.
27. Kaldor, J.; Mace, J.; Bejda, M.; Gao, E.; Kuropatwa, W.; O’Neill, J.; Ong, K.W.; Schaller, B.; Shan, P.; Viscomi, B.; Venkataraman, V.; Veeraraghavan, K.; Song, Y.J. Canopy: An End-to-End Performance Tracing And Analysis System. Proceedings of the 26th Symposium on Operating Systems Principles; Association for Computing Machinery: New York, NY, USA, 2017; SOSP ’17, p. 34–50.
28. OpenTelemetry. https://opentelemetry.io/, 2024.
29. Las-Casas, P.; Mace, J.; Guedes, D.; Fonseca, R. Weighted Sampling of Execution Traces: Capturing More Needles and Less Hay. Proceedings of the ACM Symposium on Cloud Computing; Association for Computing Machinery: New York, NY, USA, 2018; SoCC ’18, p. 326–332.
30. Google Cloud Observability: Trace Sampling. https://cloud.google.com/trace/docs/trace-sampling, 2024.
31. OpenCensus: Sampling. https://opencensus.io/tracing/sampling/ , 2024.
32. Azure Monitor: Sampling in Application Insights.https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling-classic-api, 2024.
33. He, S.; Feng, B.; Li, L.; Zhang, X.; Kang, Y.; Lin, Q.; Rajmohan, S.; Zhang, D. STEAM: Observability-Preserving Trace Sampling. FSE’23 Industry, 2023.
34. AWS: Advanced Sampling using ADOT.https://aws-otel.github.io/docs/getting-started/advanced-sampling#best-practices-for-advanced-sampling, 2024.
35. Jaeger: Sampling. https://www.jaegertracing.io/docs/1.55/sampling/, 2024.
36. Anomaly Detection in Zipkin Trace Data.https://engineering.salesforce.com/anomaly-detection-in-zipkin-trace-data-87c8a2ded8a1/, 2024.
37. LightStep: Sampling, verbosity, and the case for (much) broader applications of distributed tracing.https://medium.com/lightstephq/sampling-verbosity-and-the-case-for-much-broader-applications-of-distributed-tracing-f3500a174c17, 2024.
38. Datadog: Trace Sampling Use Cases.https://docs.datadoghq.com/tracing/guide/ingestion_sampling_use_cases/, 2024.
39. Partial trace sampling: A new approach to distributed trace sampling.https://engineering.dynatrace.com/blog/partial-trace-sampling-a-new-approach-to-distributed-trace-sampling/, 2024.
40. New Relic: Technical distributed tracing details.https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/#sampling, 2024.
41. OpenTelemetry Trace Sampling.https://docs.appdynamics.com/observability/cisco-cloud-observability/en/application-performance-monitoring/opentelemetry-trace-sampling, 2024.
42. When to Sample. https://docs.honeycomb.io/manage-data-volume/sample/guidelines/, 2024.
43. An introduction to trace sampling with Grafana Tempo and Grafana Agent.https://grafana.com/blog/2022/05/11/an-introduction-to-trace-sampling-with-grafana-tempo-and-grafana-agent/, 2024.
44. Application Performance Monitoring: Transaction sampling.https://www.elastic.co/guide/en/observability/current/apm-sampling.html, 2024.
45. Solé, M.; Muntés-Mulero, V.; Rana, A.I.; Estrada, G. Survey on models and techniques for root-cause analysis. ArXiv:1701.08546, 2017, [arXiv:cs.AI/1701.08546].
46. Harutyunyan, A.N.; Poghosyan, A.V.; Grigoryan, N.M.; Hovhannisyan, N.A.; Kushmerick, N. On machine learning approaches for automated log management. J. Univers. Comput. Sci. (JUCS) 2019, 25, 925–945.
47. Poghosyan, A.; Ashot, N.; G.M., N.; Kushmerick, N. Incident Management for Explainable and Automated Root Cause Analysis in Cloud Data Centers. JUCS - Journal of Universal Computer Science 2021, 27, 1152–1173.
48. Poghosyan, A.V.; Harutyunyan, A.N.; Grigoryan, N.M. Managing cloud infrastructures by a multi-layer data analytics. 2016 IEEE International Conference on Autonomic Computing, ICAC 2016, Wuerzburg, Germany, July 17-22, 2016; Kounev, S.; Giese, H.; Liu, J., Eds. IEEE Computer Society, 2016, pp. 351–356.
49. Poghosyan, A.; Harutyunyan, A.; Grigoryan, N.; Pang, C.; Oganesyan, G.; Ghazaryan, S.; Hovhannisyan, N. An Enterprise Time Series Forecasting System for Cloud Applications Using Transfer Learning. Sensors 2021, 21.
50. Marvasti, M.A.; Poghosyan, A.V.; Harutyunyan, A.N.; Grigoryan, N.M. Pattern detection in unstructured data: An experience for a virtualized IT infrastructure. 2013 IFIP/IEEE International Symposium on Integrated Network Management, IM 2013, Ghent, Belgium, May 27-31, 2013; Turck, F.D.; Diao, Y.; Hong, C.S.; Medhi, D.; Sadre, R., Eds. IEEE, 2013, pp. 1048–1053.
51. Reynolds, P.; Killian, C.E.; Wiener, J.L.; Mogul, J.C.; Shah, M.A.; Vahdat, A. Pip: Detecting the Unexpected in Distributed Systems. Symposium on Networked Systems Design and Implementation, 2006.
52. Harutyunyan, A.; Poghosyan, A.; Harutyunyan, L.; Aghajanyan, N.; Bunarjyan, T.; Vinck, A.H. Challenges and Experiences in Designing Interpretable KPI-diagnostics for Cloud Applications. JUCS - Journal of Universal Computer Science 2023, 29, 1298–1318. [CrossRef]
53. Harutyunyan, A.N.; Poghosyan, A.V.; Grigoryan, N.M.; Hovhannisyan, N.A.; Kushmerick, N. On Machine Learning Approaches for Automated Log Management. JUCS - Journal of Universal Computer Science 2019, 25, 925–945. [CrossRef]
54. Fürnkranz, J.; Gamberger, D.; Lavrač, N. Foundations of rule learning; Cognitive Technologies, Springer, Heidelberg, 2012; pp. xviii+334. With a foreword by Geoffrey I. Webb.
55. Fürnkranz, J.; Kliegr, T. A brief overview of rule learning. Rule technologies: Foundations, tools, and applications; Bassiliades, N.; Gottlob, G.; Sadri, F.; Paschke, A.; Roman, D., Eds.; Springer International Publishing: Cham, 2015; pp. 54–69.
56. Fürnkranz, J. Pruning Algorithms for Rule Learning. Mach. Learn. 1997, 27, 139–172.
57. Fürnkranz, J.; Widmer, G. Incremental reduced error pruning. Proc. 11th International Conference on Machine Learning. Morgan Kaufmann, 1994, pp. 70–77.
58. Hühn, J.; Hüllermeier, E. FURIA: An algorithm for unordered fuzzy rule induction. Data Min. Knowl. Discov. 2009, 19, 293–319.
59. Lin, F.; Muzumdar, K.; Laptev, N.P.; Curelea, M.V.; Lee, S.; Sankar, S. Fast Dimensional Analysis for Root Cause Investigation in a Large-Scale Service Environment. Proceedings of the ACM on Measurement and Analysis of Computing Systems 2020, 4, 1–23.
60. Lee, W.; Stolfo, S.J. Data Mining Approaches for Intrusion Detection. Proceedings of the 7th Conference on USENIX Security Symposium - Volume 7; USENIX Association: USA, 1998; SSYM’98, p. 6.
61. Helmer, G.; Wong, J.; Honavar, V.; Miller, L. Intelligent agents for intrusion detection. Proceedings IEEE Information Technology Conference, Syracuse, NY. Springer, 1998, pp. 121–124.
62. Helmer, G.; Wong, J.S.; Honavar, V.; Miller, L. Automated discovery of concise predictive rules for intrusion detection. Journal of Systems and Software 2002, 60, 165–175.
63. Mannila, H.; Toivonen, H.; Verkamo, A.I. Discovering Frequent Episodes in Sequences Extended Abstract. Proceedings of the First International Conference on Knowledge Discovery and Data Mining. AAAI Press, 1995, KDD’95, p. 210–215.
64. Liu, H.; Motoda, H., Perspectives of Feature Selection. In Feature Selection for Knowledge Discovery and Data Mining; Springer US: Boston, MA, 1998; pp. 17–41.
65. John, G.H.; Kohavi, R.; Pfleger, K. Irrelevant features and the subset selection problem. Machine Learning: Proceedings of the 11th International Conference. Morgan Kaufmann, 1994, pp. 121–129.
66. Agrawal, R.; Imieliński, T.; Swami, A. Mining association rules between sets of items in large databases. Proceedings of the 1993 ACM SIGMOD international conference on Management of data, 1993, pp. 207–216.
67. Shafer, G. A Mathematical Theory of Evidence; Princeton University Press: Princeton, 1976.
68. Peñafiel, S.; Baloian, N.; Sanson, H.; Pino, J.A. Applying Dempster–Shafer theory for developing a flexible, accurate and interpretable classifier. Expert Syst. Appl. 2020, 148.
69. Poghosyan, A.; Harutyunyan, A.; Davtyan, E.; Petrosyan, K.; Baloian, N. A Study on Automated Problem Troubleshooting in Cloud Environments with Rule Induction and Verification. Applied Sciences 2024, 14.
70. Leys, C.; Ley, C.; Klein, O.; Bernard, P.; Licata, L. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology 2013, 49, 764–766.
71. Dunning, T. The t-digest: Efficient estimates of distributions. Software Impacts 2021, 7, 100049.
72. Witten, I.H.; Frank, E.; Hall, M.A.; Pal, C.J. Practical machine learning tools and techniques. Morgan Kaufmann 2005.