Intelligent systems for the medical domain often require processing data streams that evolve over time and are only partially labeled. At the same time, the need for explanations is of utmost importance not only due to various regulations, but also to increase trust among systems' users. In this work, an online data-driven learning method with focus on the explainability of evolving models equipped with incremental semi-supervised learning algorithms is considered. The proposed method combines: (i) the Dynamic Incremental Semi-Supervised Fuzzy C-Means (DISSFCM) algorithm to incrementally classify subsets of data; with (ii) Linguistic Summarization, which provides explanations of the classification results in terms of short sentences in a natural language. The approach has been illustrated for streaming data collected from voice calls of patients affected by Bipolar Disorder. The results show the effectiveness of the proposed method in classifying instances belonging to healthy and affective states, and explaining the approximate reasoning behind the classification of new acoustic data related to patients.