Predicting user satisfaction from customer service chats

Abstract

Customer service is a determining factor in the user experience of Fintech companies. This work seeks to  understand, using machine learning techniques, what factors lead the clients of a specific Fintech company to positively evaluate their experience. Two data sources were used to achieve this: user records from their sign up and the log of conversations with customer service via WhatsApp. We experimented with predictive models based on XGBoost, trained with features of the user context, the characteristics of the conversations and the semantics of the words used in the conversations. The results were lower than expected (AUC = 0.5152), but they leave valuable lessons for those who face similar problems in the future, related to the challenges of the following critical aspects: i. avoid data leakage, ii. evaluate models and scoring metrics thoroughly, iii. carry out intermediate checkpoints, iv. do not underestimate the time required for data transformation, v. perform a unit testing process and vi. know the domain. This paper describes the different stages of the methodology: data extraction and transformation, feature generation, predictive model training, optimal model selection and test data evaluation