In this video, Rasa Senior Developer Advocate Rachael will talk about what tokenization is and introduce you to some common methods along with their benefits and drawbacks.
– Tokenization As The Initial Phase In NLP, Jonathan J. Webster, Chunyu Kit (COLING 1992). Link: https://www.aclweb.org/anthology/C92-4173/
– Components section of Rasa Open Source documentation (various authors, continuously updated). Link: https://rasa.com/docs/rasa/nlu/components/
If you want to reach out to us, feel free to leave a message at our community forum; https://forum.rasa.com/.