Laura's Wiki
  • README
  • C++
    • CS 246
      • Lecture 1
      • Lecture 10
      • Lecture 11
      • Lecture 12
      • Lecture 13
      • Lecture 14
      • Lecture 15
      • Lecture 16
      • Lecture 17
      • Lecture 18
      • Lecture 19
      • Lecture 2
      • Lecture 20
      • Lecture 21
      • Lecture 22
      • Lecture 23
      • Lecture 3
      • Lecture 4
      • Lecture 5
      • Lecture 6
      • Lecture 7
      • Lecture 8
      • Lecture 9
      • Notes To Rememeber
  • Data Science
    • Jupyter Notebooks
      • Jupyter Notebook With Vs Code
      • Jupyter Notebooks Virtual Env
      • Jupyter Notify
      • Troubleshooting
    • Pandas
      • Box And Whisker Plots
      • Lists And Dictionaries
      • Manipulating Built Dataframes
      • Troubleshooting
    • Time Series
      • Autocorrelation Serial Correlation
      • Correlation Pct Change
      • Linear Regression
      • Time Series
  • Databases
    • SQL Commands
      • Basic Commands
      • Foreign Key Commands
      • Foreign Key
    • MySQL
    • Postgres
    • SQLAlchemy
  • DevOps
    • AWS
      • AWS EKS
      • Tmux
    • Docker
      • Docker Commands
      • docker-compose.yml
      • Docker Images
      • Docker Mapping
      • Docker Networks
      • Docker Storage
      • Dockerfile
      • Overview
    • Jenkins
      • Jenkins Configurations
      • Jenkins Master Slave
      • Jenkinsfile
      • Troubleshooting Pipelines
    • Kubernetes
      • Certificates
      • Example
      • Helm Commands
      • Introduction
      • Kubernetes Architecture
      • Kubernetes Commands
      • Kubernetes External Access
      • Kubernetes Networking
      • Kubernetes Troubleshoot
      • Kubernetes Volumes
      • Routing
      • Volumes
  • GUI
    • PyQT
      • Overview
  • Interview
    • Data Science Interview
      • Knowledge Interview Questions
    • SE Interview
      • Linked Lists
      • Trees
    • Interview Questions
  • Machine Learning
    • Algorithms
      • Algorithms Pros And Cons
      • Machine Learning Models Algorithms
    • Convolutional Neural Networks
      • Capsule Networks
      • CNN Input Output Shape
      • CNN Layers
      • Convolutional Hypercolumns
      • Groundbreaking Architectures
      • Image Augmentation
      • Localization Object Detection
      • Prepare Images For CNN
    • Evaluation
      • Kullback Leibler Divergence
      • Loss Functions
      • Metrics To Evaluate Ml Algorithms
    • fast.ai
      • Class Learner
      • Nlp And Databunch
      • Training
    • GAN
      • Introduction To Gan
    • LLMs
      • Guardrails
    • Machine Learning Audio Classification
      • Background
      • Building CNN
      • Deepplaylist RNNs To Predict Song Similarity
      • Loading Data
      • Model Preparation
      • Plotting Calculating And Cleaning
    • Natural Language Processing
      • Filtering Stop Words
      • Gibberish N Gram Language Model
      • Key Word Finder
      • N Gram Language Models
    • Neural Networks
      • Activation Functions
      • Backpropagation Reverse Mode Differentiation
      • Introduction To Neural Networks
      • Neural Networks
      • Training A Neural Network
      • Training Own Datasets
    • Object Tracking
      • Deepsort Overview
      • Detection Vs Tracking
    • PyTorch
      • Audio Classifier Tutorial
      • Convolutional Neural Networks
      • Tensors
    • Rasa
      • Bot Get Info Pipelines
      • Building Bot
      • Evaluating Entities
      • Improving Nlu Model
      • Optimizing Entitiy Recognition
      • Optimizing Entity Recognition
      • Optimizing Hyperparameter
      • Optimizing Intent
      • Text Featurizers
      • Tokenizers
    • Recurrent Neural Networks
      • Lstm Networks
      • Recurrent Neural Networks
      • Vanishing Gradients In RNNs
    • Research Papers
      • Convolutional Neural Networks For Automatic Image Colorization
      • Learning Representation For Automatic Colorization
      • Stack GAN
      • Style Transfer
    • Scikit Learn
      • Example Of Training And Predicting Model
      • Measuring Model Accuracy
      • Model Validation
      • Random Forests
      • Selecting Data For Modeling
    • Spacy
      • Language Processing Pipelines
    • Stanford ML Course
      • Classification And Representation
      • Computing Parameters Analytically
      • Linear Algebra Review
      • Model Cost Function
      • Multivariate Linear Regression
      • Parameter Learning
      • Supervised Unsupervised Learning
    • Tensorflow
      • Multiinput Keras
      • Tensorflow Pretrained
      • Tensorflow Tpu
  • Math
    • Math 235
    • Math 239 Formula Theorems
    • Math 239
  • Miscellaneous
    • Bazel
      • Commands
    • Kafka
      • Introduction
    • Regex
      • Useful Regex Patterns
    • API
    • Callbacks
    • Cron Job
    • Gitbook
    • Google Colab
    • Processors
  • Networking
    • Https
    • Ngrok
    • Port Forwarding
    • Tcp
  • Python
    • Object Oriented Programming
      • Args And Kwargs
      • Dataclasses
      • Built-in Methods
    • Argparse
    • Configparser
    • Contextmanager
    • Decorators
    • Functools
    • Imports
    • Linked Lists
    • Logging
    • Multiprocessing
    • Multithreading
    • Nested Dictionaries
    • Operators
    • Pickle
    • Profiling
    • Regex
    • Singletons
    • String Tricks
    • Troubleshooting
    • Unit Testing
  • R
    • Useful Commands
  • Shell
    • Linux
      • Environment Variables
      • File Management
      • IO
      • Networking
      • Permissions
      • Processes
      • Profiling
      • Storage Space Commands
      • Virtual Environment
    • Mac OS X
      • Python
    • Windows
      • Environment
  • Version Control
    • Git
      • Committing Pushing
      • Git Branches
      • Git Command Options
      • Git Merge Conflicts
      • Pr Checklist
      • Pre Commit
      • Ssh
      • Troubleshooting
  • Web
    • Django
      • Activating Grantmatch
      • Forms
      • Manipulating Models
      • Migrating Models
      • Running Tests
      • Signals
      • Troubleshooting
      • Tv Dashboard Stuff
    • Flask
      • Files
      • Passing Data Between Backend And Frontend
      • Web Cache
    • Laravel
      • Adding Js Css To Ftx Templates
      • Database Queries
      • Migrations
      • Troubleshooting
    • Npm
      • Untitled
    • React
      • Formatting
      • Lifecycle Method
      • State
    • Selenium
      • Beautiful Soup Integration
      • Finding Elements
      • Setting Up Headless Chrome
    • Web Scraping
      • Beautifulsoup
      • Data Cleaning
      • PDF To CSV
      • Read PDFs
      • Requests And Json
      • Tor
    • HTMLCSS
  • Work
    • LLM
      • Nemo
Powered by GitBook
On this page
  • WhitespaceTokenizer
  • MitieTokenizer
  • SpacyTokenizer
  • ConveRTTokenizer

Was this helpful?

  1. Machine Learning
  2. Rasa

Tokenizers

WhitespaceTokenizer

  • Whitespace separated

  • "today's" does not split

MitieTokenizer

  • Creates tokens for MITIE entity extractor

SpacyTokenizer

  • requires SpacyNLP

  • supports splitting by quotations

ConveRTTokenizer

  • used when ConveRTFeaturizer is used

  • Code Implementation

PreviousText FeaturizersNextRecurrent Neural Networks

Last updated 3 years ago

Was this helpful?