Contents
When to use test data vs training data?
On the other hand, the test set is just a formalism used to estimate how good the model is. You cannot know for sure how accurate your model it is going to be with future credit applications, but what you can do is to save a small part of your training data, and use it only to check the model’s performance after it has been built.
Is there a difference between clustering and t-SNE?
The difference is subtle, but affects any density- or distance based algorithm. While clustering after t-SNE will sometimes (often?) work, you will never know whether the “clusters” you find are real, or just artifacts of t-SNE. You may just be seeing shapes in clouds. To see this effect, simply generate a multivariate Gaussian distribution.
Is there a problem with the t-SNE algorithm?
The problem with t-SNE is that it does not preserve distances nor density. It only to some extent preserves nearest-neighbors. The difference is subtle, but affects any density- or distance based algorithm.
What is the purpose of training a model?
The ultimate purpose of training a model is to apply it to what you call UNSEEN data. Even in your German credit lending example, at the end of the day you will have a trained model that you could use to predict if new – unseen – credit applications will default or not.
How are data sets divided into training and test sets?
The previous module introduced the idea of dividing your data set into two subsets: training set —a subset to train a model. test set —a subset to test the trained model.
What is a training data set in machine learning?
There are hybrid machine learning models that allow you to use a combination of supervised and unsupervised learning. Training data comes in many forms, reflecting the myriad potential applications of machine learning algorithms. Training datasets can include text (words and numbers), images, video, or audio.
What happens if you don’t have quality training data?
Without a foundation of high-quality training data, even the most performant algorithms can be rendered useless. Indeed, robust machine learning models can be crippled when they are trained on inadequate, inaccurate, or irrelevant data in the early stages.