# Toggle on-off code text
from IPython.display import HTML
HTML('''<script>
code_show=true;
function code_toggle() {
if (code_show){
$('div.input').hide();
} else {
$('div.input').show();
}
code_show = !code_show
}
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()">
<input type="submit" value="Click to toggle on/off the raw code.">
</form>''')
Machine Learning 1 Individual Project¶
In Partial Fulfillment Of The Requirements for the Subject Machine Learning 1
Notebook Prepared by : Matthew R. Maulion
Master of Science in Data Science (2021 Cohort)
Asian Institute of Management, Eugenio Lopez Foundation Building, 123 Paseo de Roxas, Makati City 1226, Philippines
2nd Generation Queens of K-pop:
Classifying Music Tracks Based on Audio Features
1 Executive Summary¶
K-pop is a musical genre originating in South Korea which is characterized by a wide variety of audiovisual concepts. Currently, via the Korean wave, it has spread to other regions other than from South Korea and its neigboring countries. The spread is evident in parts of Latin America, Northeast India, the Middle East, North Africa, and immigrant enclaves of the Western world. This milestone is attributed to the numerous idol groups years back which built the name K-pop has today. This project tackled on two big groups during the 2nd generation of K-pop Idols - Girls' Generation and 2NE1. The main objective is to know what differentiated the music of one group over the other based on audio features obtained via Spotify API.
Through exploratory data analysis, Girls' Generation's music is somehow comparable with 2NE1's in most features that are available for the analysis (danceability, tempo, acousticness, liveness). Though alike, there were instances where an artist have more tracks dominating a certain feature than the other. Interestingly, high valence values were observed for Girls' Generation than 2NE1.
MinMax scaling of features and one-hot encoding of categorical variables with more than classes were done before implementing the data into machine learning models. With a test accuracy of 92.42% beating the baseline accuracy (1.25PCC) of 62.52%, a gradient boosted classifier performed best in terms of correctly predicting datapoints in our test set. Based on the feature importance results from decision tree model variants, valence topped the list as the most important feature that has the greatest impact on the prediction model. This was consistent with what was obtained using SHAP (SHapley Additive exPlanations), valence still topped the feature importance rank.
Overall, Girls' Generation's music is comparable with 2NE1's which makes it difficult to diffentiate one track from the other. However, with the help of machine learning, valence values separate both artist's more distinctly. Hence, with valence as the top predictor, Girls' Generation's tracks exude more musical positiveness (invoking feeling of happiness, cheerfulness, and euphoria) than 2NE1's.
2 Introduction¶
K-pop (short for Korean pop; Korean: 케이팝) is a genre of popular music that originated in South Korea during the early 2000s. Undeniably, it has truly become a global phenomenon thanks to its distinctive blend of addictive melodies characterized by a mixture of modern Western sounds and African-American influences, slick choreography and production, and an endless parade of attractive South Korean performers who spend years training in entertainment agencies and studio systems learning to sing and dance in synchronized perfection (Romano, 2018).
Like any other trend, K-pop had its humble beginnings before it became the force it is today. It all started with 1st Generation idols, H.O.T under SM Entertainment was the first group that debuted which pioneered the so-called K-pop formula - synchronous choreography (group dance) and on-point vocals. There were groups and soloists that came after H.O.T but all of them were short-lived, disbanding after a year or two after debuting. This paved way for 2nd generation groups, which were considered as the ones who cemented K-pop in the music industry. The rise of second-generation idols can be attributed to the 2007 release of Wonder Girls’ "Tell Me", which became so popular that it was named a "syndrome." TVXQ, SS501, Super Junior and Big Bang, all of which debuted earlier, subsequently joined the popularity triggered by Wonder Girls. ("History of Korean idols," n.d.) Unlike first-generation groups, the rivalry present among second-generation idols was evident as concepts and music genres became diversified. And one of the most famous rivalry even up to this date was the one started by Girls' Generation and 2NE1.
Girls' Generation (also know as SNSD - So-nyuh Shi-dae) debuted last 2007 under SM Entertainment. Started with 9 members, now 8 (Jessica Jung left the group last 2014), they had spread the K-pop wave worldwide through their famous hits "Gee" and "I Got a Boy" that devoted fans still listen to until now. On the other hand, 2NE1 debuted two (2) years later under YG Entertainment. Composed of 4 members, their hit songs, "Fire" and "I am the Best" topped numerous music charts worldwide. Unfortunately, the group was forced to disband due to YG last 2016. The group then decided to release a song "Goodbye" last early 2017 as a farewell song to their fans. Though not disbanding, Girls' Generation does not currently have any activities as a group. Current members are doing solos while under the same group name and management.
3 Problem Statement and Objective¶
Given the girl groups' inactivity, fans are still rooted in supporting their songs in music platforms like Spotify. This poses an interesting query, what is in their music that fans are still into them? Why do people prefer one group over the other? More succinctly, what differentiates Girls' Generation's music over 2NE1? This project aims to look at what audio feature/s separate/s Girl's Generation's and 2NE1's music with each other.
4 Methodology¶
4.1 Data Collection¶
The data collection process is done on a separate Jupyter notebook which will be contained in a zip file alongside this main notebook. The data is obtained from a popular music platform - Spotify. It is a digital music, podcast, and video streaming service that gives you access to millions of songs and other content from artists all over the world. Overall, the project obtained 528 datapoints from Spotify through the use of APIs. Additionally, it provided 28 feature columns. However, only 14 (which are audio features) were used in the analysis.
Feature Definitions are explained as follows (Spotify Web API):
| Feature | DataType | Definition |
|---|---|---|
| acousticness | float | A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic. |
| danceability | float | Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable. |
| duration_ms | int | The duration of the track in milliseconds. |
| energy | float | Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy. |
| explicit | int | Whether or not the track has explicit lyrics ( true = 1 it does; false = 0 it does not). |
| instrumentalness | float | Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0. |
| key | int | The key the track is in. Integers map to pitches using standard Pitch Class notation . E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on. |
| liveness | float | Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live. |
| loudness | float | The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db. |
| mode | int | Mode indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived. Major is represented by 1 and minor is 0. |
| speechiness | float | Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks. |
| tempo | float | The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration. |
| time_signature | int | An estimated overall time signature of a track. The time signature (meter) is a notational convention to specify how many beats are in each bar (or measure). |
| valence | float | A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry) |
| artist_names (target variable) | int | 1 for Girls' Generation and 0 for 2NE1 |
4.2 Data Exploration (Exploratory Data Analysis)¶
In this section, data cleaning was implemented (checked for nulls, data abnormalities, etc.). Then after, the data was explored through its featues by looking at their distributions. Further, a lineplot was displayed to explain the music evolution of each K-pop group along the years.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
sns.set()
filename = 'ml_data.csv'
data = pd.read_csv(filename)
data.columns = data.columns.str.lower().str.replace(' ', '_')
# #data.album_release_date = pd.to_datetime(data.album_release_date, format='%Y-%m-%d')
# data.head()
Checked for nulls
data.isnull().sum()
data.info()
from sklearn.preprocessing import LabelEncoder
# Drop all non-numerical features.
to_drop = ['type', 'uri', 'track_href', 'analysis_url', 'track_id',
'artist_ids', 'album_id', 'local', 'track_title',
'album_name', 'duration_(ms)',
'album_release_date','year']
target = LabelEncoder().fit_transform(data.artist_names)
target_names = data.artist_names.unique()
features = data.drop(to_drop, axis=1)
feature_names = features.columns.to_list()
# Convert Explicit from bool to int
features.explicit = features.explicit.astype(int)
print(f'A total of {len(feature_names) - 1} features and {len(target)} data points '
'are considered in the analysis.')
features.head()
fig, axes = plt.subplots(2, 3, figsize=(20,10))
danceability = sns.distplot(features[(features['artist_names'] == '2NE1')]['danceability'],
label='2NE1', ax=axes[0,0])
danceability = sns.distplot(features[(features['artist_names'] == 'Girls\' Generation')]['danceability'],
label='Girls\' Generation', ax=axes[0,0])
danceability.legend()
energy = sns.distplot(features[(features['artist_names'] == '2NE1')]['energy'],
label='2NE1', ax=axes[0,1])
energy = sns.distplot(features[(features['artist_names'] == 'Girls\' Generation')]['energy'],
label='Girls\' Generation', ax=axes[0,1])
energy.legend()
acousticness = sns.distplot(features[(features['artist_names'] == '2NE1')]['acousticness'],
label='2NE1', ax=axes[0,2])
acousticness = sns.distplot(features[(features['artist_names'] == 'Girls\' Generation')]['acousticness'],
label='Girls\' Generation', ax=axes[0,2])
acousticness.legend()
liveness = sns.distplot(features[(features['artist_names'] == '2NE1')]['liveness'],
label='2NE1', ax=axes[1,0])
liveness = sns.distplot(features[(features['artist_names'] == 'Girls\' Generation')]['liveness'],
label='Girls\' Generation', ax=axes[1,0])
liveness.legend()
tempo = sns.distplot(features[(features['artist_names'] == '2NE1')]['tempo'],
label='2NE1', ax=axes[1,1])
tempo = sns.distplot(features[(features['artist_names'] == 'Girls\' Generation')]['tempo'],
label='Girls\' Generation', ax=axes[1,1])
tempo.legend()
valence = sns.distplot(features[(features['artist_names'] == '2NE1')]['valence'],
label='2NE1', ax=axes[1,2])
valence = sns.distplot(features[(features['artist_names'] == 'Girls\' Generation')]['valence'],
label='Girls\' Generation', ax=axes[1,2])
valence.legend()
plt.subplots_adjust(hspace = 0.2)
fig.suptitle("Feature Comparison between Girls' Generation's and 2NE1's Tracks", size=25)
fig.savefig('Feature_comparison.png', dpi=300, bbox_inches='tight', transparent=True)
plt.show()
Observations:
Danceability
- Comparable (spread out from 0.2 to 1)
- Girls' Generation got more danceable tracks than 2NE1
Energy
- Comparable (both are exudes great energy in tracks)
- Girls' Generation got more energetic tracks than 2NE1
Acousticness
- Comparable (both are less acoustic)
- 2NE1 got more less acoustic tracks than Girls' Generation
Liveness
- Comparable (both have tracks that have less liveness)
- Girls' Generation got more tracks that has less liveliness than 2NE1
- Girls' Generation tend to center between (0 - 0.2)
- A huge portion of 2NE1 tracks has less liveness but a portion of it are scattered to having more liveness
Tempo
- Comparable (scattered across tempo levels 60-200) centered at 120-130
- 2NE1 tracks tend to center at 120-140 tempo levels with erratic scatterness towards tails
- Girls' Generation tracks though tend center to 120-140 tempo level, other tracks are more evenly scattered towards tails
Valence
- Interestingly, Girls' Generation tend to get higher valence values than 2NE1
- The valence distribution of 2NE1 is more evenly distributed than gg which tends to clutter to the right
# Replacing years that have zero as value
# Changing Years of Release that were incorrect
data['year'] = data['year'].replace(0, 2014)
data.loc[data['album_name'] == '2NE1 1st Mini Album', 'year'] = 2009
data.loc[data['album_name'] == 'To Anyone', 'year'] = 2010
data.loc[data['album_name'] == '2NE1 2nd Mini Album', 'year'] = 2011
dataviz = data.groupby(['artist_names', 'year']).mean()
dataviz
dataviz.reset_index(level=[0,1], inplace=True)
dataviz.head()
fig, axes = plt.subplots(1, 2, figsize= (15, 7))
cols = ['danceability', 'energy', 'speechiness', 'acousticness', 'valence']
for i in cols:
gg = sns.lineplot(y=dataviz[i].loc[dataviz['artist_names'] == 'Girls\' Generation'],
x=dataviz['year'].loc[dataviz['artist_names'] == 'Girls\' Generation'],
ax=axes[0], label=i)
gg.legend()
gg.set_title('Trend of Girls\' Generation Audio Features (averaged)',size = 14)
gg.set(ylabel="feature value", xlabel = "year")
gg.grid(b=True)
for i in cols:
Tne1 = sns.lineplot(y=dataviz[i].loc[dataviz['artist_names'] == '2NE1'],
x=dataviz['year'].loc[dataviz['artist_names'] == '2NE1'],
ax=axes[1], label=i)
Tne1.legend()
Tne1.set_title('Trend of 2NE1 Audio Features (averaged)',size = 14)
Tne1.set(ylabel="feature value", xlabel = "year")
Tne1.grid(b=True)
plt.subplots_adjust(hspace = 0.2)
plt.show()
Girls' Generation's Music Evolution
Girls' Generation has been somehow consistent with the tracks they produced. The selected audio features as seen in the graph are at their highest between 2013-2014 (inclusive). The reason for this sudden change might be attributed to the management's change in concept/genre for the girl group in the hopes of gaining traction in other fan segments. However, along the years starting from 2015, the tracks they have produced are gradually becoming the ones they producd in their earlier years in terms of audio features.
2NE1's Music Evolution
2NE1's evolution is evidently erratic unlike Girls' Generation's. Given its erracticness, most of its features follow a certain pattern where its audio feature values started high (2009), then it consitently decreased as years passed by. One interesting thing to note is its acousticness which started really low but consistently grew considerably starting 2011. Similar to Girls' Generation, a sudden sharp shift of audio features can be observed between 2013-2015 (inclusive) which might be caused by changes in the group's musical choice/production.
4.3 Data Preparation¶
from sklearn.preprocessing import LabelEncoder
# Drop all non-numerical features.
to_drop = ['type', 'uri', 'track_href', 'analysis_url', 'track_id',
'artist_ids', 'album_id', 'local', 'track_title',
'artist_names', 'album_name', 'duration_(ms)',
'album_release_date', 'year']
target = LabelEncoder().fit_transform(data.artist_names)
target_names = data.artist_names.unique()
features = data.drop(to_drop, axis=1)
feature_names = features.columns.to_list()
# Convert Explicit from bool to int
features.explicit = features.explicit.astype(int)
print(f'A total of {len(feature_names)} features and {len(target)} data point '
'are considered in the analysis.')
features.head()
One-hot Encoding of Categorical Features
# `key` is considered categorical variables because their values
# does not represent an order but a classification.
features_one = features.copy()
#features_one.year = features_one.year.astype(str)
features_one.key = features_one.key.astype(str)
features_one = pd.get_dummies(features_one)
features_one_names = features_one.columns.to_list()
print(f'Size of the one-hot-encoded (OHE) features is {features_one.shape}\n'
'OHE introduced an additional of '
f'{len(features_one_names)-len(feature_names)} features.')
features_one.head(5)
# import seaborn as sns
# to_plot = features.join(data.artist_names)
# sns.pairplot(to_plot, hue='artist_names', diag_kind='kde');
4.4 Machine Learning Implementation¶
import time
import warnings
from itertools import product
from collections import Counter
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC, SVC
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.exceptions import ConvergenceWarning
from IPython.display import display, HTML
warnings.filterwarnings("ignore", category=FutureWarning)
# warnings.filterwarnings("ignore", category=RuntimeWarning)
warnings.filterwarnings("ignore", category=ConvergenceWarning)
Machine Learning Algorithms in Python Functions¶
(click arrow-head at the side to see code when in code mode)
def model_comparator(X, y, scaler=None, \
feature_name=None, target_name=None, trials=100, \
Cs=None, gammas=None, ratio=25):
start_time = time.time()
knn = knn_trials(X, y, scaler=scaler, summarize=True,
trials=trials, ratio=ratio)
print(f'knn Complete : {(time.time() - start_time):0.2f} secs.')
start_time = time.time()
log_l1 = logr_trials(X, y, trials=trials, scaler=scaler,
reg='l1', summarize=True, Cs=Cs,
feature_name=feature_name, target_name=target_name)
print(f'log_l1 Complete : {(time.time() - start_time):0.2f} secs.')
start_time = time.time()
log_l2 = logr_trials(X, y, trials=trials, scaler=scaler,
reg='l2', summarize=True, Cs=Cs,
feature_name=feature_name, target_name=target_name)
print(f'log_l2 Complete : {(time.time() - start_time):0.2f} secs.')
start_time = time.time()
lsvm_l1 = lsvm_trials(X, y, trials=trials, scaler=scaler,
reg='l1', summarize=True, Cs=Cs,
feature_name=feature_name, target_name=target_name)
print(f'lsvm_l1 Complete : {(time.time() - start_time):0.2f} secs.')
start_time = time.time()
lsvm_l2 = lsvm_trials(X, y, trials=trials, scaler=scaler,
reg='l2', summarize=True, Cs=Cs,
feature_name=feature_name, target_name=target_name)
print(f'lsvm_l2 Complete : {(time.time() - start_time):0.2f} secs.')
start_time = time.time()
nsvm_rbf = nsvm_trials(X, y, trials=trials, scaler=scaler, Cs=Cs,
summarize=True, gammas=gammas)
print(f'nsvm_rbf Complete : {(time.time() - start_time):0.2f} secs.')
cols = ['Machine Learning Method', 'Test Accuracy',
'Best Parameter', 'Top Predictor Variable']
df = pd.DataFrame([knn, log_l1, log_l2, lsvm_l1, lsvm_l2, nsvm_rbf],
columns=cols)
display(HTML(df.to_html().replace('\\n', '<br>')))
def knn_trials(X, y, trials=100, scaler=None, \
neighborhoods=None, ratio=0.25, graph=False,\
summarize=False):
# Seed Generation
# seeds = []
# while len(seeds) < trials:
# s = np.random.randint(0, trials*10)
# if s not in seeds: seeds.append(s)
# seeds = np.array(seeds)
seeds = np.arange(trials)
# Defining the neighborhoods
if neighborhoods is None:
neighborhoods = np.arange(1, (len(X)//4) - 1, 1)
else:
neighborhoods = np.arange(1, neighborhoods + 1, 1)
train_score = pd.DataFrame()
test_score = pd.DataFrame()
for col, seed in enumerate(seeds):
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=ratio,
random_state=seed,
stratify=y)
if scaler is not None:
try:
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
except BaseException as e:
print(f'ERROR: Ensure that scaler is a Scaler class object')
print(f'{e}')
break
# Result per seed/ trial
train_accuracy = []
test_accuracy = []
for n in neighborhoods:
knn = KNeighborsClassifier(n_neighbors=n)
knn.fit(X_train, y_train)
train_accuracy.append(knn.score(X_train, y_train))
test_accuracy.append(knn.score(X_test, y_test))
# Storing the result in the dataframe
train_score[col] = train_accuracy
test_score[col] = test_accuracy
# Summarizing the results
mean_train = train_score.mean(axis=1)
mean_test = test_score.mean(axis=1)
ideal_n = neighborhoods[np.argmax(mean_test.values)]
if graph:
# Define values to plot the error/ standard deviation
train_err = (np.array([[-1],[1]]) *
(train_score.std(axis=1) + mean_train).values)
test_err = (np.array([[-1],[1]]) *
(test_score.std(axis=1) + mean_test).values)
# Plotting Paraphernalia
plt.figure(figsize=(5,5))
plt.plot(neighborhoods, mean_train, label='train score',
color='blue', marker='o', linestyle='dashed')
plt.plot(neighborhoods, mean_test, label='test score',
color='green', marker='^', linestyle='dashed')
plt.fill_between(neighborhoods, test_err[0], test_err[1],
alpha=0.2, color='green')
plt.title(f'Mean kNN Accuracy graphs of the dataset')
plt.title('Accuracy Behavior of kNN Classifier at varying '
f'n-Neighbors\ntrials={trials}, test_ratio={ratio},\n'
f'best_n={ideal_n}, max_score={mean_test.max():0.4f}')
plt.xlabel('Neighbors, n')
plt.ylim(min(mean_test)*3/4, 1)
plt.ylabel('Accuracy/ Score')
plt.legend(loc='lower left')
plt.show()
if summarize:
return ['kNN', mean_test.max(), f'n_neighbor={ideal_n}', 'NA']
else:
return [mean_train, mean_test]
def logr_trials(X, y, trials=100, scaler=False, \
ratio=0.25, reg='l2', graph=False, \
Cs=None, summarize=False, feature_name=None,\
target_name=None):
# Seed Generation
# seeds = []
# while len(seeds) < trials:
# s = np.random.randint(0, trials*10)
# if s not in seeds: seeds.append(s)
# seeds = np.array(seeds)
seeds = np.arange(trials)
# Defining the C Parameter
if Cs is None:
Cs = [1e-8, 1e-4, 1e-3, 1e-2, 0.1, 0.2, 0.4, 0.75, 1,
1.5, 3, 5, 10, 15, 20, 100, 300, 1000, 5000]
# Defining solver
if reg == 'l1':
solver = 'liblinear'
else:
solver = 'lbfgs'
train_score = pd.DataFrame()
test_score = pd.DataFrame()
weighted_coefs = pd.DataFrame()
for col, seed in enumerate(seeds):
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=ratio,
random_state=seed,
stratify=y)
if scaler is not None:
try:
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
except BaseException as e:
print(f'ERROR: Ensure that scaler is a Scaler class object')
print(f'{e}')
break
training_accuracy = []
test_accuracy = []
coefs = []
for C in Cs:
logr = LogisticRegression(C=C, penalty=reg, solver=solver,
max_iter=10000)
logr.fit(X_train, y_train)
training_accuracy.append(logr.score(X_train, y_train))
test_accuracy.append(logr.score(X_test, y_test))
coefs.append(logr.coef_)
train_score[col] = training_accuracy
test_score[col]= test_accuracy
weighted_coefs[col] = coefs
mean_train = train_score.mean(axis=1)
mean_test = test_score.mean(axis=1)
mean_coef = pd.Series(np.mean(weighted_coefs.values, axis=1))
ind = np.argmax(mean_test.values)
# Generating the list of Relevant Features
top_ind = np.argmax(abs(mean_coef[ind]), axis=1)
top_wts = abs(mean_coef[ind]).max(axis=1)
# Dummy feature_name Generator
if feature_name is None:
feature_name = [f'feature{i}' for i in range(X.shape[1])]
top_name = [feature_name[i] for i in top_ind]
# Generating dummy target_name
if target_name is None:
target_name = [f'class{i}' for i in np.unique(y)]
top_report = '\n'.join([f'{t}:{f} = {w:0.4f}'
for t, f, w in zip(target_name,
top_name,
top_wts)])
if graph:
# Define values to plot the error/ standard deviation
train_err = (np.array([[-1],[1]]) *
(train_score.std(axis=1) + mean_train).values)
test_err = (np.array([[-1],[1]]) *
(test_score.std(axis=1) + mean_test).values)
# Plot particulars
fig, ax = plt.subplots(figsize=(7,7))
ax.set_xscale('log')
plt.plot(Cs, mean_train, label="training accuracy",
color='blue', marker='o', linestyle='dashed')
plt.plot(Cs, mean_test, label="test accuracy",
color='green', marker='^', linestyle='-')
plt.fill_between(Cs, test_err[0], test_err[1],
alpha=0.2, color='green')
plt.ylabel('Accuracy/ Score, $R^2$')
plt.ylim(min(mean_test)*3/4, 1)
plt.xlabel('Regularization parameter, $C$')
plt.title(f'$R^2$ Behavior of {reg.upper()}-Regularized Logistics '
'Regression\nagainst the Regularization Parameter, $C$\n'
f'trials={trials}, test_ratio={ratio},\nmax'
f'$C$={Cs[ind]}, max$R^2$={mean_test.max():0.4f}')
plt.legend(loc='lower left')
if summarize:
return [f'Logistic Regression with {reg.upper()}-Regularization',
mean_test.max(),
f'C = {Cs[ind]}', top_report]
else:
return [mean_train, mean_test, mean_coef]
def lsvm_trials(X, y, trials=100, scaler=False, graph=False, \
ratio=0.25, reg='l2', Cs=None, \
summarize=False, feature_name=None, \
target_name=None):
# Seed Generation
# seeds = []
# while len(seeds) < trials:
# s = np.random.randint(0, trials*10)
# if s not in seeds: seeds.append(s)
# seeds = np.array(seeds)
seeds = np.arange(trials)
# Defining the C Parameter
if Cs is None:
Cs = [1e-8, 1e-4, 1e-3, 1e-2, 0.1, 0.2, 0.4, 0.75, 1,
1.5, 3, 5, 10, 15, 20, 100, 300, 1000, 5000]
train_score = pd.DataFrame()
test_score = pd.DataFrame()
weighted_coefs = pd.DataFrame()
for col, seed in enumerate(seeds):
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=ratio,
random_state=seed,
stratify=y)
if scaler is not None:
try:
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
except BaseException as e:
print(f'ERROR: Ensure that scaler is a Scaler class object')
print(f'{e}')
break
training_accuracy = []
test_accuracy = []
coefs = []
for C in Cs:
lsvm = LinearSVC(C=C, penalty=reg, dual=False, max_iter = 1000000)
lsvm.fit(X_train, y_train)
training_accuracy.append(lsvm.score(X_train, y_train))
test_accuracy.append(lsvm.score(X_test, y_test))
coefs.append(lsvm.coef_)
train_score[col] = training_accuracy
test_score[col]= test_accuracy
weighted_coefs[col] = coefs
mean_train = train_score.mean(axis=1)
mean_test = test_score.mean(axis=1)
mean_coef = pd.Series(np.mean(weighted_coefs.values, axis=1))
ind = np.argmax(mean_test.values)
# Generating the list of Relevant Features
top_ind = np.argmax(abs(mean_coef[ind]), axis=1)
top_wts = abs(mean_coef[ind]).max(axis=1)
# Dummy feature_name Generator
if feature_name is None:
feature_name = [f'feature{i}' for i in range(X.shape[1])]
top_name = [feature_name[i] for i in top_ind]
# Generating dummy target_name
if target_name is None:
target_name = [f'class{i}' for i in np.unique(y)]
top_report = '\n'.join([f'{t}:{f} = {w:0.4f}'
for t, f, w in zip(target_name,
top_name,
top_wts)])
if graph:
# Define values to plot the error/ standard deviation
train_err = (np.array([[-1],[1]]) *
(train_score.std(axis=1) + mean_train).values)
test_err = (np.array([[-1],[1]]) *
(test_score.std(axis=1) + mean_test).values)
# Plot particulars
fig, ax = plt.subplots(figsize=(7,7))
ax.set_xscale('log')
plt.plot(Cs, mean_train, label="training accuracy",
color='blue', marker='o', linestyle='dashed')
plt.plot(Cs, mean_test, label="test accuracy",
color='green', marker='^', linestyle='-')
plt.fill_between(Cs, test_err[0], test_err[1],
alpha=0.2, color='green')
plt.ylabel('Accuracy/ Score, $R^2$')
plt.ylim(min(mean_test)*3/4, 1)
plt.xlabel('Regularization parameter, $C$')
plt.title(f'$R^2$ Behavior of {reg.upper()}-Regularized LinearSVM '
'\nagainst the Regularization Parameter, $C$\n'
f'trials={trials}, test_ratio={ratio},\nmax'
f'$C$={Cs[ind]}, max$R^2$={mean_test.max():0.4f}')
plt.legend(loc='lower left')
if summarize:
return [f'Linear SVM with {reg.upper()}-Regularization',
mean_test.max(),
f'C = {Cs[ind]}', top_report]
else:
return [mean_train, mean_test, mean_coef]
def nsvm_trials(X, y, trials=100, scaler=False, graph=False, \
ratio=0.25, Cs=None, gammas=None,\
summarize=False):
# Seed Generation
# seeds = []
# while len(seeds) < trials:
# s = np.random.randint(0, trials*10)
# if s not in seeds: seeds.append(s)
# seeds = np.array(seeds)
seeds = np.arange(trials)
# Defining the C Parameter
if Cs is None:
Cs = [1e-8, 1e-4, 1e-3, 1e-2, 0.1, 0.2, 0.4, 0.75, 1,
1.5, 3, 5, 10, 15, 20, 100, 300, 1000, 5000]
# Defining the gamma Parameter
if gammas is None:
gammas = [1e-8, 1e-4, 1e-3, 1e-2, 0.1, 0.2, 0.4, 0.75, 1,
1.5, 3, 5, 10, 15, 20, 100, 300, 1000, 5000]
train_score = (pd.DataFrame(list(product(Cs, gammas)),
columns=['C', 'gamma'])
.set_index(['C', 'gamma']))
test_score = (pd.DataFrame(list(product(Cs, gammas)),
columns=['C', 'gamma'])
.set_index(['C', 'gamma']))
# weighted_coefs = pd.DataFrame()
for col, seed in enumerate(seeds):
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=ratio,
random_state=seed,
stratify=y)
if scaler is not None:
try:
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
except BaseException as e:
print(f'ERROR: Ensure that scaler is a Scaler class object')
print(f'{e}')
break
training_accuracy = []
test_accuracy = []
# coefs = []
for C, gamma in product(Cs, gammas):
nsvm = SVC(kernel='rbf', gamma=gamma, C=C, max_iter = 1000000)
nsvm.fit(X_train, y_train)
training_accuracy.append(nsvm.score(X_train, y_train))
test_accuracy.append(nsvm.score(X_test, y_test))
# coefs.append(nsvm.coef_)
train_score[col] = training_accuracy
test_score[col]= test_accuracy
# rows are defined by the sequence of c_gamma
# weighted_coefs[col] = coefs
mean_train = train_score.mean(axis=1)
mean_test = test_score.mean(axis=1)
# mean_coef = pd.Series(np.mean(weighted_coefs.values, axis=1))
ind = np.argmax(mean_test.values)
# Best parameters
C, g = mean_test.index[ind]
if graph:
# Extract information for plotting
gamma_X = mean_test.loc[(C)].index.to_list()
mean_test_Y = mean_test.loc[(C)].values
mean_train_Y = mean_train.loc[(C)].values
# Define values to plot the error/ standard deviation
train_err = (np.array([[-1],[1]]) *
(train_score.std(axis=1).loc[(C)] +
mean_train.loc[(C)]).values)
test_err = (np.array([[-1],[1]]) *
(test_score.std(axis=1).loc[(C)] +
mean_test.loc[(C)]).values)
# Plot particulars
fig, ax = plt.subplots(figsize=(7,7))
ax.set_xscale('log')
plt.plot(gamma_X, mean_train_Y, label="training accuracy",
color='blue', marker='o', linestyle='dashed')
plt.plot(gamma_X, mean_test_Y, label="test accuracy",
color='green', marker='^', linestyle='-')
plt.fill_between(gamma_X, test_err[0], test_err[1],
alpha=0.2, color='green')
plt.ylabel('Accuracy/ Score, $R^2$')
plt.ylim(min(mean_test)*3/4, 1)
plt.xlabel('Radial parameter, $\gamma$')
plt.title(f'$R^2$ Behavior of NonlinearSVM (Radial Based Function)'
'\nagainst the Radial Parameter, $\gamma$\n'
f'trials={trials}, test_ratio={ratio}, max_C = {C}\nmax_'
f'$\gamma$={g}, max_$R^2$={mean_test.max():0.4f}')
plt.legend(loc='lower left')
if summarize:
return [f'Nonlinear SVM (Radial Basis Function)', mean_test.max(),
f'gamma = {g}\n C = {C}', 'NA']
else:
return [mean_train, mean_test]
def PCC_calculator(target, target_name=[], dataset_name='given', graph=True):
"""Returns the Proportion Chance Criteria (PCC) of a dataset's
target labels.
Parameter
---------
target : 1-D np.array or pd.Series or list
Countable list of targets. This will be passed through the Counter
function.
target_name : list of strings
Numerical or non-numerical labels for the targets to be used as labels
of the xtickmarks of the graph
dataset_name : string
Name of the dataset to be used in the graph/ visualization
graph : Bool
If `True` a frequency graph of the targets will be shown
Return
------
PCC : Float
1.25PCC value of the target
"""
target_count = Counter(target)
df_target_count = pd.DataFrame.from_dict(target_count,
orient='index',
columns=['n_i'])
PCC = ((df_target_count.n_i/ df_target_count.n_i.sum())**2).sum()*1.25*100
if not any(target_name):
target_name = df_target_count.index.to_list()
if graph:
ax = df_target_count.plot(kind='bar', rot=0, figsize=(7,7))
ax.get_legend().remove()
ax.set_ylabel('Frequency of each target')
ax.set_xlabel(f'Target labels of the {dataset_name} Dataset')
ax.set_title(f'"Targets" Distribution of the {dataset_name} Dataset\n'
f'(1.25PCC = {PCC:0.2f}%)')
ax.set_xticklabels(target_name)
return PCC
4.1 Proportion Change Criteria¶
pcc_value = PCC_calculator(target, target_name=target_names,
dataset_name='Queens of Kpop', graph=True)
5 Results and Discussion¶
A number of combinations were used in obtaining the highest test accuracy for the dataset. The train/test split ratio chosen is 0.25 (25%).
Below are the results of the tests:
Model Results using non-OHE Features¶
model_comparator(features, target, feature_name=feature_names,
target_name=target_names, trials=50)
Model Results using non-OHE Features (Using MinMaxScaler())¶
model_comparator(features, target, scaler=MinMaxScaler(),
feature_name=feature_names, target_name=target_names,
trials=50)
Model Results using OHE Features¶
model_comparator(features_one, target,
feature_name=features_one_names, target_name=target_names,
trials=50)
Model Results using OHE Features (Using MinMaxScaler())¶
model_comparator(features_one, target, scaler=MinMaxScaler(),
feature_name=features_one_names, target_name=target_names,
trials=50)
Machine Learning Algorithm (cont. Decision Tree)¶
Model Results using non-OHE Features¶
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
lahat_training = pd.DataFrame()
lahat_test = pd.DataFrame()
for seedN in range(1,10,1):
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(features, target, stratify=target,
test_size=0.25, random_state=seedN)
training_accuracy = []
test_accuracy = []
maxdepth_settings = range(1, 50) # try n_neighbors from 1 to 50
for depth in maxdepth_settings:
tree = DecisionTreeClassifier(max_depth=depth) # build the model
tree.fit(X_train, y_train)
training_accuracy.append(tree.score(X_train, y_train)) # record training set accuracy
test_accuracy.append(tree.score(X_test, y_test)) # record generalization accuracy
lahat_training[seedN]=training_accuracy
lahat_test[seedN] = test_accuracy
fig = plt.figure(figsize=(10, 4))
plt.errorbar(maxdepth_settings, lahat_training.mean(axis=1),
yerr=lahat_training.var(axis=1), label="training accuracy")
plt.errorbar(maxdepth_settings, lahat_test.mean(axis=1),
yerr=lahat_test.std(axis=1)/3, label="test accuracy")
plt.ylabel("Accuracy")
plt.xlabel("max_depth")
plt.title("Decision Tree Results")
plt.legend()
bestdepth=np.argmax(lahat_test.mean(axis=1))+1
print("Highest Average Test Set Achieved = %f" % np.amax(lahat_test.mean(axis=1)))
print("Best max_depth Parameters = %d" %bestdepth )
Machine Learning Algorithm (cont. Random Forest)¶
Model Results using non-OHE Features¶
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
lahat_training = pd.DataFrame()
lahat_test = pd.DataFrame()
for seedN in range(1,10,1):
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(features, target, stratify=target,
test_size=0.25, random_state=seedN)
training_accuracy = []
test_accuracy = []
maxdepth_settings = range(1, 50) # try n_neighbors from 1 to 50
for depth in maxdepth_settings:
forest = RandomForestClassifier(max_depth=depth) # build the model
forest.fit(X_train, y_train)
training_accuracy.append(forest.score(X_train, y_train)) # record training set accuracy
test_accuracy.append(forest.score(X_test, y_test)) # record generalization accuracy
lahat_training[seedN]=training_accuracy
lahat_test[seedN] = test_accuracy
fig = plt.figure(figsize=(10, 4))
plt.errorbar(maxdepth_settings, lahat_training.mean(axis=1),
yerr=lahat_training.var(axis=1), label="training accuracy")
plt.errorbar(maxdepth_settings, lahat_test.mean(axis=1),
yerr=lahat_test.std(axis=1)/3, label="test accuracy")
plt.ylabel("Accuracy")
plt.xlabel("max_depth")
plt.legend()
bestdepth=np.argmax(lahat_test.mean(axis=1))+1
print("Highest Average Test Set Achieved = %f" % np.amax(lahat_test.mean(axis=1)))
print("Best max_depth Parameters = %d" %bestdepth )
Machine Learning Algorithm (cont. Gradient Boosting Method)¶
Model Results using non-OHE Features¶
#grid search
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(features, target,
test_size = 0.25,
random_state = 1 )
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import GridSearchCV
# from sklearn.grid_search import GridSearchCV
param_grids = {'learning_rate': [.2, 0.1, 0.05, 0.02, 0.5, 0.01, 0.015],
'max_depth': [3, 4, 6, 10, 14, 16, 19],
'min_samples_leaf': [2, 3, 4, 5, 6, 7, 8, 9],
'max_features': [.7,.6,.5,.3,.2,.1]
}
est = GradientBoostingRegressor(n_estimators=400)
gs_cv = GridSearchCV(est, param_grids, n_jobs=-1).fit(X_train, y_train)
print(gs_cv.best_params_)
from sklearn.ensemble import GradientBoostingClassifier
X_train, X_test, y_train, y_test = train_test_split(features, target, random_state=0)
gbrt = GradientBoostingClassifier(learning_rate = 0.05,
max_depth = 16,
max_features = 0.2,
min_samples_leaf = 8,
random_state=44)
gbrt.fit(X_train, y_train)
print("accuracy on training set: %f" % gbrt.score(X_train, y_train))
print("accuracy on test set: %f" % gbrt.score(X_test, y_test))
Model Results using OHE Features¶
#grid search
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(features_one, target,
test_size = 0.25,
random_state = 1 )
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import GridSearchCV
# from sklearn.grid_search import GridSearchCV
param_grids = {'learning_rate': [.2, 0.1, 0.05, 0.02, 0.5, 0.01, 0.015],
'max_depth': [3, 4, 6, 10, 14, 16, 19],
'min_samples_leaf': [2, 3, 4, 5, 6, 7, 8, 9],
'max_features': [.7,.6,.5,.3,.2,.1]
}
est = GradientBoostingRegressor(n_estimators=400)
gs_cv = GridSearchCV(est, param_grids, n_jobs=-1).fit(X_train, y_train)
print(gs_cv.best_params_)
from sklearn.ensemble import GradientBoostingClassifier
X_train, X_test, y_train, y_test = train_test_split(features, target, random_state=0)
gbrt = GradientBoostingClassifier(learning_rate = 0.05,
max_depth = 16,
max_features = 0.2,
min_samples_leaf = 8,
random_state=44)
gbrt.fit(X_train, y_train)
print("accuracy on training set: %f" % gbrt.score(X_train, y_train))
print("accuracy on test set: %f" % gbrt.score(X_test, y_test))
Based on the results obtained above, the three (3) highest test accuracies were obtained from:
- Nonlinear SVM (Radial Basis Function)
- Test Accuracy: 87.28%
- Parameters: {gamma = 10.0, C = 15.0)
- Random Forest
- Test Accuracy: 90.23%
- Parameteres: {max_depth = 25 in 10 trials}
- Gradient Boosting Method
- Test Accuracy: 92.42%
- Parameters: {'learning_rate': 0.05, 'max_depth': 14 or 19, 'max_features': 0.2, 'min_samples_leaf': 7 or 8}
5.1 Feature Importance of Top Models¶
def plot_feature_importances(model):
n_features=features.shape[1]
plt.barh(range(n_features), model.feature_importances_, align='center')
plt.yticks(np.arange(n_features), feature_names)
plt.xlabel("Feature importance")
plt.ylabel("Feature")
plt.ylim(-1,n_features)
plt.savefig("DT Feature Importance.png", bbox_inches='tight',
dpi=300, transparent=True)
Feature Importance of Gradient Boosted Model¶
plot_feature_importances(gbrt)
Given the Gradient Boosting Classifier feature importance results above, it can be observed that valence is the most important feature that has the highest impact in the prediction. This is followed by duration_ms and energy.
Feature Importance of Random Forest Model¶
plot_feature_importances(forest)
Given the Random Forest feature importance results above, it can be observed that valence is still the most important feature that has the highest impact in the prediction. This is followed by danceability and energy.
Feature Importance of Random Forest Model¶
plot_feature_importances(tree)
Given the Decision Tree feature importance results above, it can be observed that valence is the most important feature that has the highest impact in the prediction. This is followed by tempo and danceability.
SHAP (SHapley Additive exPlanations)
SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the output of any machine learning model. It connects optimal credit allocation with local explanations using the classic Shapley values from game theory and their related extensions (see papers for details and citations.) (Lundberg, 2020)
import shap
#feature importance SHAP
#SHAP (SHapley Additive exPlanations) package in Python to get closer to explainable machine learning results.
explainer = shap.TreeExplainer(gbrt, X_test)
values = explainer.shap_values(X_test, check_additivity=False)
shap.summary_plot(values, X_test)
Interpreting the SHAP graph of the Gradient Boosting Classifier Using Tree Explainer
The higher these features rank on the left y-axis, the more impact they have on the model prediction. The magnitude of positive and negative impact on model output is shown by how spread out the dots are from the center. Positive influence means influence toward prediction that the track is from Girls' Generation, and negative influence toward the other side (2NE1 tracks).
Colors blue and red in the chart indicate the value of a feature. For instance, the red dots represent the higher levels of
valenceand the blue represents the lower values of this feature.Combining these pieces of information, the plot above shows that
valenceis the most important feature for model prediction. The higher the level of speechiness, the more that song influenced our machines in predicting it to be by Girls' Generation.
# saving the shap graph
# shap.summary_plot(values, X_test, show=False)
# plt.savefig('shap.png', bbox_inches='tight',
# dpi=300, transparent=True)
5.2 Model Evaluation¶
Confusion Matrix¶
A confusion matrix is used in order to determine how many observations were correctly or incorrectly classified. It compares the observed and the predicted outcome values and shows the number of correct and incorrect predictions categorized by type of outcome.
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
X_train, X_test, y_train, y_test = train_test_split(features, target, stratify= target,
test_size=0.25, random_state=1)
gbrt = GradientBoostingClassifier(learning_rate = 0.05,
max_depth = 16,
max_features = 0.2,
min_samples_leaf = 8,
random_state=44)
gbrt.fit(X_train, y_train)
y_pred = gbrt.predict(X_test)
confmat = confusion_matrix(y_true=y_test, y_pred=y_pred)
fig, ax = plt.subplots(figsize=(5, 5))
ax.matshow(confmat, cmap=plt.cm.Blues, alpha=0.3)
for i in range(confmat.shape[0]):
for j in range(confmat.shape[1]):
ax.text(x=j, y=i, s=confmat[i, j], va='center', ha='center')
plt.xlabel('predicted label')
plt.ylabel('true label')
plt.show()
print(classification_report(y_test, y_pred))
Findings:
The diagonal elements of the confusion matrix indicate correct predictions, while the off-diagonals represent incorrect predictions. So, the correct classification rate or accuracy is the sum of the number on the diagonal divided by the sample size in the test data. In our example, that is (56 + 58)/132 = 86%.
For our dataset, let us assume that Girls' Generation class (1) and 2NE1 is class (0).
Precision is the proportion of true positives among all the individuals that have been predicted to be positive by the model. This represents the accuracy of a predicted positive outcome. Sensitivity (or Recall), which is the True Positive Rate (TPR) or the proportion of identified positives among positive population (class = 1) In our case, 56/(56+9) = 86% for both precision and recall for class (0). On the other hand, precision and recall for class (1) is 87% respectively.
ROC Curve (Receiver Operating Characteristic)¶
The ROC curve (or receiver operating characteristics curve ) is a popular graphical measure for assessing the performance or the accuracy of a classifier, which corresponds to the total proportion of correctly classified observations. (Statistical tools for high-throughput data analysis, 2018)
from sklearn.metrics import roc_curve, roc_auc_score
#Prediction Probabilities
r_probs = [0 for i in range(len(y_test))]
gbrt_probs = gbrt.predict_proba(X_test)
forest_probs = forest.predict_proba(X_test)
tree_probs = tree.predict_proba(X_test)
gbrt_probs = gbrt_probs[:,1]
forest_probs = forest_probs[:,1]
tree_probs = tree_probs[:,1]
r_auc = roc_auc_score(y_test, r_probs)
gbrt_auc = roc_auc_score(y_test, gbrt_probs)
forest_auc = roc_auc_score(y_test,forest_probs)
tree_auc = roc_auc_score(y_test, tree_probs)
print('Random (Chance) Prediction: AUROC (%) = ', r_auc*100)
print('Gradient Boosting Classifier: AUROC (%) = ', gbrt_auc*100)
print('Random Forest Classifier: AUROC (%) = ', forest_auc*100)
print('Decision Tree Classifier: AUROC (%) = ', tree_auc*100)
r_fpr, r_tpr, _ = roc_curve(y_test, r_probs)
gbrt_fpr, gbrt_tpr, _ = roc_curve(y_test, gbrt_probs)
forest_fpr, forest_tpr, _ = roc_curve(y_test, forest_probs)
tree_fpr, tree_tpr, _ = roc_curve(y_test, tree_probs)
plt.figure(figsize=(9,9))
plt.plot(r_fpr, r_tpr, linestyle='--', label='Random (Chance) Prediction (AUROC = %0.4f)' %r_auc )
plt.plot(gbrt_fpr, gbrt_tpr, linestyle='-.', label='Gradient Boosting Classifier (AUROC = %0.4f)' %gbrt_auc )
plt.plot(forest_fpr, forest_tpr, linestyle='-.', label='Random Classifier (AUROC = %0.4f)' %forest_auc )
plt.plot(tree_fpr, tree_tpr, linestyle='-.', label='Decision Tree Classifier (AUROC = %0.4f)' %tree_auc )
plt.title('ROC Curve', size=20)
plt.xlabel('False Positive Rate', size=10)
plt.ylabel('True Positive Rate', size=10)
plt.legend()
plt.savefig('roc.png', dpi=300, bbox_inches='tight', transparent=True)
plt.show()
Findings:
For the AUC, we want the curve to cover the area up until the topleft corner of the graph. This signifies that we want AUC to be as close to 1. Interestingly, the Random Forest Classifier (98.86%) got a better AUROC score than the Gradient Boosting Classifier (93.57) which topped the list of models having the highest test accuracy score. Why is this?
An explanation by Christos Iraklis Tsatsoulis (2017) click here stated that the (decision) threshold in binary classification is the value above which we decide to label a sample as 1 (recall that probabilistic classifiers actually return a value p in [0, 1], usually interpreted as a probability - in scikit-learn it is what predict_proba returns).
Now, this threshold, in methods like scikit-learn predict which return labels (1/0), is set to 0.5 by default, but this is not the only possibility, and it may not even be desirable in come cases (imbalanced data, for example).
The point to take home is that:
- When you ask for score (which under the hood uses predict, i.e. labels and not probabilities), you have also implicitly set this threshold to 0.5
- When you ask for AUC (which, in contrast, uses probabilities returned with predict_proba), no threshold is involved, and you get (something like) the accuracy averaged across all possible thresholds
6 Conclusion¶
K-pop has considerably dominated the music industry around the world. This milestone is attributed to the numerous idol groups years back which built the name K-pop has today. This project tackled on two big groups during the 2nd generation of K-pop Idols - Girls' Generation and 2NE1. The main objective is to know what differentiated the music of one group over the other based on audio features obtained via Spotify. The results of the analysis are as follows:
As a result of the Data Exploration through distribution plots, it was observed that Girls' Generation's music is somehow comparable with 2NE1's in most features that are available for the analysis (
danceability,tempo,acousticness,liveness). Though similar, there were instances where an artist have more tracks dominating a certain feature than the other artist (vice versa). Interestingly, highvalencevalues were observed for Girls' Generation than 2NE1.Most machine Learning models used in the analysis out-performed the baseline test accuracy of 62.52% (1.25 Proportion Chance Criteria) with Gradient Boosting Classifier obtaining the highest test-accuracy of 92.42%. Interestingly, in model evaluation, the Random Forest classifier (98.86%) got the highest AUROC beating Gradient Boosting Classifier which was only at (93.57%)
Based on the feature importance results from Decision Tree, Gradient Boosting Method, and Random Forest model,
valencetopped the list as the most important feature that has the greatest impact on the prediction model. This was consistent with what was obtained using SHAP (SHapley Additive exPlanations),valencestill topped the feature importance rank.
Conclusively, Girls' Generation's music has a lot of similarities with the ones from 2NE1's which makes it difficult to diffentiate one track from the other. However, with the help of machine learning, valence values separate both artist's more distinctly. Hence, with valence as the top predictor, Girls' Generation's tracks exude more musical positiveness (invoking feeling of happiness, cheerfulness, and euphoria) than 2NE1's.
7 References¶
- Jung H. (Oct 2018) The Data Science of K-Pop: Understanding BTS through data and A.I. Towards Data Science. Retrieved from: https://towardsdatascience.com/the-data-science-of-k-pop-understanding-bts-through-data-and-a-i-part-1-50783b198ac2
- Kassambara A. (Nov 2018) Evaluation of Classification Model Accuracy: Essentials. STHDA Statistical tools for high-throughput data analysis. Retrieved from: http://www.sthda.com/english/articles/36-classification-methods-essentials/143-evaluation-of-classification-model-accuracy-essentials/
- Lundberg, S. ( Feb 2020). SHAP Documentation Retrieved from: https://shap.readthedocs.io/_/downloads/en/stable/pdf/
- Romano, A. (Feb 2018). How K-pop became a global phenomenon. Retrived from https://www.vox.com/culture/2018/2/16/16915672/what-is-kpop-history-explained
- History of Korean idols. (n.d.) In Wikipedia. Retrieved from https://en.wikipedia.org/wiki/History_of_Korean_idols