wayfinderak/udacity_eda
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
udacity_eda/EDA_Project/EDA_Project.rmd
Dusty.P a9bec51e55 Final Project Submitted
2018-07-19 22:54:51 -08:00

492 lines
22 KiB
Plaintext
Raw Blame History

---
title: "EDA_Project"
author: "Dusty P"
date: "July 19, 2018"
output: html_document
---
```{r echo=FALSE, message=FALSE, warning=FALSE, setup}
knitr::opts_knit$set(root.dir = normalizePath("C:/Users/Dusty/Documents/coding/projects/Udacity/Data Analysis/eda/EDA_Project"))
# load the ggplot graphics package and the others
library(ggplot2)
library(GGally)
library(scales)
library(memisc)
library(gridExtra)
library(RColorBrewer)
library(bitops)
library(RCurl)
cuberoot_trans = function() trans_new('cuberoot', transform = function(x) x^(1/3),
inverse = function(x) x^3)
```
# Exploration of White Wines by Dustin Pianalto
This report explores a dataset containing chemical information and ratings on almost 4900 white wine tastings.
```{r echo=FALSE, message=FALSE, warning=FALSE, Load_the_Data}
# Load the Data
wqw <- read.csv('wineQualityWhites.csv')
# because the first column is just row numbers I am going to remove it
wqw <- subset(wqw, select = -X)
```
# Univariate Plots Section
```{r echo=FALSE, warning=FALSE, Data_Dimensions}
dim(wqw)
```
```{r echo=FALSE, warning=FALSE, Data_Structure}
str(wqw)
```
```{r echo=FALSE, warning=FALSE, Data_Summary}
summary(wqw)
```
Our Data consists of 11 numerical variables and one Integer attribute which is the output with almost 4900 observations
```{r echo=FALSE, warning=FALSE, quality_histogram}
ggplot(aes(x = quality), data = wqw) +
geom_histogram(binwidth = 1)
```
The distribution of the quality seems fairly normal with a peak at 6
```{r echo=FALSE, warning=FALSE, alcohol_histogram}
ggplot(aes(x = alcohol), data = wqw) +
geom_histogram(binwidth = .1)
```
The Alcohol seems to be slightly long tailed, I want to see what it is like with a log transformation.
```{r echo=FALSE, warning=FALSE, alcohol_histogram_log}
ggplot(aes(x = alcohol), data = wqw) +
geom_histogram(binwidth = .005) +
scale_x_log10()
```
```{r echo=FALSE, warning=FALSE, fixed.acidity_histogram}
ggplot(aes(x = fixed.acidity), data = wqw) +
geom_histogram(binwidth = .1)
```
The fixed.acidity definately has some outliers but besides that has a pretty normal distribution.
```{r echo=FALSE, warning=FALSE, fixed.acidity_summary}
summary(wqw$fixed.acidity)
```
Most Wines have a acidity between 6.3 and 7.3
I am going to plot the data again removing both the high and low 1% of values to remove the outliers.
```{r echo=FALSE, warning=FALSE, fixed.acidity_histogram_quantile}
ggplot(aes(x = fixed.acidity), data = wqw) +
geom_histogram(binwidth = .1) +
xlim(quantile(wqw$fixed.acidity, 0.01), quantile(wqw$fixed.acidity, 0.99))
```
And we see a fairly normal distribution with a peak around 6.8 which matches both the median (6.8) and mean (6.855) from the summary above.
```{r echo=FALSE, warning=FALSE, volatile.acidity_histogram}
ggplot(aes(x = volatile.acidity), data = wqw) +
geom_histogram(binwidth = .01)
```
We have another long tailed distribution. I am going to plot again with a log_10 transformation this time.
```{r echo=FALSE, warning=FALSE, volatile.acidity_histogram_log}
ggplot(aes(x = volatile.acidity), data = wqw) +
geom_histogram(binwidth = .04) +
scale_x_log10()
```
```{r echo=FALSE, warning=FALSE, citric.acid_histogram}
ggplot(aes(x = citric.acid), data = wqw) +
geom_histogram(binwidth = .01) +
xlim(quantile(wqw$citric.acid, 0.01), quantile(wqw$citric.acid, 0.99))
```
There is an odd spike at about 0.49 I might want to look into that more later.
```{r echo=FALSE, warning=FALSE, residual.sugar_histogram}
ggplot(aes(x = residual.sugar), data = wqw) +
geom_histogram(binwidth = .1) +
xlim(quantile(wqw$residual.sugar, 0.01), quantile(wqw$residual.sugar, 0.99))
```
Even with the top and bottom 1% removed the plot is still very long tailed
```{r echo=FALSE, warning=FALSE, residual.sugar_histogram_log}
p1 <- ggplot(aes(x = residual.sugar), data = wqw) +
geom_histogram(binwidth = .05) +
scale_x_log10()
p2 <- ggplot(aes(x = residual.sugar), data = wqw) +
geom_histogram(binwidth = .01) +
scale_x_log10(breaks = seq(0, 20, 2))
grid.arrange(p1, p2)
```
```{r echo=FALSE, warning=FALSE, residual.sugar_summary}
summary(wqw$residual.sugar)
```
Using a log_10 transform with a bin width of .05 indicates a bimodal distribution but if you decrease the binwidth to 0.01 it shows that while there are a lot of observations between ~4 and ~20 they are a lot more spread out and there are more individual of each value from ~0.5 to ~2. This is probably is because it is harder to measure the residual sugar as a continuous scale and so the steps are more apparent at the lower, more spread out, values. And we can see in the summary of the data that the median is 5.2 and the mean is 6.4 which puts both of them inbetween the two peaks.
```{r echo=FALSE, warning=FALSE, chlorides_histogram}
ggplot(aes(x = chlorides), data = wqw) +
geom_histogram(binwidth = .001) +
xlim(0, quantile(wqw$chlorides, 0.97))
```
Here I just removed the top 3% of values to remove the long tail.
```{r echo=FALSE, warning=FALSE, sulfur.dioxide_histograms}
p1 <- ggplot(aes(x = free.sulfur.dioxide), data = wqw) +
geom_histogram(binwidth = 1) +
xlim(0, quantile(wqw$free.sulfur.dioxide, 0.99))
p2 <- ggplot(aes(x = total.sulfur.dioxide), data = wqw) +
geom_histogram(binwidth = 1) +
xlim(quantile(wqw$total.sulfur.dioxide, 0.01), quantile(wqw$total.sulfur.dioxide, 0.99))
grid.arrange(p1, p2)
```
I plotted the Free Sulphur Dioxide and Total Sulphur Dioxide together to save room and because they are related. Note the difference in scales on both axies.
```{r echo=FALSE, warning=FALSE, density_histogram}
ggplot(aes(x = density), data = wqw) +
geom_histogram(binwidth = .0001) +
xlim(quantile(wqw$density, 0.01), quantile(wqw$density, 0.99))
```
```{r echo=FALSE, warning=FALSE, pH_histogram}
ggplot(aes(x = pH), data = wqw) +
geom_histogram(binwidth = .01)
```
The pH plot doesn't need any modification.
```{r echo=FALSE, warning=FALSE, sulphates_histogram}
ggplot(aes(x = sulphates), data = wqw) +
geom_histogram(binwidth = .01)
```
# Univariate Analysis
### What is the structure of your dataset?
There are 4898 samples in the dataset with 11 different variables and a resulting quality assesment. All of the variables are continuous number variables and the quality is an integer scale from 1 to 10 with max value of 9 and min of 3.
Observations:
* The most common quality is 6 and it is a fairly normal distribution slightly skewed towards the low end.
* Most of the variables are similar in distribution, most of them are long tailed but besides that have a fairly normal distribution.
* There are a couple interesting features though, the Citric Acid has an odd spike around 4.9 and the Residual Sugar appears to be more of a bimodal distribution.
### What is/are the main feature(s) of interest in your dataset?
My main interest in this dataset is trying to determine which features have the greatest effect on the quality.
### What other features in the dataset do you think will help support your investigation into your feature(s) of interest?
I think that the Alcohol, Acidity, Density, and Ph will have the greatest impact on the quality.
### Did you create any new variables from existing variables in the dataset?
I did not create any new variables.
### Of the features you investigated, were there any unusual distributions? \
Did you perform any operations on the data to tidy, adjust, or change the form \
of the data? If so, why did you do this?
I either log transformed or removed the outliers on most of the datapoints to better view the data as most of them were longtailed.
# Bivariate Plots Section
```{r echo=FALSE, warning=FALSE, Bivariate_Plots}
ggpairs(wqw, upper = list(continuous = wrap("cor", size = 1.8)), lower = list(continuous = wrap("smooth", alpha=0.2, color = "orange"))) +
theme_grey(base_size = 6)
```
Looking at this grid there are not very many variables that appear to correlate with each other which I find suprising as some of them seem like they should. I am going to explore some of them in more detail.
One pair of variables that look like they have some correlation are density and residual sugar so I am going to start with them.
```{r echo=FALSE, warning=FALSE, residual.sugar_vs_density}
ggplot(aes(x = residual.sugar, y = density), data = wqw) +
geom_point(color = "orange")
```
Narrowing in on the main section and adding a smoothing line.
```{r echo=FALSE, warning=FALSE, residual.sugar_vs_density_mod}
ggplot(aes(x = residual.sugar, y = density), data = wqw) +
geom_point(alpha=0.3, color = "orange") +
xlim(0, 30) +
ylim(0.987, 1.0025) +
geom_smooth(method = "gam")
```
We can see a general trend as residual sugar increases the density also increases. Lets see both of these plotted against our output variable.
```{r echo=FALSE, warning=FALSE, quality_vs_density}
ggplot(aes(x = quality, y = density), data = wqw) +
ylim(0.985, 1.005) +
geom_point(alpha=0.1, position = position_jitter(h = 0), color = "blue") +
geom_line(stat = 'summary', fun.y = mean, color = "blue") +
geom_line(stat = 'summary', fun.y = median) +
geom_line(stat = 'summary', fun.y = quantile, fun.args = list(probs = 0.1), color = 'red', linetype = 2) +
geom_line(stat = 'summary', fun.y = quantile, fun.args = list(probs = 0.9), color = 'red', linetype = 2)
```
```{r echo=FALSE, warning=FALSE, quality_vs_residual.sugar}
ggplot(aes(y = residual.sugar, x = quality), data = wqw) +
geom_point(alpha=0.1, color = "blue") +
xlim(0, 10) +
ylim(0, 30)
```
There doesn't seem to be any direct corelation between these variables and the quality. Lets look at some others.
```{r echo=FALSE, warning=FALSE, quality_vs_alcohol}
ggplot(aes(x = quality, y = alcohol), data = wqw) +
geom_point(alpha=0.1, color = "blue")
```
```{r echo=FALSE, warning=FALSE, quality_vs_alcohol_w_summaries}
ggplot(aes(x = quality, y = alcohol), data = wqw) +
geom_point(alpha=0.1, position = position_jitter(h = 0), color = "blue") +
geom_line(stat = 'summary', fun.y = mean, color = "blue") +
geom_line(stat = 'summary', fun.y = median) +
geom_line(stat = 'summary', fun.y = quantile, fun.args = list(probs = 0.1), color = 'red', linetype = 2) +
geom_line(stat = 'summary', fun.y = quantile, fun.args = list(probs = 0.9), color = 'red', linetype = 2)
```
Adding jitter to the alcohol plot reveals that there could possibly be a corelation to quality but it is very weak.
```{r echo=FALSE, warning=FALSE, quality_vs_fixed.acidity}
ggplot(aes(x = quality, y = fixed.acidity), data = wqw) +
geom_point(alpha=0.1, position = position_jitter(h = 0), color = "blue")
```
```{r echo=FALSE, warning=FALSE, quality_vs_chlorides}
ggplot(aes(x = quality, y = chlorides), data = wqw) +
geom_point(alpha=0.1, position = position_jitter(h = 0), color = "blue") +
ylim(0, 0.1) +
geom_line(stat = 'summary', fun.y = mean, color = "blue") +
geom_line(stat = 'summary', fun.y = median) +
geom_line(stat = 'summary', fun.y = quantile, fun.args = list(probs = 0.1), color = 'red', linetype = 2) +
geom_line(stat = 'summary', fun.y = quantile, fun.args = list(probs = 0.9), color = 'red', linetype = 2)
```
```{r echo=FALSE, warning=FALSE, quality_vs_tsd}
ggplot(aes(x = quality, y = total.sulfur.dioxide), data = wqw) +
geom_point(alpha=0.1, position = position_jitter(h = 0), color = "blue")
```
Looking at these other variables shows that there is little to no relationship to the quality individually I think this will change when we start combining variables in the Multivariate plots.
One other interesting corelation that I want to look at is density vs alcohol.
```{r echo=FALSE, warning=FALSE, alcohol_vs_density}
ggplot(aes(x = alcohol, y = density), data = wqw) +
geom_point(alpha=0.1, color = "blue") +
ylim(0.985, 1.005) +
geom_smooth(method = "gam")
```
Interestingly it appears that as the aocohol content increases the density decreases, this is the inverse of the residual sugar vs density that we plotted earlier. This probably has something to do with the fact that sugar is what the alcohol is created from so it would follow that as the alcohol increases the sugar and thence the density would decrease.
We can see this more directly by plotting residual sugar against alcohol.
```{r echo=FALSE, warning=FALSE, alcohol_vs_residual_sugar}
ggplot(aes(x = alcohol, y = residual.sugar), data = wqw) +
geom_point(alpha=0.1, color = "blue") +
ylim(0, 30) +
geom_smooth(method = "gam")
```
We can see that there is a seemingly exponential relationship between alcohol and residual sugar.
```{r echo=FALSE, warning=FALSE, alcohol_vs_chlorides}
ggplot(aes(x = alcohol, y = chlorides), data = wqw) +
geom_point(alpha=0.1, color = "blue") +
ylim(0, 0.1) +
geom_smooth(method = "gam")
```
There does seem to be a slight corelation between alcohol and chlorides.
```{r echo=FALSE, warning=FALSE, alcohol_vs_ph}
ggplot(aes(x = pH, y = alcohol), data = wqw) +
geom_point(alpha=0.1, color = "blue") +
geom_smooth(method = "gam")
```
```{r echo=FALSE, warning=FALSE, alcohol_vs_fixed_acidity}
ggplot(aes(x = fixed.acidity, y = alcohol), data = wqw) +
geom_point(alpha=0.1, color = "blue")
```
```{r echo=FALSE, warning=FALSE, alcohol_vs_volatile_acidity}
ggplot(aes(x = volatile.acidity, y = alcohol), data = wqw) +
geom_point(alpha=0.1, color = "blue")
```
There does not seem to be any correlation between our other features of interest.
# Bivariate Analysis
### Talk about some of the relationships you observed in this part of the investigation. How did the feature(s) of interest vary with other features in the dataset?
I discovered some interesting relationships between density, residual sugar and alcohol. The other features appear to have very little corelation to each other or to the quality. The other relationships that I noted are the ones that were expected. For instance the pH has a mild corelation to the fixed acidity although I expected a higher corelation. Same with total sulfur dioxide and free sulfur dioxide.
It does seem like there is a mild corelation between the quality and alcohol as well as quality and density which are 2 of the features I noted in the previous section. There also might be a slight relationship between quality and chlorides.
### Did you observe any interesting relationships between the other features (not the main feature(s) of interest)?
One relationship that I found interesting is between alcohol and chlorides as well as between chlorides and quality. I wonder if this will show itself more in the multivariate exploration.
### What was the strongest relationship you found?
By far the strongest relationship I found was between density and residual sugar.
# Multivariate Plots Section
```{r echo=FALSE, warning=FALSE, alcohol_chlorides_quality}
ggplot(aes(x = alcohol, y = chlorides), data = wqw) +
geom_point(aes(color = quality))
```
```{r echo=FALSE, warning=FALSE, alcohol_residual.sugar_quality}
ggplot(aes(x = alcohol, y = residual.sugar), data = wqw) +
geom_point(aes(color = quality)) +
ylim(0, 30)
```
```{r echo=FALSE, warning=FALSE, density_pH_quality}
ggplot(aes(x = density, y = pH), data = wqw) +
geom_point(aes(color = quality)) +
xlim(0.985, 1.005)
```
```{r echo=FALSE, warning=FALSE, free.sulfur.dioxide_pH_quality}
ggplot(aes(x = free.sulfur.dioxide, y = pH), data = wqw) +
geom_point(aes(color = quality)) +
xlim(0, 100)
```
```{r echo=FALSE, warning=FALSE, alcohol_pH_quality}
ggplot(aes(x = alcohol, y = pH), data = wqw) +
geom_point(aes(color = quality))
```
```{r echo=FALSE, warning=FALSE, alcohol_density_quality}
ggplot(aes(x = alcohol, y = density), data = wqw) +
geom_point(aes(color = quality), position = position_jitter(h = 0)) +
ylim(0.985, 1.005)
```
This is really the only plot that I have tried that seems to indicate any sort of corelation between any of the variables and the quality and it is very weak. The quality is only slightly squewed towards higher alcohol and lower density (which we discovered an inverse corelation between alcohol and density earlier so that should make sense).
Lets see if a linear model can make any predictions.
```{r echo=FALSE, warning=FALSE, Building_the_Linear_Model}
m1 <- lm(I(quality) ~ I(alcohol), data = wqw)
m2 <- update(m1, ~ . + density)
m3 <- update(m2, ~ . + residual.sugar)
m4 <- update(m3, ~ . + chlorides)
m5 <- update(m4, ~ . + sulphates)
m6 <- update(m5, ~ . + pH)
m7 <- update(m6, ~ . + fixed.acidity)
m8 <- update(m7, ~ . + volatile.acidity)
m9 <- update(m8, ~ . + citric.acid)
m10 <- update(m9, ~ . + free.sulfur.dioxide)
m11 <- update(m10, ~ . + total.sulfur.dioxide)
mtable(m1, m2, m5, m6, m9, m11, sdigits = 3)
```
As we can see even when taking into account every feature the R-squared is still only 0.282 which is dismal at best and indicates that we can not make any predictions based on the data that we have.
(I had to remove some of the intermediary steps to make it fit on the page.)
# Multivariate Analysis
### Talk about some of the relationships you observed in this part of the \
investigation. Were there features that strengthened each other in terms of \
looking at your feature(s) of interest?
All of the features that I investigated in this section show a dramatic lack of corelation. Even when combining features in different ways there was little to no interaction.
There were a few things that I discovered earlier that were confirmed but there wasn't really anything new to explore.
### Were there any interesting or surprising interactions between features?
The only interesting thing was the complete lack of interesting interactions between features.
### OPTIONAL: Did you create any models with your dataset? Discuss the \
strengths and limitations of your model.
I did create a basic model and it was not able to predict anything. The main limitation of the model is that none of the features are corelated to the quality in any meaningful way.
------
# Final Plots and Summary
### Plot One
```{r echo=FALSE, Plot_One}
ggpairs(wqw, upper = list(continuous = wrap("cor", size = 1.8)), lower = list(continuous = wrap("smooth", alpha=0.2, color = "orange"))) +
theme_grey(base_size = 6) +
ggtitle("Scatterplot Matrix") +
theme(plot.title = element_text(size=22, hjust = 0.5))
```
### Description One
This is a good summary of the data that we have and it shows how there is no direct corelation between any of the variables and the quality. You can see some moderate corelation between some of the features such as residual sugar and density. Some of these corelations are something I focused on.
### Plot Two
```{r echo=FALSE, warning=FALSE, Plot_Two}
ggplot(aes(x = quality, y = alcohol), data = wqw) +
geom_point(alpha=0.1, position = position_jitter(h = 0), color = "blue") +
geom_line(stat = 'summary', fun.y = mean, color = "blue") +
geom_line(stat = 'summary', fun.y = median, color = "black") +
geom_line(stat = 'summary', fun.y = quantile, fun.args = list(probs = 0.1), color = 'red', linetype = 2) +
geom_line(stat = 'summary', fun.y = quantile, fun.args = list(probs = 0.9), color = 'red', linetype = 2) +
ggtitle("Alcohol vs Quality") +
xlab("Quality") +
ylab("Alcohol") +
theme(plot.title = element_text(size=22, hjust = 0.5))
```
### Description Two
Alcohol content is the closest that any of the features came to corelating with the quality and here you can see that even that corelation is very weak. The only thing we can tell is that more higher quality wines had a higher alcohol content than lower quality, but the spread on the data makes this a very weak corelation at 0.436.
### Plot Three
```{r echo=FALSE, warning=FALSE, Plot_Three}
ggplot(aes(x = density, y = alcohol), data = wqw) +
geom_point(aes(color = quality)) +
xlim(0.985, 1.005) +
labs(x = "Density", y = "Alcohol", title = "Alcohol vs Density by Quality", color = "Quality") +
theme(plot.title = element_text(size=22, hjust = 0.5))
```
### Description Three
I include this plot just to show how there is no clear distinction in the quality when compared to the features of the data. This is representative of all of the plots I made in the multivariate section.
------
# Reflection
The Wine dataset that I used contained information from almost 5,000 wine tastings with their quality rating included. Initially I examined the data to see the shape of each of the features and then started exploring how they interact with each other. Then I compared the features against the quality to see if any of the features could help to predict the quality of the product. Finally I created a linear model to see if there was anything I missed in the data that could create predictions.
In the beginning I thought that the quality would have something to do with the alcohol, density, pH, and acidity. As I examined the data it became more and more clear that there was little to no correlation between any of the features and the quality. I found this suprising and really wanted to find any little thing that would point towards a corelation but nothing showed up. Finally when I created the linear model it was clear that you could not predict the quality of the wine from the data that we have in this dataset. We do see some small corelation between the alcohol content and the quality, it appears that the higher the alcohol content the more likely the wine will have a higher quality but there is definately not enough destinction to make any predictions.
I don't know if more datapoints could make a difference but it seems at this point that the quality of wine is subjective and is difficult if not impossible to predict. I might be able to improve the models with more manipulation of the data but other models that I have seen max out at ~70% accuracy such as PennState's STAT 897D Analysis of Wine Quality Data (https://onlinecourses.science.psu.edu/stat857/node/223/), and R-bloggers Predicting wine quality using Random Forests (https://www.r-bloggers.com/predicting-wine-quality-using-random-forests/) which use a lot more complex modeling than a basic linear model. R-bloggers use a random forest and acheived a 71.5% accuracy which is still very low for making predictions. They also had to modify the quality variable into groups where 3, 4, and 5 were considered low quality, 6 and 7 were medium, and 8 and 9 were high quality. I personally find this unacceptable as it degrades the quality of the output and artifically pushes the prediction rate higher. On a scale of 1-10 the difference between 8 and 9 can be quite substantial.
Reference in New Issue View Git Blame Copy Permalink