Skip to main content

1.12 Replacing Missing Data: Median Imputation Method

1.12 Replacing Missing Data: Median Imputation Method

So far we have a lot with missing data and have replaced many of them. If we look at the excel sheet, we see that those cells marked in green are already dealt with.
And in this post we are going to learn about how to proxy the missing value in the Employees and Growth column using median imputation method.
We are using the median values instead of mean because, median is least affected with the outliers and we are taking industry median, because it is very logical to take the median of number of employees belonging to the same industry rather than taking the median of whole column Employees. You can take median based on other variables as well like rows belonging to same state or city, whichever is applicable to your analysis.
So let's start with out with our coding and have a look at the complete.cases().
We see there are two missing rows in here, one in industry Retail and one in Financial Services

The above function will give the result as NA because the column Employee contains NAs or missing values. So, let's take out the NA and then calculate the median.

So, we see that the median of whole employee column is 56. Now, let's take the median of the employees belonging to Retail Industry.
So, we understand that median for Employees belonging to Retail industry is much lesser than that of whole column median. Let's save this value in variable named med_empl_retail

Now, lets replace the missing data with this variable.

Thus we replaced the missing value in Employees column belonging to Retail industry.
Let's repeat the same procedure for Financial Service industry. Looking at complete.cases()

Creating the variable called med_empl_fs to save median of employees of Financial Services industry.

Now replacing the missing data with the value in variable med_empl_fs.

Checking complete.cases() will show us only 4 rows with missing data.

 Now, we will apply the same method as above to Growth column.
Quite easy, right??
Here is the complete code.
Have a look at complete.case(), and it shows only 4 rows with missing data.

 Now we will also learn to put the value in Revenue and Expenses column. We will follow the similar procedure as above and replace the missing value with the median of Revenue for industry Construction.

Let's check the complete.cases().

We see that Revenue column is filled for rows with the median values belonging to the Industry "Construction". Now let's deal with the column Expenses, but here we have to be very cautious as we don't want to replace all missing data in this column because missing value in Expenses column or row number 15 can be calculated from Revenue and Profit. So, we are performing the operation only on row number 8 and 42. In a way we are lucky that the row number 15 does not belong to the construction industry, hence we can use following code

 If this would not have been the case then we would have added one more extra protection code, putting a condition that profit should also be NA, like below

Now we have just one more step is remaining, where we have to derive the values of Profit and Expenses wherever it is missing.

Labels:

Comments

Post a Comment

Popular posts from this blog

1.9 Removing Records with Missing Data

1.9 Removing Records with Missing Data In the post number 1.5 Dealing with missing data, we saw various methods and lets implement few of those in this tutorial. First of all let's have a look at the CSV file. We see that, we have decided the option of removing rows where values in Industry column are missing. Before proceeding to the R, I would suggest you to always make a back up of the data so that in case you do any mistake in between you always have the original data to start again. Let's create the backup of our fin dataset. . And this one line can save us a lot of trouble. Now, let's find out all of the rows that have empty value in any of the column. We see two rows where values in Industry column is missing. Let's single out these rows using is.na() So we got two rows with ID 14, 15 where value in Industry column is missing. Now to remove these two rows, we just do the opposite and find out the rows which don't have NA in them and assign it b...
Post a Comment
Read more

1.3 The Factor Variable Trap

1.3 The Factor Variable Trap The Factor Variable Trap or the FVT comes into play when we ate trying to convert a variable from factor to non-factor. It is a known phenomenon, but isn’t very well publicized. Let’s create a vector named a with the values “12“,“13“,“14“,“12”, “12”. (five values all in quotation marks. The values are in character because of double quotation marks, we can verify this with the function typeof() Now let’s convert this vector into type integer with the function an.numeric() So, the above code was to convert characters into numeric. but how to convert factors into numeric?? For this, let’s create a factor Z which contains exactly same values as of vector a.   When we run the above command, output is shown without quotation marks and levels are also displayed. Thus R is recognizing it as categories. Now let’s convert it into numeric, as done before and save it in vector Y to see the output. OOPS!!! What happ...
Post a Comment
Read more
1.8 Data Filters: is.na() for Missing Data In previous post we have learned how to filter data for non missing data. In this one, we will learn how to filter out missing data using is.na(). Let's look at first 24 rows using head() to see the missing values. Just like previous post, if we use the same logic we get NA. Thus, it is not helping at all. The other way to tackle this is is.na(). This function checks if the value contained is NA or not. We try this function, by creating a vector named "a" putting some NAs in it and checking it with is.na(). It gives the value FALSE if its not NA and TRUE if it contains NA. We will use the similar function for our dataset to find out NA in Revenue column. It correctly identifies the values in Revenue column which are equal to NAs. Try to implement it in other columns as well and find out the rows which contains missing values .
Post a Comment
Read more