Skip to main content

1.11 Replacing Missing Data: Factual Analysis Method

1.11 Replacing Missing Data: Factual Analysis Method

In this post we are going to proceed with correcting our dataset with the factual analysis method i.e replacing the missing data with 100% certainty. Looking at our excel sheet we see that the missing data in column state can be replaced with 100% accuracy from the city.

Similarly, missing data in expense column can also be replaced. First of all let's check all the missing values.

This gives the list of rows which are having NA in the dataset fin. So, let's single out the rows which are having NA in State column.

Now we want to fix this by putting NY in the rows having city as New York and CA(California) in the rows having city as San Fransisco. We can definitely do it one by one based upon the ID or row number, etc. because it has only four rows with missing data. But in case out dataset has many rows with missing values in state column, this approach wont be useful. So we are going to use extra filter, and replace the missing data with the relevant value in one go.
The filter below gives the rows with missing data in state column but with the city name as New York.

Now lets replace the missing data with the state NY.

This would put NY in rows with missing value in State column where city is New York. Now, let's check it with the help of row numbers.

We see that, it has been done. Let's check the complete.cases() once again and it shows two rows less.

 Follow the similar steps for rows with the city as San Fransisco.

 Let's have a look at the complete.cases() once again and it shows only 6 more rows with missing data and we have to deal with only 5 of them because we have decided to let the row with missing data in Inception column to be kept as it is.
The complete code is
Labels:

Comments

Post a Comment

Popular posts from this blog

1.9 Removing Records with Missing Data

1.9 Removing Records with Missing Data In the post number 1.5 Dealing with missing data, we saw various methods and lets implement few of those in this tutorial. First of all let's have a look at the CSV file. We see that, we have decided the option of removing rows where values in Industry column are missing. Before proceeding to the R, I would suggest you to always make a back up of the data so that in case you do any mistake in between you always have the original data to start again. Let's create the backup of our fin dataset. . And this one line can save us a lot of trouble. Now, let's find out all of the rows that have empty value in any of the column. We see two rows where values in Industry column is missing. Let's single out these rows using is.na() So we got two rows with ID 14, 15 where value in Industry column is missing. Now to remove these two rows, we just do the opposite and find out the rows which don't have NA in them and assign it b...
Post a Comment
Read more

1.3 The Factor Variable Trap

1.3 The Factor Variable Trap The Factor Variable Trap or the FVT comes into play when we ate trying to convert a variable from factor to non-factor. It is a known phenomenon, but isn’t very well publicized. Let’s create a vector named a with the values “12“,“13“,“14“,“12”, “12”. (five values all in quotation marks. The values are in character because of double quotation marks, we can verify this with the function typeof() Now let’s convert this vector into type integer with the function an.numeric() So, the above code was to convert characters into numeric. but how to convert factors into numeric?? For this, let’s create a factor Z which contains exactly same values as of vector a.   When we run the above command, output is shown without quotation marks and levels are also displayed. Thus R is recognizing it as categories. Now let’s convert it into numeric, as done before and save it in vector Y to see the output. OOPS!!! What happ...
Post a Comment
Read more
1.8 Data Filters: is.na() for Missing Data In previous post we have learned how to filter data for non missing data. In this one, we will learn how to filter out missing data using is.na(). Let's look at first 24 rows using head() to see the missing values. Just like previous post, if we use the same logic we get NA. Thus, it is not helping at all. The other way to tackle this is is.na(). This function checks if the value contained is NA or not. We try this function, by creating a vector named "a" putting some NAs in it and checking it with is.na(). It gives the value FALSE if its not NA and TRUE if it contains NA. We will use the similar function for our dataset to find out NA in Revenue column. It correctly identifies the values in Revenue column which are equal to NAs. Try to implement it in other columns as well and find out the rows which contains missing values .
Post a Comment
Read more