Skip to main content

1.6 What is NA

1.6 What is NA

For dealing with missing data, it is very important to understand what is NA. In R missing values are represented by the symbol NA i.e it is a special variable. So, let's read a little bit about it by typing question mark(?) in front of it.

We see that it stands for Not Available or Missing Values. It is a logical constant with length 1 and contains a missing value indicator.
Let's understand about it more detail. We have two know logical constants TRUE which stands for 1 and FALSE which stands for 0. So what does NA contain?? Let's check it using following code

These logical constants behave as expected. Now, let's try using NA

The expected output you think would be FALSE, but it is not the case with NA. The output is NA only

When we try other options with NA, the following output is given

We see that, missing value is compared with, TRUE or FALSE or some other values, Hence, it gives NA as output.

Now, lets locate this missing data in out dataset. For this we look at top 24 rows in our dataset.


We see that at some places there NAs and few empty values as well, We also see some NA in brackets in column Inception. Now we want to pull out these rows from dataset which have only NA or missing values. In R, there is a very good way to do it with the function complete.cases(). If you run it, it will give the values as TRUE and FALSE depending on the tows being complete or not.
Now let us subset this dataset  and extract the rows which are incomplete or has missing data. For this we will put a negation(i.e ! mark) ahead of complete.cases(), so that it will return TRUE values of incomplete rows.
This shows that there are only 6 rows which are having missing data but in actual dataset there are many more rows which are incomplete. So, howcome it is only picking up 6 rows?
The answer to this it that, some of the missing data is not indicated by NA but it just does not have any value, which is not the same as NA that's why R thinks that it is a complete row. Like line number 14 and 15, which has no value in the column Industry, hence R recognizes it as complete row. The best way to tackle this is to fix the problem at its core.i.e at the stage of importing the data in R.
Hence, try the following code..

na.strings=c("") will replace all the rows which has value mentioned within quotation mark will be replaced by NA.
Let's run head(fin, 20) command and we will see that the empty records now contain NA in it.
Now run the complete.cases() function and check for the incomplete rows.

 Now we see that it is picking many more rows than previously. You must have observed that there are some rows which are having brackets around NA and some are not, why so??
Answer is pretty simple, i.e. all the variables which are recognized as factors are having brackets around NAs and integers don't have the brackets. Thus, R helps us to recognize the categories based on NAs as well.
In next post we will learn, how to tackle these missing data. Until then Happy Coding!!
Labels:

Comments

Post a Comment

Popular posts from this blog

1.9 Removing Records with Missing Data

1.9 Removing Records with Missing Data In the post number 1.5 Dealing with missing data, we saw various methods and lets implement few of those in this tutorial. First of all let's have a look at the CSV file. We see that, we have decided the option of removing rows where values in Industry column are missing. Before proceeding to the R, I would suggest you to always make a back up of the data so that in case you do any mistake in between you always have the original data to start again. Let's create the backup of our fin dataset. . And this one line can save us a lot of trouble. Now, let's find out all of the rows that have empty value in any of the column. We see two rows where values in Industry column is missing. Let's single out these rows using is.na() So we got two rows with ID 14, 15 where value in Industry column is missing. Now to remove these two rows, we just do the opposite and find out the rows which don't have NA in them and assign it b...
Post a Comment
Read more

1.3 The Factor Variable Trap

1.3 The Factor Variable Trap The Factor Variable Trap or the FVT comes into play when we ate trying to convert a variable from factor to non-factor. It is a known phenomenon, but isn’t very well publicized. Let’s create a vector named a with the values “12“,“13“,“14“,“12”, “12”. (five values all in quotation marks. The values are in character because of double quotation marks, we can verify this with the function typeof() Now let’s convert this vector into type integer with the function an.numeric() So, the above code was to convert characters into numeric. but how to convert factors into numeric?? For this, let’s create a factor Z which contains exactly same values as of vector a.   When we run the above command, output is shown without quotation marks and levels are also displayed. Thus R is recognizing it as categories. Now let’s convert it into numeric, as done before and save it in vector Y to see the output. OOPS!!! What happ...
Post a Comment
Read more
1.8 Data Filters: is.na() for Missing Data In previous post we have learned how to filter data for non missing data. In this one, we will learn how to filter out missing data using is.na(). Let's look at first 24 rows using head() to see the missing values. Just like previous post, if we use the same logic we get NA. Thus, it is not helping at all. The other way to tackle this is is.na(). This function checks if the value contained is NA or not. We try this function, by creating a vector named "a" putting some NAs in it and checking it with is.na(). It gives the value FALSE if its not NA and TRUE if it contains NA. We will use the similar function for our dataset to find out NA in Revenue column. It correctly identifies the values in Revenue column which are equal to NAs. Try to implement it in other columns as well and find out the rows which contains missing values .
Post a Comment
Read more