Data Preprocessing :: Data Analysis and Statistics (Mathematics)

Mathematics

Data Preprocessing

This section tells you how to work with

Missing Values

The special value, NaN, stands for Not-a-Number in MATLAB. IEEE floating-point arithmetic convention specifies NaN as the result of undefined expressions such as 0/0.

The correct handling of missing data is a difficult problem and often varies in different situations. For data analysis purposes, it is often convenient to use NaNs to represent missing values or data that are not available.

MATLAB treats NaNs in a uniform and rigorous way. They propagate naturally through to the final result in any calculation. Any mathematical calculation involving NaNs produces NaNs in the results.

For example, consider a matrix containing the 3-by-3 magic square with its center element set to NaN.

a = magic(3); a(2,2) = NaN
 
a =
     8     1     6
     3   NaN     7
     4     9     2

Compute a sum for each column in the matrix.

```
sum(a) 
 
ans = 
    15   NaN    15
```

Any mathematical calculation involving NaNs propagates NaNs through to the final result as appropriate.

You should remove NaNs from the data before performing statistical computations. Here are some ways to use isnan to remove NaNs from data.

Code
Description

i = find(~isnan(x));
x = x(i)
Find indices of elements in vector that are not NaNs, then keep only the non-NaN elements.

x = x(find(~isnan(x)))
Remove NaNs from vector.

x = x(~isnan(x));
Remove NaNs from vector (faster).

x(isnan(x)) = [];
Remove NaNs from vector.

X(any(isnan(X)'),:) = [];
Remove any rows of matrix X containing NaNs.

Code	Description
`i = find(~isnan(x));` `x = x(i)`	Find indices of elements in vector that are not `NaN`s, then keep only the non-`NaN` elements.
`x = x(find(~isnan(x)))`	Remove `NaN`s from vector.
`x = x(~isnan(x));`	Remove `NaN`s from vector (faster).
`x(isnan(x)) = [];`	Remove `NaN`s from vector.
`X(any(isnan(X)'),:) = [];`	Remove any rows of matrix `X` containing `NaN`s.

Note You must use the special function isnan to find NaNs because, by IEEE arithmetic convention, the logical comparison, NaN == NaN always produces 0. You cannot use x(x==NaN) = [] to remove NaNs from your data.

If you frequently need to remove NaNs, write a short M-file function.

function X = excise(X)
X(any(isnan(X)'),:) = [];

Now, typing

```
X = excise(X);
```

accomplishes the same thing.

Finite Differences Removing Outliers