---
title: Input
updated: 2022-06-22 17:55:33Z
created: 2022-06-22 13:15:11Z
---

### Drop unwanted columns
```
import pandas as pd

df = pd.read_csv('data/energydata_complete.csv', sep=',')
df.drop(columns=['date'], inplace=True)
```

### test isnull values
```
df.isnull().sum()
```

### Outliers
```
outliers = {}
for i in range(data.shape[1]):
    min_t = data[data.columns[i]].mean() \
        - (3 * data[data.columns[i]].std())
    max_t = data[data.columns[i]].mean() \
        + (3 * data[data.columns[i]].std())
    count = 0
    for j in data[data.columns[i]]:
        if j < min_t or j > max_t:
            count += 1
    percentage = count / data.shape[0]
    outliers[data.columns[i]] = "%.3f" % percentage

all_values = outliers.values()
max_value = max(all_values)
max_value
```

### Get numerical data columns
```
cols = df.columns
num_cols = df._get_numeric_data().columns
print(num_cols)
set(cols) - set(num_cols)
```


### Normalize input
```
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

X = df.iloc[:, 1:]
Y = df.iloc[:, 0]
X = pd.DataFrame(scaler.fit_transform(X), columns=X.columns)

```

[source](https://towardsdatascience.com/data-normalization-with-pandas-and-scikit-learn-7c1cc6ed6475)

### Split
```
from sklearn.model_selection import train_test_split

train_ratio = 0.60
validation_ratio = 0.20
test_ratio = 0.20

x_train, x_test, y_train, y_test = train_test_split(X,
                                                    Y,
                                                    test_size=1 - train_ratio)

x_val, x_test, y_val, y_test = train_test_split(
    x_test, y_test, test_size=test_ratio / (test_ratio + validation_ratio))

print(x_train.shape, y_train.shape)
print(x_val.shape, y_val.shape)
print(x_test.shape, y_test.shape)
```
