在深度学习中读取csv数据时遇到的问题

rplt

在学习LSTM模型时，读取一个csv数据文件，里面包含有日期时间数据，用pandas读取时报错了：ValueError: could not convert string to float: '2017-01-01 00:00:00'
请问是怎么一回事呢？是因为包含空格吗？还是其他字符？请各位高手帮忙解答

rsj0315

贴下读取的代码看看撒

rplt

rsj0315 发表于 2022-2-25 08:11
贴下读取的代码看看撒

df_data_5minute=pd.read_csv('raw_data.csv',error_bad_lines=False,encoding='gbk',lineterminator="\n" ,encoding_errors='ignore')
df_data_5minute.drop('Wind_plant', axis=1, inplace=True)


df=df_data_5minute
#df.drop(labels=['close'], axis=1,inplace = True)
#df.insert(0, 'close', close)
data_train =df.iloc[:30000, :]
data_test = df.iloc[30000:, :]
print(data_train.shape, data_test.shape)


scaler = MinMaxScaler(feature_range=(0, 1))
scaler.fit(data_train)

data_train = scaler.transform(data_train)
data_test = scaler.transform(data_test)


output_dim = 1
batch_size = 1000
epochs = 500
seq_len = 5
hidden_size = 128


TIME_STEPS = 5
INPUT_DIM = 14

lstm_units = 64
X_train = np.array([data_train[i : i + seq_len, :] for i in range(data_train.shape[0] - seq_len)])
y_train = np.array([data_train[i + seq_len, 0] for i in range(data_train.shape[0]- seq_len)])
X_test = np.array([data_test[i : i + seq_len, :] for i in range(data_test.shape[0]- seq_len)])
y_test = np.array([data_test[i + seq_len, 0] for i in range(data_test.shape[0] - seq_len)])
maxy=y_test.max();
miny=y_test.min();
print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)

inputs = Input(shape=(TIME_STEPS, INPUT_DIM))

x = Conv1D(filters = 32, kernel_size = 1, activation = 'relu')(inputs)  #, padding = 'same'
x = MaxPooling1D(pool_size = 5)(x)
x = Dropout(0.1)(x)

lstm_out = Bidirectional(LSTM(lstm_units, activation='relu'), name='bilstm')(x)

output = Dense(1, activation='sigmoid')(lstm_out)

model = Model(inputs=inputs, outputs=output)

model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X_train, y_train, epochs=epochs, batch_size=batch_size, shuffle=False)
model.save('model.h5')
#model=load_model('model.h5')


y_pred = model.predict(X_test)



data_train=scaler.inverse_transform(data_train);
data_test=scaler.inverse_transform(data_test);
y_test = np.array([data_test[i + seq_len, 0] for i in range(data_test.shape[0] - seq_len)])
y_train = np.array([data_train[i + seq_len, 0] for i in range(data_train.shape[0]- seq_len)])
y_raw=np.hstack((y_train,y_test))

#RMSE
print('MSE Train loss:', model.evaluate(X_train, y_train, batch_size=batch_size))
print('MSE Test loss:', model.evaluate(X_test, y_test, batch_size=batch_size))
Rmse = sqrt(mean_squared_error(y_test, y_pred))
print('RMSE: ', Rmse)


plt.plot(np.arange(len(y_raw)), np.hstack((y_train,y_test)) , 'b', label="Raw Data")
plt.plot(np.arange(len(y_train),len(y_raw)),y_pred,'r', label="Prediction")
plt.legend()
plt.show()

rplt

有没有大神帮忙解答一下啊

isdkz

rplt 发表于 2022-2-25 14:54
有没有大神帮忙解答一下啊

把报错信息贴详细点

cflying

想把数据直接读取成日期时间的，记得加参数dtype={'时间所在列名称':np.datetime64}

rplt

isdkz 发表于 2022-2-25 14:56
把报错信息贴详细点

(30000, 20) (5040, 20)
Traceback (most recent call last):
  File "C:\Users\lenovo\Desktop\LSTM\Tensorflow-Wind-Power-Prediction-master\Tensorflow-Wind-Power-Prediction-master\main.py", line 32, in <module>
    scaler.fit(data_train)
  File "E:\anaconda\envs\tensorflow\lib\site-packages\sklearn\preprocessing\_data.py", line 416, in fit
    return self.partial_fit(X, y)
  File "E:\anaconda\envs\tensorflow\lib\site-packages\sklearn\preprocessing\_data.py", line 453, in partial_fit
    X = self._validate_data(
  File "E:\anaconda\envs\tensorflow\lib\site-packages\sklearn\base.py", line 566, in _validate_data
    X = check_array(X, **check_params)
  File "E:\anaconda\envs\tensorflow\lib\site-packages\sklearn\utils\validation.py", line 746, in check_array
    array = np.asarray(array, order=order, dtype=dtype)
  File "E:\anaconda\envs\tensorflow\lib\site-packages\pandas\core\generic.py", line 1993, in __array__
    return np.asarray(self._values, dtype=dtype)
ValueError: could not convert string to float: '2017-01-01 00:00:00'

rplt

rplt 发表于 2022-2-25 14:54
有没有大神帮忙解答一下啊

df_data_5minute=pd.read_csv('raw_data.csv',error_bad_lines=False,encoding='gbk',lineterminator="\n" ,encoding_errors='ignore')   老哥是直接在这个语句后面加吗

cflying

你还没懂，对于“2017-01-01 00:00:00”这种数据，pandas默认读出来的是object，也就是python的str字符串，你要是能转换成float运算才奇了怪了，


df_data_5minute=pd.read_csv('raw_data.csv',dtype={'时间所在列名称':np.datetime64})加上这个则可以直接读取成日期时间，把np.date那个换成float则直接为float（劝你别直接试，“2017-01-01 00:00:00”这玩意儿就算你写了float也要报错，没法直接转换的格式肯定要报错），建议你就弄成日期时间进行处理

如果你的数据直接是1.111，2.34等等的话，你不用加dtpye直接读出来就是float，当然，也可以直接dtype={'时间所在列名称':object}读取为object

不信你可以用df.info()。读出来看看

rplt

cflying 发表于 2022-2-26 17:23
你还没懂，对于“2017-01-01 00:00:00”这种数据，pandas默认读出来的是object，也就是python的s ...

好的谢谢老哥 这种关于数据处理我该怎么学啊老哥是找书看 还是上csdn找例子看