bert模型训练时报错

Terence888

1. ---------------------------------------------------------------------------
   
2. AttributeError                            Traceback (most recent call last)
   
3. Cell In[50], line 4
   
4.       1 print('Training Start!')
   
5.       2 print('=' * 100)
   
6. ----> 4 train(model,
   
7.       5     device,
   
8.       6     train_dataloader,
   
9.       7     valid_dataloader,
   
10.       8     CFG.epochs,
    
11.       9     loss_fn,
    
12.      10     optimizer,
    
13.      11     metric)
    
14.      13 del model,train_dataloader, valid_dataloader
    
15.      14 gc.collect()
    
16. 
    
17. Cell In[49], line 17, in train(model, device, train_dataloader, valid_dataloader, epochs, loss_fn, optimizer, metric)
    
18.      14 train_step = 0
    
19.      15 pbar = tqdm(train_dataloader)#tqdm参数是一个iterable
    
20. ---> 17 for batch in pbar: # you can also write like "for batch in tqdm(train_dataloader"
    
21.      18     optimizer.zero_grad() # initialize
    
22.      19     train_step += 1
    
23. 
    
24. File /opt/conda/lib/python3.10/site-packages/tqdm/notebook.py:250, in tqdm_notebook.__iter__(self)
    
25.     248 try:
    
26.     249     it = super().__iter__()
    
27. --> 250     for obj in it:
    
28.     251         # return super(tqdm...) will not catch exception
    
29.     252         yield obj
    
30.     253 # NB: except ... [ as ...] breaks IPython async KeyboardInterrupt
    
31. 
    
32. File /opt/conda/lib/python3.10/site-packages/tqdm/std.py:1181, in tqdm.__iter__(self)
    
33.    1178 time = self._time
    
34.    1180 try:
    
35. -> 1181     for obj in iterable:
    
36.    1182         yield obj
    
37.    1183         # Update and possibly print the progressbar.
    
38.    1184         # Note: does not call self.update(1) for speed optimisation.
    
39. 
    
40. File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:630, in _BaseDataLoaderIter.__next__(self)
    
41.     627 if self._sampler_iter is None:
    
42.     628     # TODO(https://github.com/pytorch/pytorch/issues/76750)
    
43.     629     self._reset()  # type: ignore[call-arg]
    
44. --> 630 data = self._next_data()
    
45.     631 self._num_yielded += 1
    
46.     632 if self._dataset_kind == _DatasetKind.Iterable and \
    
47.     633         self._IterableDataset_len_called is not None and \
    
48.     634         self._num_yielded > self._IterableDataset_len_called:
    
49. 
    
50. File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:1344, in _MultiProcessingDataLoaderIter._next_data(self)
    
51.    1342 else:
    
52.    1343     del self._task_info[idx]
    
53. -> 1344     return self._process_data(data)
    
54. 
    
55. File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:1370, in _MultiProcessingDataLoaderIter._process_data(self, data)
    
56.    1368 self._try_put_index()
    
57.    1369 if isinstance(data, ExceptionWrapper):
    
58. -> 1370     data.reraise()
    
59.    1371 return data
    
60. 
    
61. File /opt/conda/lib/python3.10/site-packages/torch/_utils.py:706, in ExceptionWrapper.reraise(self)
    
62.     702 except TypeError:
    
63.     703     # If the exception takes multiple arguments, don't try to
    
64.     704     # instantiate since we don't know how to
    
65.     705     raise RuntimeError(msg) from None
    
66. --> 706 raise exception
    
67. 
    
68. AttributeError: Caught AttributeError in DataLoader worker process 0.
    
69. Original Traceback (most recent call last):
    
70.   File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 309, in _worker_loop
    
71.     data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
    
72.   File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    
73.     data = [self.dataset[idx] for idx in possibly_batched_index]
    
74.   File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
    
75.     data = [self.dataset[idx] for idx in possibly_batched_index]
    
76.   File "/tmp/ipykernel_31/3318300818.py", line 15, in __getitem__
    
77.     text = self.df.loc[idx]['text'] # extracting text from each row
    
78. AttributeError: 'list' object has no attribute 'loc'

复制代码

报错如上，代码如下

1. print('Training Start!')
   
2. print('=' * 100)
   
3. 
   
4. train(model,
   
5.     device,
   
6.     train_dataloader,
   
7.     valid_dataloader,
   
8.     CFG.epochs,
   
9.     loss_fn,
   
10.     optimizer,
    
11.     metric)
    
12. 
    
13. del model,train_dataloader, valid_dataloader
    
14. gc.collect()

复制代码

FishC

根据错误信息，问题出在 `__getitem__` 方法中。具体来说，`self.df` 被当作一个 `list` 对象，而不是一个支持 `.loc` 方法的对象（如 `pandas.DataFrame`）。

要解决这个问题，请检查数据集类中 `__getitem__` 方法的实现，确保 `self.df` 是一个 `pandas.DataFrame` 对象。以下是一个可能的修正示例：

1. class YourDatasetClass:
   
2.     def __init__(self, dataframe):
   
3.         self.df = dataframe
   
4. 
   
5.     def __getitem__(self, idx):
   
6.         # 确保 self.df 是一个 DataFrame
   
7.         if isinstance(self.df, pd.DataFrame):
   
8.             text = self.df.loc[idx, 'text_column_name']  # 替换 'text_column_name' 为实际的列名
   
9.             # 其他处理逻辑
   
10.             return text
    
11.         else:
    
12.             raise TypeError("self.df 应该是一个 pandas DataFrame")

复制代码

请确保在初始化数据集类时传入的是一个 `pandas.DataFrame` 对象，而不是一个 `list`。如果 `self.df` 是从其他地方传入的，检查传入的对象类型是否正确。

以上回复来自 FishC AI 小助理，如未能正确解答您的问题，请继续追问。

Terence888

FishC 发表于 2024-11-2 23:25
根据错误信息，问题出在 `__getitem__` 方法中。具体来说，`self.df` 被当作一个 `list` 对象，而不是一个 ...

1. Training Start!
   
2. ====================================================================================================
   
3. /tmp/ipykernel_31/152112840.py:32: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
   
4.   'token_type_ids':torch.tensor(encoded_dict['token_type_ids'],dtype=torch.long).unsqueeze(dim=0)}
   
5. /tmp/ipykernel_31/152112840.py:32: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
   
6.   'token_type_ids':torch.tensor(encoded_dict['token_type_ids'],dtype=torch.long).unsqueeze(dim=0)}
   
7. /tmp/ipykernel_31/152112840.py:32: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
   
8.   'token_type_ids':torch.tensor(encoded_dict['token_type_ids'],dtype=torch.long).unsqueeze(dim=0)}
   
9. &#8199;&#8199;0%
   
10. &#8199;0/229&#8199;[00:00<?,&#8199;?it/s]
    
11. /tmp/ipykernel_31/152112840.py:32: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
    
12.   'token_type_ids':torch.tensor(encoded_dict['token_type_ids'],dtype=torch.long).unsqueeze(dim=0)}
    
13. /tmp/ipykernel_31/152112840.py:32: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
    
14.   'token_type_ids':torch.tensor(encoded_dict['token_type_ids'],dtype=torch.long).unsqueeze(dim=0)}
    
15. /tmp/ipykernel_31/152112840.py:32: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
    
16.   'token_type_ids':torch.tensor(encoded_dict['token_type_ids'],dtype=torch.long).unsqueeze(dim=0)}
    
17. /tmp/ipykernel_31/152112840.py:32: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
    
18.   'token_type_ids':torch.tensor(encoded_dict['token_type_ids'],dtype=torch.long).unsqueeze(dim=0)}
    
19. /tmp/ipykernel_31/152112840.py:32: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
    
20.   'token_type_ids':torch.tensor(encoded_dict['token_type_ids'],dtype=torch.long).unsqueeze(dim=0)}
    
21. ---------------------------------------------------------------------------
    
22. KeyError                                  Traceback (most recent call last)
    
23. Cell In[58], line 4
    
24.       1 print('Training Start!')
    
25.       2 print('=' * 100)
    
26. ----> 4 train(model,
    
27.       5     device,
    
28.       6     train_dataloader,
    
29.       7     valid_dataloader,
    
30.       8     CFG.epochs,
    
31.       9     loss_fn,
    
32.      10     optimizer,
    
33.      11     metric)
    
34.      13 del model,train_dataloader, valid_dataloader
    
35.      14 gc.collect()
    
36. 
    
37. Cell In[39], line 17, in train(model, device, train_dataloader, valid_dataloader, epochs, loss_fn, optimizer, metric)
    
38.      14 train_step = 0
    
39.      15 pbar = tqdm(train_dataloader)#tqdm参数是一个iterable
    
40. ---> 17 for batch in pbar: # you can also write like "for batch in tqdm(train_dataloader"
    
41.      18     optimizer.zero_grad() # initialize
    
42.      19     train_step += 1
    
43. 
    
44. File /opt/conda/lib/python3.10/site-packages/tqdm/notebook.py:250, in tqdm_notebook.__iter__(self)
    
45.     248 try:
    
46.     249     it = super().__iter__()
    
47. --> 250     for obj in it:
    
48.     251         # return super(tqdm...) will not catch exception
    
49.     252         yield obj
    
50.     253 # NB: except ... [ as ...] breaks IPython async KeyboardInterrupt
    
51. 
    
52. File /opt/conda/lib/python3.10/site-packages/tqdm/std.py:1181, in tqdm.__iter__(self)
    
53.    1178 time = self._time
    
54.    1180 try:
    
55. -> 1181     for obj in iterable:
    
56.    1182         yield obj
    
57.    1183         # Update and possibly print the progressbar.
    
58.    1184         # Note: does not call self.update(1) for speed optimisation.
    
59. 
    
60. File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:630, in _BaseDataLoaderIter.__next__(self)
    
61.     627 if self._sampler_iter is None:
    
62.     628     # TODO(https://github.com/pytorch/pytorch/issues/76750)
    
63.     629     self._reset()  # type: ignore[call-arg]
    
64. --> 630 data = self._next_data()
    
65.     631 self._num_yielded += 1
    
66.     632 if self._dataset_kind == _DatasetKind.Iterable and \
    
67.     633         self._IterableDataset_len_called is not None and \
    
68.     634         self._num_yielded > self._IterableDataset_len_called:
    
69. 
    
70. File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:1344, in _MultiProcessingDataLoaderIter._next_data(self)
    
71.    1342 else:
    
72.    1343     del self._task_info[idx]
    
73. -> 1344     return self._process_data(data)
    
74. 
    
75. File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:1370, in _MultiProcessingDataLoaderIter._process_data(self, data)
    
76.    1368 self._try_put_index()
    
77.    1369 if isinstance(data, ExceptionWrapper):
    
78. -> 1370     data.reraise()
    
79.    1371 return data
    
80. 
    
81. File /opt/conda/lib/python3.10/site-packages/torch/_utils.py:706, in ExceptionWrapper.reraise(self)
    
82.     702 except TypeError:
    
83.     703     # If the exception takes multiple arguments, don't try to
    
84.     704     # instantiate since we don't know how to
    
85.     705     raise RuntimeError(msg) from None
    
86. --> 706 raise exception
    
87. 
    
88. KeyError: Caught KeyError in DataLoader worker process 0.
    
89. Original Traceback (most recent call last):
    
90.   File "/opt/conda/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    
91.     return self._engine.get_loc(casted_key)
    
92.   File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
    
93.   File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
    
94.   File "pandas/_libs/hashtable_class_helper.pxi", line 2606, in pandas._libs.hashtable.Int64HashTable.get_item
    
95.   File "pandas/_libs/hashtable_class_helper.pxi", line 2630, in pandas._libs.hashtable.Int64HashTable.get_item
    
96. KeyError: 1287
    
97. 
    
98. The above exception was the direct cause of the following exception:
    
99. 
    
100. Traceback (most recent call last):
     
101.   File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 309, in _worker_loop
     
102.     data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
     
103.   File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
     
104.     data = [self.dataset[idx] for idx in possibly_batched_index]
     
105.   File "/opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
     
106.     data = [self.dataset[idx] for idx in possibly_batched_index]
     
107.   File "/tmp/ipykernel_31/152112840.py", line 15, in __getitem__
     
108.     text = self.df.loc[idx]['cleaned'] # extracting text from each row
     
109.   File "/opt/conda/lib/python3.10/site-packages/pandas/core/indexing.py", line 1191, in __getitem__
     
110.     return self._getitem_axis(maybe_callable, axis=axis)
     
111.   File "/opt/conda/lib/python3.10/site-packages/pandas/core/indexing.py", line 1431, in _getitem_axis
     
112.     return self._get_label(key, axis=axis)
     
113.   File "/opt/conda/lib/python3.10/site-packages/pandas/core/indexing.py", line 1381, in _get_label
     
114.     return self.obj.xs(label, axis=axis)
     
115.   File "/opt/conda/lib/python3.10/site-packages/pandas/core/generic.py", line 4301, in xs
     
116.     loc = index.get_loc(key)
     
117.   File "/opt/conda/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
     
118.     raise KeyError(key) from err
     
119. KeyError: 1287

复制代码

报错如上

FishC

Terence888 发表于 2024-11-03 10:33
Training Start!
================================= ...

根据错误信息，`KeyError: 1287` 表示在尝试访问数据集的某个索引时，索引超出了数据集的范围。这个问题通常是由于数据集的大小与索引不匹配导致的。

以下是一些可能的解决方案：

1. **检查数据集大小**：确保数据集的大小与您期望的一致。可以通过打印数据集的长度来确认：

  

1. print(len(self.df))

复制代码

2. **检查索引范围**：在 `__getitem__` 方法中，确保索引 `idx` 在数据集的范围内。可以在方法中添加一个检查：

  

1. def __getitem__(self, idx):
   
2.        if idx >= len(self.df):
   
3.            raise IndexError("Index out of range")
   
4.        # 继续处理

复制代码

3. **数据集分割**：如果您在分割数据集时出现问题，可能导致某个分割为空或大小不一致。检查数据集分割的代码，确保每个分割都有数据。

4. **数据加载器**：检查 `DataLoader` 的 `batch_size` 和 `shuffle` 参数，确保它们与数据集的大小兼容。

5. **调试信息**：在 `__getitem__` 方法中添加调试信息，帮助定位问题：

  

1. def __getitem__(self, idx):
   
2.        print(f"Accessing index: {idx}")
   
3.        # 继续处理

复制代码

通过这些步骤，您应该能够更好地定位和解决问题。如果问题仍然存在，请检查数据集的其他部分，确保数据的完整性和一致性。

以上回复来自 FishC AI 小助理，如未能正确解答您的问题，请继续追问。

Terence888

FishC 发表于 2024-11-3 10:33
根据错误信息，`KeyError: 1287` 表示在尝试访问数据集的某个索引时，索引超出了数据集的范围。这个问 ...

1. from torch.utils.data import Dataset
   
2. import torch
   
3. 
   
4. #定义数据集
   
5. class LLMDataset(Dataset):
   
6.     def __init__(self,df,is_grad,tokenizer):
   
7.         self.df = df # Pandas.DataFrame
   
8.         self.is_grad = is_grad # True: train,valid / False: test
   
9.         self.tokenizer = tokenizer
   
10. 
    
11.     def __len__(self):
    
12.         return len(self.df) # number of samples
    
13. 
    
14.     def __getitem__(self,idx):
    
15.         text = self.df.loc[idx]['cleaned'] # extracting text from each row
    
16. 
    
17.         encoded_dict = self.tokenizer.encode_plus(
    
18.             text,
    
19.             add_special_tokens=True,#自动在每个文本前后添加特殊标记(如CLS和SEP)
    
20.             padding='max_length',#补0
    
21.             truncation=True,#句子长度大于max_length时截断
    
22.             max_length=84, # given to the max_length of tokenized text
    
23.             return_tensors='pt', # PyTorch
    
24.             return_attention_mask=True, # We should put it into the model，计算注意力（attention）时忽略那些paddle值
    
25.         )
    
26. 
    
27.         if self.is_grad:#训练集
    
28.             
    
29.             # [batch,1,max_len(84)] -> [batch,max_len]#使用squeeze降维
    
30.             return {'input_ids':encoded_dict['input_ids'].squeeze(),
    
31.                     'attention_mask':encoded_dict['attention_mask'].squeeze(),
    
32.                     'token_type_ids':torch.tensor(encoded_dict['token_type_ids'],dtype=torch.long).unsqueeze(dim=0)}
    
33.         else:#测试集
    
34.             # [batch,1,max_len(84)] -> [batch,max_len]
    
35.             return {'input_ids':encoded_dict['input_ids'].squeeze(),
    
36.                     'attention_mask':encoded_dict['attention_mask'].squeeze()}

复制代码

上述处理数据集的代码有什么问题

FishC

Terence888 发表于 2024-11-03 10:38
from torch.utils.data import Dataset
import torch ...

在您提供的代码中，有几个问题需要注意：

1. **`self.df.loc` 的使用**：
   - 在 `__getitem__` 方法中，`text = self.df.loc` 这一行是错误的。`self.df.loc` 是一个方法，您需要使用 `self.df.loc[idx, 'column_name']` 来获取特定行和列的数据。确保替换 `'column_name'` 为实际的列名。

2. **`encoded_dict` 的使用**：
   - `encoded_dict` 是一个字典，包含多个键（如 `input_ids` 和 `attention_mask`）。在返回时，您需要分别处理这些键，而不是直接对 `encoded_dict` 使用 `squeeze`。

3. **`torch.tensor` 的使用**：
   - 在 `torch.tensor(encoded_dict, dtype=torch.long)` 中，`encoded_dict` 是一个字典，不能直接转换为张量。您需要对字典中的具体值进行转换。

以下是修正后的代码示例：

1. from torch.utils.data import Dataset
   
2. import torch
   
3. 
   
4. class LLMDataset(Dataset):
   
5.     def __init__(self, df, is_grad, tokenizer):
   
6.         self.df = df
   
7.         self.is_grad = is_grad
   
8.         self.tokenizer = tokenizer
   
9. 
   
10.     def __len__(self):
    
11.         return len(self.df)
    
12. 
    
13.     def __getitem__(self, idx):
    
14.         # 假设 'text_column' 是包含文本数据的列名
    
15.         text = self.df.loc[idx, 'text_column']
    
16. 
    
17.         encoded_dict = self.tokenizer.encode_plus(
    
18.             text,
    
19.             add_special_tokens=True,
    
20.             padding='max_length',
    
21.             truncation=True,
    
22.             max_length=84,
    
23.             return_tensors='pt',
    
24.             return_attention_mask=True,
    
25.         )
    
26. 
    
27.         if self.is_grad:
    
28.             return {
    
29.                 'input_ids': encoded_dict['input_ids'].squeeze(),
    
30.                 'attention_mask': encoded_dict['attention_mask'].squeeze(),
    
31.                 'token_type_ids': torch.tensor(encoded_dict['token_type_ids'], dtype=torch.long).squeeze()
    
32.             }
    
33.         else:
    
34.             return {
    
35.                 'input_ids': encoded_dict['input_ids'].squeeze(),
    
36.                 'attention_mask': encoded_dict['attention_mask'].squeeze()
    
37.             }

复制代码

请确保在 `__getitem__` 方法中正确访问 DataFrame 的列，并对 `encoded_dict` 中的具体键进行处理。

以上回复来自 FishC AI 小助理，如未能正确解答您的问题，请继续追问。

Terence888

FishC 发表于 2024-11-3 10:54
在您提供的代码中，有几个问题需要注意：

1. **`self.df.loc` 的使用**：

1. 'input_ids': encoded_dict['input_ids'].squeeze(),
   
2.                 'attention_mask': encoded_dict['attention_mask'].squeeze(),
   
3.                 'token_type_ids': torch.tensor(encoded_dict['token_type_ids'], dtype=torch.long).squeeze()

复制代码

input_ids attention_mask token_type_ids分别是什么形状的列表

FishC

Terence888 发表于 2024-11-03 11:54
'input_ids': encoded_dict.squeeze(),
             ...

In the context of the BERT model and the `transformers` library, the shapes of `input_ids`, `attention_mask`, and `token_type_ids` are typically as follows:

1. **`input_ids`**: This is a tensor containing the token IDs for the input text. Its shape is usually `(batch_size, max_length)`, where `max_length` is the maximum sequence length specified during tokenization.

2. **`attention_mask`**: This tensor indicates which tokens should be attended to and which should not (e.g., padding tokens). Its shape is also `(batch_size, max_length)`.

3. **`token_type_ids`**: This tensor is used to distinguish between different sequences in tasks like sentence pair classification. Its shape is `(batch_size, max_length)`.

In your code, it seems like there might be a misunderstanding in how `encoded_dict` is being used. The `encoded_dict` returned by `tokenizer.encode_plus` is a dictionary containing these keys, and you should access them individually. Here's a corrected version of how you might handle these tensors:

1. {
   
2.     'input_ids': encoded_dict['input_ids'].squeeze(),
   
3.     'attention_mask': encoded_dict['attention_mask'].squeeze(),
   
4.     'token_type_ids': encoded_dict['token_type_ids'].squeeze()
   
5. }

复制代码

This assumes that `encoded_dict` contains these keys, which is typical when using `encode_plus` with `return_tensors='pt'

以上回复来自 FishC AI 小助理，如未能正确解答您的问题，请继续追问。