|
马上注册,结交更多好友,享用更多功能^_^
您需要 登录 才可以下载或查看,没有账号?立即注册
x
这是我的代码
import logging
from logging.handlers import RotatingFileHandler
from data import build_corpus
from evaluate import bilstm_train_and_eval
from utils import extend_maps, prepocess_data_for_lstmcrf, save_obj, load_obj
# 设置日志级别和格式
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# 添加文件处理器
file_handler = RotatingFileHandler('log.txt', maxBytes=10*1024*1024, backupCount=5, encoding='utf-8')
file_handler.setLevel(logging.INFO)
file_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
logging.getLogger().addHandler(file_handler)
logging.info("读取数据中...")
train_word_lists, train_tag_lists, word2id, tag2id = build_corpus("train")
dev_word_lists, dev_tag_lists = build_corpus("dev", make_vocab=False)
test_word_lists, test_tag_lists = build_corpus("test", make_vocab=False)
logging.info("正在训练评估Bi-LSTM+CRF模型...")
crf_word2id, crf_tag2id = extend_maps(word2id, tag2id, for_crf=True)
save_obj(crf_word2id, 'crf_word2id')
save_obj(crf_tag2id, 'crf_tag2id')
logging.info(f"crf_tag2id: {' '.join([i[0] for i in crf_tag2id.items()])}")
train_word_lists, train_tag_lists = prepocess_data_for_lstmcrf(
train_word_lists, train_tag_lists
)
dev_word_lists, dev_tag_lists = prepocess_data_for_lstmcrf(
dev_word_lists, dev_tag_lists
)
test_word_lists, test_tag_lists = prepocess_data_for_lstmcrf(
test_word_lists, test_tag_lists, test=True
)
lstmcrf_pred = bilstm_train_and_eval(
(train_word_lists, train_tag_lists),
(dev_word_lists, dev_tag_lists),
(test_word_lists, test_tag_lists),
crf_word2id, crf_tag2id
)
logging.info("训练评估完成!")
这个程序跑到这里就不动了
2023-05-11 21:01:11,497 - INFO - 读取数据中...
2023-05-11 21:01:23,668 - INFO - 正在训练评估Bi-LSTM+CRF模型...
2023-05-11 21:01:23,671 - INFO - crf_tag2id: O B-AFF M-AFF E-AFF B-DIS M-DIS E-DIS B-PRI E-PRI S-PRI B-LOG E-LOG S-DIS S-LOG M-LOG S-AFF M-PRI <unk> <pad> <start> <end>
start to train the bilstm_crf ...
训练数据总量:111
Epoch: 0%| | 0/10 [00:00<?, ?it/s]
Iteration: 0%| | 0/7 [00:00<?, ?it/s]
用原本的数据就可以这样跑下去
2023-05-11 20:29:58,670 - INFO - 读取数据中...
2023-05-11 20:29:59,341 - INFO - 正在训练评估Bi-LSTM+CRF模型...
2023-05-11 20:29:59,344 - INFO - crf_tag2id: B-NAME E-NAME O B-CONT M-CONT E-CONT B-RACE E-RACE B-TITLE M-TITLE E-TITLE B-EDU M-EDU E-EDU B-ORG M-ORG E-ORG M-NAME B-PRO M-PRO E-PRO S-RACE S-NAME B-LOC M-LOC E-LOC M-RACE S-ORG B-ID M-ID E-ID <unk> <pad> <start> <end>
start to train the bilstm_crf ...
训练数据总量:3832
Epoch: 0%| | 0/10 [00:00<?, ?it/s]
Iteration: 0%| | 0/240 [00:00<?, ?it/s]
Iteration: 0%| | 1/240 [00:16<1:05:46, 16.51s/it]
Iteration: 1%| | 2/240 [00:28<55:36, 14.02s/it]
Iteration: 1%|▏ | 3/240 [00:39<48:38, 12.31s/it]
Iteration: 2%|▏ | 4/240 [00:48<43:59, 11.19s/it]
Iteration: 2%|▏ | 5/240 [00:57<40:01, 10.22s/it]
Iteration: 2%|▎ | 6/240 [01:04<36:38, 9.40s/it]
Iteration: 3%|▎ | 7/240 [01:12<33:45, 8.70s/it]
Iteration: 3%|▎ | 8/240 [01:18<31:19, 8.10s/it]
Iteration: 4%|▍ | 9/240 [01:25<29:16, 7.60s/it]
Iteration: 4%|▍ | 10/240 [01:31<27:17, 7.12s/it]
Iteration: 5%|▍ | 11/240 [01:37<25:31, 6.69s/it]
这是为什么呢,能通过调整程序让他继续跑下去吗 |
|