谁有兴趣，把这个英语断句改吧改吧，看是否能完善一下呢?

blackantt

1. # -*- coding: utf-8 -*-
   
2. import re
   
3. alphabets= "([A-Za-z])"
   
4. prefixes = "(Mr|St|Mrs|Ms|Dr)[.]"
   
5. suffixes = "(Inc|Ltd|Jr|Sr|Co)"
   
6. starters = "(Mr|Mrs|Ms|Dr|Prof|Capt|Cpt|Lt|He\s|She\s|It\s|They\s|Their\s|Our\s|We\s|But\s|However\s|That\s|This\s|Wherever)"
   
7. acronyms = "([A-Z][.][A-Z][.](?:[A-Z][.])?)"
   
8. websites = "[.](com|net|org|io|gov|edu|me)"
   
9. digits = "([0-9])"
   
10. multiple_dots = r'\.{2,}'
    
11. 
    
12. def split_into_sentences(text: str) -> list[str]:
    
13.     """
    
14.     Split the text into sentences.
    
15. 
    
16.     If the text contains substrings "<prd>" or "<stop>", they would lead
    
17.     to incorrect splitting because they are used as markers for splitting.
    
18. 
    
19.     :param text: text to be split into sentences
    
20.     :type text: str
    
21. 
    
22.     :return: list of sentences
    
23.     :rtype: list[str]
    
24.     """
    
25.     text = " " + text + "  "
    
26.     text = text.replace("\n"," ")
    
27.     text = re.sub(prefixes,"\\1<prd>",text)
    
28.     text = re.sub(websites,"<prd>\\1",text)
    
29.     text = re.sub(digits + "[.]" + digits,"\\1<prd>\\2",text)
    
30.     text = re.sub(multiple_dots, lambda match: "<prd>" * len(match.group(0)) + "<stop>", text)
    
31.     if "Ph.D" in text: text = text.replace("Ph.D.","Ph<prd>D<prd>")
    
32.     text = re.sub("\s" + alphabets + "[.] "," \\1<prd> ",text)
    
33.     text = re.sub(acronyms+" "+starters,"\\1<stop> \\2",text)
    
34.     text = re.sub(alphabets + "[.]" + alphabets + "[.]" + alphabets + "[.]","\\1<prd>\\2<prd>\\3<prd>",text)
    
35.     text = re.sub(alphabets + "[.]" + alphabets + "[.]","\\1<prd>\\2<prd>",text)
    
36.     text = re.sub(" "+suffixes+"[.] "+starters," \\1<stop> \\2",text)
    
37.     text = re.sub(" "+suffixes+"[.]"," \\1<prd>",text)
    
38.     text = re.sub(" " + alphabets + "[.]"," \\1<prd>",text)
    
39.     if "”" in text: text = text.replace(".”","”.")
    
40.     if """ in text: text = text.replace("."","".")
    
41.     if "!" in text: text = text.replace("!"",""!")
    
42.     if "?" in text: text = text.replace("?"",""?")
    
43.     text = text.replace(".",".<stop>")
    
44.     text = text.replace("?","?<stop>")
    
45.     text = text.replace("!","!<stop>")
    
46.     text = text.replace("<prd>",".")
    
47. 
    
48. 
    
49. 
    
50. 
    
51.     # text = re.sub(r'([,.?!][\'"&#187;’]*)(\w)', r'\1 \2', text)
    
52.     text = re.sub(r'([,][\'"&#187;’]*)(\w)', r'\1 \2', text)
    
53. 
    
54.     # --- 保留并优化原函数中一些有用的规则 ---
    
55. 
    
56.     # 规则2: 去除标点符号（.?!,;:)之前的多余空格
    
57.     # 例如："Hello ." -> "Hello."
    
58.     text = re.sub(r'\s+([.?!,;:])', r'\1', text)
    
59.     text = re.sub(r' +', ' ', text)
    
60. 
    
61. 
    
62. 
    
63. 
    
64.     sentences = text.split("<stop>")
    
65.     sentences = [s.strip() for s in sentences]
    
66.     if sentences and not sentences[-1]: sentences = sentences[:-1]
    
67.     return sentences
    
68. 
    
69. 
    
70. text = """
    
71. I love you.I am Tom.      Hello,world!How are you ?
    
72.     Dr. Smith and Mr. Jones met at 3:30 p.m. The meeting was held at example.com.
    
73.     "I love this project!" exclaimed Prof. Brown. "It's amazing."
    
74.     The U.S. Department of Energy (DOE) reported a 2.5% increase in 3.15 at Beijing.
    
75.     I love you. Tom said.
    
76.     I love you.Tom said.
    
77.     “苏超”联赛火爆出圈，场均观众8798人远超中甲，门票一票难求！全民参与+城市荣誉模式激活经济内循环，地域热梗成看点，文旅消费激增，政府主导打造“移动的城市广告”。
    
78.     网上热度也是相当狂飙。  虎扑App紧急新增“江苏联”频道，上线首日访问量破百万；抖音话题#江苏城市联赛#播放量破亿，素人拍摄的赛事短视频占比超70%……第三轮的门票已经是花钱都抢不到了，甚至有球迷在二手物品交易平台表示，愿意花100元求购一张门票。
    
79.     But then Wei poured a shower of gold coins into her lap."Never mind where I got them", he whispered. "Let's just say... I made a brilliant business deal!"Mei said nothing — she was too busy polishing the gold.Now news travels fast.Their neighbor,Jin ,soon, heard,    that Wei had returned from a big business trip and was now rich. His wife heard too?"Brilliant deal, eh?" she said to him. "If that fool Wei can make all that money,why can't you?"
    
80. 
    
81. 
    
82. 
    
83. """
    
84. 
    
85. print(split_into_sentences(text))

复制代码

某一个“天”

必须手搓吗？能用自然语言处理库吗

某一个“天”

手搓，我最近没时间

blackantt

某一个“天” 发表于 2025-6-12 13:05
必须手搓吗？能用自然语言处理库吗

NLTK ， spaCy ， blingfire 这些估计都够呛。  不但要考虑正常句子。还要考虑 I am Tom.I love you. It's 3.15.  这种缺少空格的情况。

某一个“天”

那你搞个这种特殊的字符串，我测试下

smallwh

像这种特殊情况，基于规则的处理也很麻烦。如果句子不规范，很容易引起歧义。
比如同为 3:30 p.m.
在Dr. Smith and Mr. Jones met at 3:30 p.m. The meeting was held at example.com.是两个句子，
而在At 3:30 p.m. Tom attended a meeting which was held at example.com.是一个句子。
我的建议是规范输入，或者采用大模型(Chatgpt3.0在这方面表现出色)【基于统计和概率才是文本处理的最终解】。


下面是修改了一些错误，并经Chatgpt3.0优化的代码。但是还会存在一些问题。

1. import re
   
2. 
   
3. # 定义相关的正则表达式模式
   
4. alphabets = "([A-Za-z])"
   
5. prefixes = "(Mr|St|Mrs|Ms|Dr|Prof)[.]"
   
6. suffixes = "(Inc|Ltd|Jr|Sr|Co)"
   
7. starters = "(Mr|Mrs|Ms|Dr|Prof|Capt|Cpt|Lt|He\s|She\s|It\s|They\s|Their\s|Our\s|We\s|But\s|However\s|That\s|This\s|Wherever)"
   
8. acronyms = "([A-Z][.][A-Z][.](?:[A-Z][.])?)"
   
9. websites = "[.](com|net|org|io|gov|edu|me)"
   
10. digits = "([0-9])"
    
11. multiple_dots = r'\.{2,}'
    
12. 
    
13. def split_into_sentences(text):
    
14.     """
    
15.     Split the text into sentences.
    
16. 
    
17.     :param text: text to be split into sentences
    
18.     :type text: str
    
19. 
    
20.     :return: list of sentences
    
21.     :rtype: list[str]
    
22.     """
    
23.     # 初始化文本
    
24.     text = " " + text + "  "
    
25.     text = text.replace("\n", " ")
    
26.    
    
27.     # 处理缩写
    
28.     text = re.sub(prefixes, "\\1<prd>", text)
    
29.     text = re.sub(websites, "<prd>\\1", text)
    
30.     text = re.sub(digits + "[.]" + digits, "\\1<prd>\\2", text)
    
31.     text = re.sub(multiple_dots, lambda match: "<prd>" * len(match.group(0)) + "<stop>", text)
    
32. 
    
33.     # 处理特定缩写（如Ph.D.）
    
34.     if "Ph.D" in text:
    
35.         text = text.replace("Ph.D.", "Ph<prd>D<prd>")
    
36. 
    
37.     # 处理字母后的点
    
38.     text = re.sub("\s" + alphabets + "[.] ", " \\1<prd> ", text)
    
39. 
    
40.     # 处理缩写后的句子拆分
    
41.     text = re.sub(acronyms + " " + starters, "\\1<stop> \\2", text)
    
42.     text = re.sub(alphabets + "[.]" + alphabets + "[.]" + alphabets + "[.]", "\\1<prd>\\2<prd>\\3<prd>", text)
    
43.     text = re.sub(alphabets + "[.]" + alphabets + "[.]", "\\1<prd>\\2<prd>", text)
    
44. 
    
45.     # 处理后缀
    
46.     text = re.sub(" " + suffixes + "[.] " + starters, " \\1<stop> \\2", text)
    
47.     text = re.sub(" " + suffixes + "[.]", " \\1<prd>", text)
    
48.     text = re.sub(" " + alphabets + "[.]", " \\1<prd>", text)
    
49. 
    
50.     # 处理引号和标点符号的情况
    
51.     if "”" in text: text = text.replace(".”", "”.")
    
52.     if '"' in text: text = text.replace('."', '".')
    
53.     if "!" in text: text = text.replace('!"', '"!')
    
54.     if "?" in text: text = text.replace('?"', '"?')
    
55. 
    
56.     # 替换句号，问号和感叹号后加 <stop>
    
57.     text = text.replace(".", ".<stop>")
    
58.     text = text.replace("?", "?<stop>")
    
59.     text = text.replace("!", "!<stop>")
    
60. 
    
61.     # 还原已替换的 <prd> 为句点
    
62.     text = text.replace("<prd>", ".")
    
63. 
    
64.     # 处理其他标点符号的间隔问题
    
65.     text = re.sub(r'([,][\'"&#187;’]*)(\w)', r'\1 \2', text)
    
66.     text = re.sub(r'\s+([.?!,;:])', r'\1', text)
    
67.     text = re.sub(r' +', ' ', text)
    
68. 
    
69.     # 处理中文与英文间的断句问题
    
70.     text = re.sub(r'(\w)([，。！？])', r'\1 <stop> \2', text)
    
71.     text = re.sub(r'([。！？])(\w)', r'\1 <stop> \2', text)
    
72. 
    
73.     # 按照<stop>分割句子
    
74.     sentences = text.split("<stop>")
    
75.     sentences = [s.strip() for s in sentences]
    
76.    
    
77.     # 清除空句子
    
78.     if sentences and not sentences[-1]:
    
79.         sentences = sentences[:-1]
    
80.    
    
81.     return sentences
    
82. 
    
83. 
    
84. 
    
85. text = """
    
86. I love you.I am Tom.      Hello,world!How are you ?
    
87.     Dr. Smith and Mr. Jones met at 3:30 p.m. The meeting was held at example.com.
    
88.     "I love this project!" exclaimed Prof. Brown. "It's amazing."
    
89.     The U.S. Department of Energy (DOE) reported a 2.5% increase in 3.15 at Beijing.
    
90.     I love you. Tom said.
    
91.     I love you.Tom said.
    
92.     “苏超”联赛火爆出圈，场均观众8798人远超中甲，门票一票难求！全民参与+城市荣誉模式激活经济内循环，地域热梗成看点，文旅消费激增，政府主导打造“移动的城市广告”。
    
93.     网上热度也是相当狂飙。  虎扑App紧急新增“江苏联”频道，上线首日访问量破百万；抖音话题#江苏城市联赛#播放量破亿，素人拍摄的赛事短视频占比超70%……第三轮的门票已经是花钱都抢不到了，甚至有球迷在二手物品交易平台表示，愿意花100元求购一张门票。
    
94.     But then Wei poured a shower of gold coins into her lap."Never mind where I got them", he whispered. "Let's just say... I made a brilliant business deal!"Mei said nothing — she was too busy polishing the gold.Now news travels fast.Their neighbor,Jin ,soon, heard,    that Wei had returned from a big business trip and was now rich. His wife heard too?"Brilliant deal, eh?" she said to him. "If that fool Wei can make all that money,why can't you?"
    
95. 
    
96. 
    
97. 
    
98. """
    
99. 
    
100. for s in split_into_sentences(text):
     
101.     print(s)
     

复制代码

输出：

1. I love you.
   
2. I am Tom.
   
3. Hello, world!
   
4. How are you?
   
5. Dr. Smith and Mr. Jones met at 3:30 p.m. The meeting was held at example.com.
   
6. "I love this project"!
   
7. exclaimed Prof. Brown.
   
8. "It's amazing".
   
9. The U.S. Department of Energy (DOE) reported a 2.5% increase in 3.15 at Beijing.
   
10. I love you.
    
11. Tom said.
    
12. I love you.
    
13. Tom said.
    
14. “苏超”联赛火爆出圈
    
15. ，场均观众8798人远超中甲
    
16. ，门票一票难求
    
17. ！
    
18. 全民参与+城市荣誉模式激活经济内循环
    
19. ，地域热梗成看点
    
20. ，文旅消费激增
    
21. ，政府主导打造“移动的城市广告”。 网上热度也是相当狂飙
    
22. 。 虎扑App紧急新增“江苏联”频道
    
23. ，上线首日访问量破百万；抖音话题#江苏城市联赛#播放量破亿
    
24. ，素人拍摄的赛事短视频占比超70%……第三轮的门票已经是花钱都抢不到了
    
25. ，甚至有球迷在二手物品交易平台表示
    
26. ，愿意花100元求购一张门票
    
27. 。 But then Wei poured a shower of gold coins into her lap".
    
28. Never mind where I got them", he whispered.
    
29. "Let's just say...
    
30. I made a brilliant business deal"!
    
31. Mei said nothing — she was too busy polishing the gold.
    
32. Now news travels fast.
    
33. Their neighbor, Jin, soon, heard, that Wei had returned from a big business trip and was now rich.
    
34. His wife heard too"?
    
35. Brilliant deal, eh"?
    
36. she said to him.
    
37. "If that fool Wei can make all that money, why can't you"?

复制代码