-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
utf-8 fix in unpickle #2653
utf-8 fix in unpickle #2653
Conversation
Co-authored-by: Adam Stephens <2071575+adamcstephens@users.noreply.github.com>
Per @adamcstephens' fix, we should probably be passing |
@obfuscurity Thanks for reporting. We are probably breaking Python 2 support with this change, thus the failed tests. |
hmm, well actually py2 should still work, only the test is incompatible. |
|
||
def test_load(self): | ||
unpickler = util.unpickle() | ||
p = b'\x80\x04\x95\r\x00\x00\x00\x00\x00\x00\x00\x8c\ttest.d\xc3\xb8d\x94.' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a possibly better example that may work as intended for both python2.7 and python3: b"S'test.d\\xc3\\xb8d'\np0\n."
python2.7:
>>> s = 'test.død'
>>> pickle.dumps(s)
"S'test.d\\xc3\\xb8d'\np0\n."
>>> pickle.loads(b"S'test.d\\xc3\\xb8d'\np0\n.")
'test.d\xc3\xb8d'
>>> print(pickle.loads(b"S'test.d\\xc3\\xb8d'\np0\n."))
test.død
python3.7:
>>> pickle.loads(b"S'test.d\\xc3\\xb8d'\np0\n.")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)
>>> pickle.loads(b"S'test.d\\xc3\\xb8d'\np0\n.", encoding='utf8')
'test.død'
I added test in #2660, let's continue there. |
should fix #2652