[python-win32] extension modules and msvc version

[Zachary Turner]

Does anyone understand the technical reasons why an extension module must
be compiled with the same version of msvc as python itself? Are there any
workarounds? It's really quite an inconvenience.

If the reason is because python27.dll and the extension module free each
others' memory, then it seems like this could be solved by having each
supply the other with an alloc and free function pointer, and using the
correct allocator on each side.

Is the situation better in python 3 than it is in 2.7? And is anyone aware
of any ways to get around this restriction so that i can write an extension
module that will work with a binary release of python?

[Chris Angelico]

Post by Zachary Turner
Is the situation better in python 3 than it is in 2.7? And is anyone aware
of any ways to get around this restriction so that i can write an extension
module that will work with a binary release of python?

It's going to be better, starting from Python 3.5; Microsoft is
guaranteeing a measure of binary compatibility for future compiler
versions. Unfortunately the guarantee requires some internal changes,
so it's moving forward only, not backward. The fate of Python 2.7 is
very much up in the air, because the compiler that was used for 2.7.0
won't be supported in 2020 (I think it might be already out of
support), so there's a big question of whether or not to change
compilers in the middle of a minor version. Neither option is good.

ChrisA

[Zachary Turner]

Is it completely out of the question to change the way extension modules
and python talk to each other? For example, suppose python is changed in a
minor version such that it will be able to detect an extension module
compiled with some flag, say PYTHON_EXTENSION_ABI_COMPATIBLE. Python sees
that you compiled your extension module this way, and does something
different internally that allows them to not free each others' memory. Like
registering allocators with each other at startup. Unless both python and
the extension module are built against the same minor version or higher,
both sides continue to behave as they currently do.

I don't know anything about Python implementation internals, but is
something like this possible?

Post by Zachary Turner

Post by Zachary Turner
Is the situation better in python 3 than it is in 2.7? And is anyone

aware

Post by Zachary Turner
of any ways to get around this restriction so that i can write an

extension

Post by Zachary Turner
module that will work with a binary release of python?

It's going to be better, starting from Python 3.5; Microsoft is
guaranteeing a measure of binary compatibility for future compiler
versions. Unfortunately the guarantee requires some internal changes,
so it's moving forward only, not backward. The fate of Python 2.7 is
very much up in the air, because the compiler that was used for 2.7.0
won't be supported in 2020 (I think it might be already out of
support), so there's a big question of whether or not to change
compilers in the middle of a minor version. Neither option is good.
ChrisA
_______________________________________________
python-win32 mailing list
https://mail.python.org/mailman/listinfo/python-win32

[Zachary Turner]

In other words, I'm postulating that the strict abi compatibility
requirement might be something we can fix in Python, and not rely on
Microsoft to support this in CRT

Post by Zachary Turner
Is it completely out of the question to change the way extension modules
and python talk to each other? For example, suppose python is changed in a
minor version such that it will be able to detect an extension module
compiled with some flag, say PYTHON_EXTENSION_ABI_COMPATIBLE. Python sees
that you compiled your extension module this way, and does something
different internally that allows them to not free each others' memory. Like
registering allocators with each other at startup. Unless both python and
the extension module are built against the same minor version or higher,
both sides continue to behave as they currently do.
I don't know anything about Python implementation internals, but is
something like this possible?

Post by Zachary Turner

Post by Zachary Turner
Is the situation better in python 3 than it is in 2.7? And is anyone

aware

Post by Zachary Turner
of any ways to get around this restriction so that i can write an

extension

Post by Zachary Turner
module that will work with a binary release of python?

It's going to be better, starting from Python 3.5; Microsoft is
guaranteeing a measure of binary compatibility for future compiler
versions. Unfortunately the guarantee requires some internal changes,
so it's moving forward only, not backward. The fate of Python 2.7 is
very much up in the air, because the compiler that was used for 2.7.0
won't be supported in 2020 (I think it might be already out of
support), so there's a big question of whether or not to change
compilers in the middle of a minor version. Neither option is good.
ChrisA
_______________________________________________
python-win32 mailing list
https://mail.python.org/mailman/listinfo/python-win32

[Chris Angelico]

Is it completely out of the question to change the way extension modules and
python talk to each other?
I don't know anything about Python implementation internals, but is
something like this possible?

I've no idea, my previous post pretty much exhausted my entire
knowledge of the building of Python extensions for Windows :) But that
sounds like a great topic for python-ideas or python-dev... and a PEP,
and a lengthy period of bikeshedding, and lots of people telling you
it isn't possible, and lots of other people telling you it's easy...
Have fun with it!

ChrisA

[Thomas Heller]

Post by Zachary Turner
Does anyone understand the technical reasons why an extension module
must be compiled with the same version of msvc as python itself? Are
there any workarounds? It's really quite an inconvenience.
If the reason is because python27.dll and the extension module free each
others' memory, then it seems like this could be solved by having each
supply the other with an alloc and free function pointer, and using the
correct allocator on each side.
Is the situation better in python 3 than it is in 2.7? And is anyone
aware of any ways to get around this restriction so that i can write an
extension module that will work with a binary release of python?

pep 384 solves this problem.

[Tim Roberts]

Does anyone understand the technical reasons why an extension module must be compiled with the same version of msvc as python itself? Are there any workarounds? It's really quite an inconvenience.
If the reason is because python27.dll and the extension module free each others' memory, then it seems like this could be solved by having each supply the other with an alloc and free function pointer, and using the correct allocator on each side.

It’s more than just the heaps, although that’s a part of it. The run-time library contains state held in static variables. When you have multiple versions of the run-time library, they don’t share that state. The C variable “stdin”, for example, is a pointer into a static array of file instances. When you have multiple run-time libraries, there are multiple buffers for “stdin”. Prints from one user will not be seamlessly integrated with prints from the other.
—
Tim Roberts, ***@probo.com
Providenza & Boekelheide, Inc.