[python-win32] Embedding python in a Win32 application

[Zachary Turner]

Hi,

I'm trying to embed python 2.7 in a Win32 application and I'm running into
a host of problems.

Originally, my problem was that when compiling a debug version of my
application and linking against the python27.lib that I installed via the
packaged installer, I would just get strange behavior and memory corruption
and random crashes. I finally determined that this was basically just
impossible, and if I want to use embedded python in a debug version of my
application, I need to build my own debug version of python.

This leads me to my first question: Was my original determination
accurate? I think all of my problems would be solved automatically if I
could just embed release python in a debug version of my application.


Fast forward a bit. I now have a custom debug build of python. My debug
build of application links against it, and release build links against
release python. So far so good, until I try to run "import ctypes". Then
I get this:

Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.

import ctypes

Traceback (most recent call last):
File "<console>", line 1, in <module>
File "d:\python_src\Python-2.7.8\Lib\ctypes\__init__.py", line 10, in
<module>
from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes

Anybody have any idea what is going on here or how I might diagnose this?
It's obviously located the ctypes module that came with the source
distribution, so what error could it be running into?

For reference, the way I've built this is to download Python 2.7 source
distribution, open up the solution in Visual Studio, and build it, then
have my application link against the binaries that are put into the build's
output directories. Is there a different way?

Thanks

[Zachary Ware]

Hi,
I'm trying to embed python 2.7 in a Win32 application and I'm running into a
host of problems.
Originally, my problem was that when compiling a debug version of my
application and linking against the python27.lib that I installed via the
packaged installer, I would just get strange behavior and memory corruption
and random crashes. I finally determined that this was basically just
impossible, and if I want to use embedded python in a debug version of my
application, I need to build my own debug version of python.
This leads me to my first question: Was my original determination accurate?
I think all of my problems would be solved automatically if I could just
embed release python in a debug version of my application.

I believe you're correct. It is a somewhat unfortunate limitation;
it's on my radar to see if it can be relaxed, but I haven't had time
to look into it very closely yet.

Fast forward a bit. I now have a custom debug build of python. My debug
build of application links against it, and release build links against
release python. So far so good, until I try to run "import ctypes". Then I
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.

Post by Zachary Turner
import ctypes

File "<console>", line 1, in <module>
File "d:\python_src\Python-2.7.8\Lib\ctypes\__init__.py", line 10, in
<module>
from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes
Anybody have any idea what is going on here or how I might diagnose this?
It's obviously located the ctypes module that came with the source
distribution, so what error could it be running into?

What do you get if you do 'import sys; print sys.path'? Is the
directory that holds _ctypes.pyd included in sys.path?

For reference, the way I've built this is to download Python 2.7 source
distribution, open up the solution in Visual Studio, and build it, then have
my application link against the binaries that are put into the build's
output directories. Is there a different way?

That ought to be about right.

[Zachary Turner]

Apologies if this message appears incorrectly threaded. Initially I didn't
subscribe to the list, so I'm replying to my own post instead of Zachary
Ware's. Will do my best to keep the quoting intact though:

On Fri, Oct 3, 2014 at 1:02 PM, Zachary Ware

Post by Zachary Ware

Post by Zachary Turner
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.

Post by Zachary Turner
import ctypes

File "<console>", line 1, in <module>
File "d:\python_src\Python-2.7.8\Lib\ctypes\__init__.py", line 10, in
<module>
from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes
Anybody have any idea what is going on here or how I might diagnose this?
It's obviously located the ctypes module that came with the source
distribution, so what error could it be running into?

What do you get if you do 'import sys; print sys.path'? Is the
directory that holds _ctypes.pyd included in sys.path?

Post by Zachary Turner
For reference, the way I've built this is to download Python 2.7 source
distribution, open up the solution in Visual Studio, and build it, then have
my application link against the binaries that are put into the build's
output directories. Is there a different way?

That ought to be about right.

I did some more digging, and I think it's because ctypes is built as an
extension module, so _ctypes_d.pyd is in my build output directory
alongside python_d.exe and python27_d.dll, and not in my libs directory. I
managed to get it working if I add my build output directory to my
PYTHONPATH. Is there some way to package up my build and then install it
so that the directory structure is the same as if I had downloaded a
pre-packaged installer and installed it? I don't think PYTHONPATH is set
by a default install, so something about the way it installs itself (maybe
python.exe is hardcoded to look in the DLLs subfolder?) allows an installed
build of python to work without PYTHONPATH. I'd like to achieve the same
thing with my custom build.

Post by Zachary Ware
Hi,
I'm trying to embed python 2.7 in a Win32 application and I'm running into
a host of problems.
Originally, my problem was that when compiling a debug version of my
application and linking against the python27.lib that I installed via the
packaged installer, I would just get strange behavior and memory corruption
and random crashes. I finally determined that this was basically just
impossible, and if I want to use embedded python in a debug version of my
application, I need to build my own debug version of python.
This leads me to my first question: Was my original determination
accurate? I think all of my problems would be solved automatically if I
could just embed release python in a debug version of my application.
Fast forward a bit. I now have a custom debug build of python. My debug
build of application links against it, and release build links against
release python. So far so good, until I try to run "import ctypes". Then
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.

Post by Zachary Turner

Post by Zachary Turner
import ctypes

File "<console>", line 1, in <module>
File "d:\python_src\Python-2.7.8\Lib\ctypes\__init__.py", line 10, in
<module>
from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes
Anybody have any idea what is going on here or how I might diagnose this?
It's obviously located the ctypes module that came with the source
distribution, so what error could it be running into?
For reference, the way I've built this is to download Python 2.7 source
distribution, open up the solution in Visual Studio, and build it, then
have my application link against the binaries that are put into the build's
output directories. Is there a different way?
Thanks

[Zachary Turner]

Also, to answer your the question more specifically about sys.path, when I
run my debug build of python_d.exe from the commandl ine and print sys.path

Post by Zachary Turner

Post by Zachary Ware

Post by Zachary Ware
sys.path

['', 'D:\\python_src\\Python-2.7.8\\Lib',
'D:\\src\\llvm\\build\\ninja\\lib\\site-packages',
'd:\\python_src\\Python-2.7.8\\pcbuild\\python27_d.zip',
'd:\\python_src\\Python-2.7.8\\DLLs',
'd:\\python_src\\Python-2.7.8\\lib\\plat-win',
'd:\\python_src\\Python-2.7.8\\lib\\lib-tk',
'd:\\python_src\\Python-2.7.8\\pcbuild', 'd:\\python_src\\Python-2.7.8',
'd:\\python_src\\Python-2.7.8\\lib\\site-packages']
[43489 refs]

When I do the same thing from my embedded interpreter, I get this:
['D:\\src\\llvm\x08uild\ninja\x08in',
'D:\\src\\llvm\x08uild\ninja\x08in\\..\\lib\\site-packages',
'D:\\python_src\\Python-2.7.8\\Lib',
'D:\\src\\llvm\\build\\ninja\\lib\\site-packages',
'D:\\python_src\\Python-2.7.8\\PCbuild\\python27_d.zip',
'C:\\Python27\\Lib', 'C:\\Python27\\DLLs', 'C:\\Python27\\Lib\\lib-tk',
'D:\\src\\llvm\\build\\ninja', 'D:\\src\\llvm\\build\\ninja\\bin',
'C:\\Python27', 'C:\\Python27\\lib\\site-packages', '.']

The first two junked up entries are bugs I need to fix, but not relevant
really to this. It's worth noting that
d:\\python_src\\Python-2.7.8\\pcbuild is not included. That's probably
where the problem lies. Both of these were run with the same system
environment, so there's no difference there. Someone running python_d.exe
gets d:\\python_src\\Python-2.7.8\\pcbuild in sys.path, but linking against
the DLL doesn't.

One more thing that i find very strange is the presence of C:\Python27\* in
my sys.path here. None of this is in my PATH environment variable or my
PYTHONPATH. Where is it picking this up from? It's where I have installed
a pre-packaged version of Python, but since it's not in my path or in any
global setting that I'm aware of, I'm not sure where this would be coming
from.

Post by Zachary Turner
Apologies if this message appears incorrectly threaded. Initially I
didn't subscribe to the list, so I'm replying to my own post instead of

Post by Zachary Ware

Post by Zachary Ware
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.

Post by Zachary Turner
import ctypes

File "<console>", line 1, in <module>
File "d:\python_src\Python-2.7.8\Lib\ctypes\__init__.py", line 10, in
<module>
from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes
Anybody have any idea what is going on here or how I might diagnose this?
It's obviously located the ctypes module that came with the source
distribution, so what error could it be running into?

What do you get if you do 'import sys; print sys.path'? Is the
directory that holds _ctypes.pyd included in sys.path?

Post by Zachary Ware
For reference, the way I've built this is to download Python 2.7 source
distribution, open up the solution in Visual Studio, and build it, then have
my application link against the binaries that are put into the build's
output directories. Is there a different way?

That ought to be about right.

I did some more digging, and I think it's because ctypes is built as an
extension module, so _ctypes_d.pyd is in my build output directory
alongside python_d.exe and python27_d.dll, and not in my libs directory. I
managed to get it working if I add my build output directory to my
PYTHONPATH. Is there some way to package up my build and then install it
so that the directory structure is the same as if I had downloaded a
pre-packaged installer and installed it? I don't think PYTHONPATH is set
by a default install, so something about the way it installs itself (maybe
python.exe is hardcoded to look in the DLLs subfolder?) allows an installed
build of python to work without PYTHONPATH. I'd like to achieve the same
thing with my custom build.

Post by Zachary Ware
Hi,
I'm trying to embed python 2.7 in a Win32 application and I'm running
into a host of problems.
Originally, my problem was that when compiling a debug version of my
application and linking against the python27.lib that I installed via the
packaged installer, I would just get strange behavior and memory corruption
and random crashes. I finally determined that this was basically just
impossible, and if I want to use embedded python in a debug version of my
application, I need to build my own debug version of python.
This leads me to my first question: Was my original determination
accurate? I think all of my problems would be solved automatically if I
could just embed release python in a debug version of my application.
Fast forward a bit. I now have a custom debug build of python. My debug
build of application links against it, and release build links against
release python. So far so good, until I try to run "import ctypes". Then
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.

Post by Zachary Ware

Post by Zachary Turner
import ctypes

File "<console>", line 1, in <module>
File "d:\python_src\Python-2.7.8\Lib\ctypes\__init__.py", line 10, in
<module>
from _ctypes import Union, Structure, Array
ImportError: No module named _ctypes
Anybody have any idea what is going on here or how I might diagnose
this? It's obviously located the ctypes module that came with the source
distribution, so what error could it be running into?
For reference, the way I've built this is to download Python 2.7 source
distribution, open up the solution in Visual Studio, and build it, then
have my application link against the binaries that are put into the build's
output directories. Is there a different way?
Thanks

[Zachary Ware]

Post by Zachary Turner
Also, to answer your the question more specifically about sys.path, when I
run my debug build of python_d.exe from the commandl ine and print sys.path

Post by Zachary Ware
sys.path

['', 'D:\\python_src\\Python-2.7.8\\Lib',
'D:\\src\\llvm\\build\\ninja\\lib\\site-packages',
'd:\\python_src\\Python-2.7.8\\pcbuild\\python27_d.zip',
'd:\\python_src\\Python-2.7.8\\DLLs',
'd:\\python_src\\Python-2.7.8\\lib\\plat-win',
'd:\\python_src\\Python-2.7.8\\lib\\lib-tk',
'd:\\python_src\\Python-2.7.8\\pcbuild', 'd:\\python_src\\Python-2.7.8',
'd:\\python_src\\Python-2.7.8\\lib\\site-packages']
[43489 refs]
['D:\\src\\llvm\x08uild\ninja\x08in',
'D:\\src\\llvm\x08uild\ninja\x08in\\..\\lib\\site-packages',
'D:\\python_src\\Python-2.7.8\\Lib',
'D:\\src\\llvm\\build\\ninja\\lib\\site-packages',
'D:\\python_src\\Python-2.7.8\\PCbuild\\python27_d.zip',
'C:\\Python27\\Lib', 'C:\\Python27\\DLLs', 'C:\\Python27\\Lib\\lib-tk',
'D:\\src\\llvm\\build\\ninja', 'D:\\src\\llvm\\build\\ninja\\bin',
'C:\\Python27', 'C:\\Python27\\lib\\site-packages', '.']
The first two junked up entries are bugs I need to fix, but not relevant
really to this. It's worth noting that
d:\\python_src\\Python-2.7.8\\pcbuild is not included. That's probably
where the problem lies. Both of these were run with the same system
environment, so there's no difference there. Someone running python_d.exe
gets d:\\python_src\\Python-2.7.8\\pcbuild in sys.path, but linking against
the DLL doesn't.

I will note that I haven't done any embedding of Python myself, so I'm
shooting in the dark a little bit here.

[Ajay Kanade]

Zachary,

Python embedding in another application (win32) is done quite a bit by many (me too). My suggestion to you is:
- Build both release & debug
- After the builds, do the "Install Python" steps at https://wiki.python.org/moin/VS2012 for *both* release & debug builds. Note: I had to modify manual_install.py because I did not build tcl (something like that)
- If above step become too much, do a proper installation of python (.msi) & replace all .pyds & dlls & exes with your built ones
- Idea is to keep both *.pyd and *_d.pyd (& *.dll & *_d.dll) in the same folders. This way both release & debug of your app will execute
- Link release/debug python libs with your release/debug exe respectively
- Take a look at http://stackoverflow.com/questions/1387906/c-with-python-embedding-crash-if-python-not-installed/2970407#2970407 to see how you can package python with your exe. Also it shows how you can set PythonHome in your C code
- If you are going to create .pyds of your own (using swig or boost), note that you must have xyz.pyd & xyz_d.pyd for release & debug respectively
- Python27.dll, Python27_d.dll & the Python27/DLLs & Python27/Lib are the only things needed at runtime

My suggestion for you would be to make it work for a pure python installation (you just have to build the python27.dll) before building the whole python thing yourself.

Hope it helps. I may have missed something. But that should give you an idea.

Ajay

-----Original Message-----
From: python-win32 [mailto:python-win32-bounces+ajay.kanade=***@python.org] On Behalf Of Zachary Ware
Sent: Saturday, October 04, 2014 2:46 AM
To: python-***@python.org
Subject: Re: [python-win32] Embedding python in a Win32 application

Post by Zachary Turner
Also, to answer your the question more specifically about sys.path,
when I run my debug build of python_d.exe from the commandl ine and

Post by Zachary Ware
sys.path

['', 'D:\\python_src\\Python-2.7.8\\Lib',
'D:\\src\\llvm\\build\\ninja\\lib\\site-packages',
'd:\\python_src\\Python-2.7.8\\pcbuild\\python27_d.zip',
'd:\\python_src\\Python-2.7.8\\DLLs',
'd:\\python_src\\Python-2.7.8\\lib\\plat-win',
'd:\\python_src\\Python-2.7.8\\lib\\lib-tk',
'd:\\python_src\\Python-2.7.8\\pcbuild',
'd:\\python_src\\Python-2.7.8',
'd:\\python_src\\Python-2.7.8\\lib\\site-packages']
[43489 refs]
['D:\\src\\llvm\x08uild\ninja\x08in',
'D:\\src\\llvm\x08uild\ninja\x08in\\..\\lib\\site-packages',
'D:\\python_src\\Python-2.7.8\\Lib',
'D:\\src\\llvm\\build\\ninja\\lib\\site-packages',
'D:\\python_src\\Python-2.7.8\\PCbuild\\python27_d.zip',
'C:\\Python27\\Lib', 'C:\\Python27\\DLLs',
'C:\\Python27\\Lib\\lib-tk', 'D:\\src\\llvm\\build\\ninja',
'D:\\src\\llvm\\build\\ninja\\bin',
'C:\\Python27', 'C:\\Python27\\lib\\site-packages', '.']
The first two junked up entries are bugs I need to fix, but not
relevant really to this. It's worth noting that
d:\\python_src\\Python-2.7.8\\pcbuild is not included. That's
probably where the problem lies. Both of these were run with the same
system environment, so there's no difference there. Someone running
python_d.exe gets d:\\python_src\\Python-2.7.8\\pcbuild in sys.path,
but linking against the DLL doesn't.

I will note that I haven't done any embedding of Python myself, so I'm shooting in the dark a little bit here.