[python-win32] USB access using win32file.deviceIOcontrol

[Sebastian Friebe]

Hi all,

I'm trying to get access to a USB mass storage device in order to send
low-level SCSI commands, like TEST_UNIT_READY and REQUEST_SENSE.
I'm looking for a pure Python solution using the deviceIOcontrol
function out of the win32file package.

Does anyone has experience with it?

I heared about libusb_win32 and pyUSB, but I couldn't figure out if
these are really required. I guess these packages are needed just in
case you want to do higher level operations in an application, right?

Thanks for your help in advance and best regards,
Sebastian

[Tim Roberts]

Post by Sebastian Friebe
I'm trying to get access to a USB mass storage device in order to send
low-level SCSI commands, like TEST_UNIT_READY and REQUEST_SENSE.
I'm looking for a pure Python solution using the deviceIOcontrol
function out of the win32file package.
Does anyone has experience with it?
I heared about libusb_win32 and pyUSB, but I couldn't figure out if
these are really required. I guess these packages are needed just in
case you want to do higher level operations in an application, right?

This can't be done as you have described it.

The problem is that the SCSI-like command set is only used between the
USB mass storage driver (usbstor.sys) and the USB host controller
driver. Above usbstor.sys, the device looks like a standard mass
storage device. There are no hooks in usbstor.sys to allow you to send
commands like this, nor does usbstor.sys allow you to send USB packets
directly. Neither libusb-win32 nor pyUSB will help, because there is
already a driver handling the device. libusb installs its own generic
driver, but it can't install a driver if another one has already claimed
the device.

Theoretically, you could write an INF file to force your device to be
claimed by libusb's driver, and then use libusb commands to talk to the
device. You would then be in COMPLETE control of the device. However,
that also means it won't be recognized as a mass storage device. It
will just be a generic USB device.

It may be that the easiest course of action for you is to write a lower
filter driver to sit between usbstor.sys and USBD, and have it expose a
"control" device object. Such a thing CAN be opened and manipulated
using the win32file APIs.

Do you have any driver experience at your shop?

[Sebastian Friebe]

Post by Tim Roberts
This can't be done as you have described it.
The problem is that the SCSI-like command set is only used between the
USB mass storage driver (usbstor.sys) and the USB host controller
driver. Above usbstor.sys, the device looks like a standard mass
storage device. There are no hooks in usbstor.sys to allow you to send
commands like this, nor does usbstor.sys allow you to send USB packets
directly.

I saw an implementation in C++ using the winAPI DeviceIOControl()
function to do low-level SCSI operations on a USB mass storage device.
It is defined as follows:

BOOL WINAPI DeviceIoControl(
__in HANDLE hDevice,
__in DWORD dwIoControlCode,
__in_opt LPVOID lpInBuffer,
__in DWORD nInBufferSize,
__out_opt LPVOID lpOutBuffer,
__in DWORD nOutBufferSize,
__out_opt LPDWORD lpBytesReturned,
__inout_opt LPOVERLAPPED lpOverlapped
);

following settings have been applied:
dwIoControlCode = IOCTL_SCSI_PASS_THROUGH_DIRECT
lpInBuffer = lpOutBuffer = a structure containing several settings
like PATH, TARGET, LUN IDs and CDB[]
array and a buffer for input and output
data

So if your statement is right, this should even not work under C++,
right? But it does.
Are there maybe some other stuff what makes it different?
I really would like to understand that topic.
Do you know some good references where I can find detailed information
about USB handling under Windows environment?

My first attempt to get it running under Python looks like this:

IOCTL_SCSI_PASS_THROUGH_DIRECT = 0x4D014

disk_handle=win32file.CreateFile("\\\\.\\j:",
win32con.GENERIC_READ|win32con.GENERIC_WRITE|win32con.GENERIC_EXECUTE,
win32file.FILE_SHARE_READ |win32file.FILE_SHARE_WRITE,
None,
win32con.OPEN_EXISTING,
0,
None);

data = array.array("B", byte_list)
string = win32file.DeviceIoControl(disk_handle,IOCTL_SCSI_PASS_THROUGH_DIRECT, data, 0, None)


It seems that Windows excepted the control code, cause if I use a
wrong one, I get an SYSTEM ERROR 50 - which is 'The request is not
supported.'

If I use the right control code I get sometimes the SYSTEM ERROR
1450 - "Insufficient system resources exist to complete the requested
service."

So it seemsfor me, that at least a contact to the USB driver is
possible.
What do you think?

Thanks again for your help.
Sebastian

[Tim Roberts]

Post by Sebastian Friebe
I saw an implementation in C++ using the winAPI DeviceIOControl()
function to do low-level SCSI operations on a USB mass storage device.
BOOL WINAPI DeviceIoControl(
...
dwIoControlCode = IOCTL_SCSI_PASS_THROUGH_DIRECT
lpInBuffer = lpOutBuffer = a structure containing several settings
like PATH, TARGET, LUN IDs and CDB[]
array and a buffer for input and output
data
So if your statement is right, this should even not work under C++,
right? But it does.

OK, you talked me into it.

Post by Sebastian Friebe
Are there maybe some other stuff what makes it different?
I really would like to understand that topic.
Do you know some good references where I can find detailed information
about USB handling under Windows environment?
IOCTL_SCSI_PASS_THROUGH_DIRECT = 0x4D014

That number is correct.

Post by Sebastian Friebe
disk_handle=win32file.CreateFile("\\\\.\\j:",
win32con.GENERIC_READ|win32con.GENERIC_WRITE|win32con.GENERIC_EXECUTE,
win32file.FILE_SHARE_READ |win32file.FILE_SHARE_WRITE,
None,
win32con.OPEN_EXISTING,
0,
None);

GENERIC_EXECUTE should not be required, but if your C++ example had it,
you might as well keep it. Does the CreateFile succeed? You get a
reasonable handle? Note that on Vista you must be elevated to open a
volume directly.

Post by Sebastian Friebe
data = array.array("B", byte_list)
string = win32file.DeviceIoControl(disk_handle,IOCTL_SCSI_PASS_THROUGH_DIRECT, data, 0, None)
It seems that Windows excepted the control code, cause if I use a
wrong one, I get an SYSTEM ERROR 50 - which is 'The request is not
supported.'

That's fundamentally correct. How are you creating the
SCSI_PASS_THROUGH structure in "byte_list"? Are you sure it is 42
bytes? Are you setting all the fields correctly? How are you setting
the DataBuffer pointer? Have you set the Length field correctly?

I would have guessed it would be easier to use the struct module to
build the buffer, rather than array.

[Sebastian Friebe]

Tim Roberts wrote:
TR> That's fundamentally correct. How are you creating the
TR> SCSI_PASS_THROUGH structure in "byte_list"? Are you sure it is 42
TR> bytes? Are you setting all the fields correctly? How are you setting
TR> the DataBuffer pointer? Have you set the Length field correctly?

That's exactly my problem.
I know there are some pointers in the C++ structure pointing to the
data inside the structure. But I don't have an idea at all, how to
port it to Python.

Could you give me an example of a very basic SCSI command, like the
TEST_UNIT_READY ?

Tim Roberts wrote:
TR> I would have guessed it would be easier to use the struct module to
TR> build the buffer, rather than array.

I didn't know if it would work, so I
started with a very strait forward approach.
I extracted the content of the SCSI_PASS_THROUGH in my C++ example
into a array of bytes.
I included the byte stream I found into my byte_list.
e.g.
SCSI_TEST_UNIT_READY=[0x2C,0x00,0x00,0x00,0x01,0x00,0x06,0x18,0x02,0x00,0x00,0x00,0x00,0x00,0x00,0x00,\
0x02,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x30,0x00,0x00,0x00,0x00,0x00,0x00,0x00,\
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,\
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,\
0x00,0x00,0x00,0x00]

byte_list = SCSI_TEST_UNIT_READY

If the basic functionality is proven, I planned to go for a more
object oriented approach like:

class SCSI_PASS_THROUGH_DIRECT(object):
def __init__(self, cdbLength = 16, transfer_length, transfer_direction):
self.Length = 0 # USHORT Length;
self.ScsiStatus = 0 # UCHAR ScsiStatus;
self.PathId = 0 # UCHAR PathId;
self.TargetId = 1 # UCHAR TargetId;
self.Lun = 0 # UCHAR Lun;
self.CdbLength = cdbLength # UCHAR CdbLength;
self.SenseInfoLength = 24 # UCHAR SenseInfoLength;
self.DataIn = transfer_direction # UCHAR DataIn;
self.DataTransferLength = transfer_length # ULONG DataTransferLength;
self.TimeOutValue = 2 # ULONG TimeOutValue;
self.DataBuffer = 0 # PVOID DataBuffer;
self.SenseInfoOffset = 0 # ULONG SenseInfoOffset;
self.Cdb = [] # UCHAR Cdb[16];

But again, I don't know how to handle the buffer pointers inside the
structure.

[Tim Roberts]

Post by Sebastian Friebe
That's exactly my problem.
I know there are some pointers in the C++ structure pointing to the
data inside the structure. But I don't have an idea at all, how to
port it to Python.
Could you give me an example of a very basic SCSI command, like the
TEST_UNIT_READY ?

I'm afraid I can't. I write Windows drivers for a living, so I do an
awful lot of DeviceIoControl calls, but so far we have managed to avoid
working the disk arena.

Post by Sebastian Friebe
I didn't know if it would work, so I
started with a very strait forward approach.
I extracted the content of the SCSI_PASS_THROUGH in my C++ example
into a array of bytes.
I included the byte stream I found into my byte_list.
e.g.
SCSI_TEST_UNIT_READY=[0x2C,0x00,0x00,0x00,0x01,0x00,0x06,0x18,0x02,0x00,0x00,0x00,0x00,0x00,0x00,0x00,\
0x02,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x30,0x00,0x00,0x00,0x00,0x00,0x00,0x00,\
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,\
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,\
0x00,0x00,0x00,0x00]

The length says 0x2C, which is 44 decimal, but unless I have miscounted,
there are 0x44 bytes of data there, which is 64 decimal.

Post by Sebastian Friebe
byte_list = SCSI_TEST_UNIT_READY
If the basic functionality is proven, I planned to go for a more
self.Length = 0 # USHORT Length;
self.ScsiStatus = 0 # UCHAR ScsiStatus;

I'm afraid that you will end up battling against the language here. I
have three possible suggestions for you.

1. Consider using ctypes.

ctypes includes mechanisms where you can build up C structures in a
Pythonic way, almost exactly like you have done here.

2. Consider using SWIG.

Swig will allow you to build a C or C++ DLL that can be called directly
from Python. You could hide the pointer ugliness inside the C code, and
still wrap it with a Python class. Swig is extraordinarily powerful,
although there is a bit of a learning curve if you need to do anything
unusual. There are lots of good example, however.

3. Consider using boost.python.

If you know C++, the Boost libraries include a very good set of template
classes that let you build Python object in C++ in a more natural way.
Like option 2, this would let you put the sticky parts in C++ and the
fun parts in Python.

[Tony Cappellini]

I recently wrote some C code to send SATA commands to a system drive,
using the ATA pass through layer.

I found this news group very helpful, since it deals with drivers.
http://groups.google.com/group/microsoft.public.windowsxp.device_driver.dev/topics

I wouldn't mentioned Python on that newsgroup though. If you post any
questions, pretend your doing it from C.
You may not get any help if you say you're trying to do it from Python. ;-)

Also- you should try to write the C or C++ code to actually send the
command first, until you get the mechanism down,
then use ctypes or a C DLL to be called from Python.

Also, stick with the Test Unit Ready command or any other command that
doesn't transfer data.

It will be easier to work out the mechanism that way. When you start
transferring data, drivers like the buffers to be aligned to some
specific boundary. Rather than get hung up on data transfer now, you
can deal with that later once you understand how to get a simple
command across.



Message: 4
Date: Wed, 12 Dec 2007 21:41:15 +0100
From: Sebastian Friebe <***@benkers-rock.de>
Subject: Re: [python-win32] USB access using win32file.deviceIOcontrol
To: Python-Win32 List <python-***@python.org>
Message-ID: <***@benkers-rock.de>
Content-Type: text/plain; charset=us-ascii

class SCSI_PASS_THROUGH_DIRECT(object):