What Is Numpy.ctypeslib.as_ctypes Exacty Doing
Solution 1:
As I still don't understand what the problem is, I'll just assume, and try to answer "blindly".
I'll start by answering the punctual question (somewhere at the end):
numpy.ctypeslib.as_ctypes converts a NumPy array (or object, or ...) into a CTypes one. But conversion only happens on the metadata (as it is different for the 2 modules). The array contents (or the pointer, or the memory address where the actual array data is (or begins)) is left unchanged (not copied / modified / altered, ...).
References:
[NumPy.Docs]: C-Types Foreign Function Interface (numpy.ctypeslib)
1.1. The source code (somewhere at the end): [GitHub]: numpy/numpy - numpy/numpy/ctypeslib.py
[Python.Docs]: ctypes - A foreign function library for Python
So, no transposition is done.
I'm not sure what you mean by "pretreatment" (do all the checks and operations performed by as_ctypes fit there?). Same about "natural bit".
Also note that as_ctypes (1 argument of pyfunc_array_by_ref_modifying) is completely unaware of the rest of them (pnbrows and pnbcols in particular).
Even if not directly impacting (I think it's a matter of "luck"), here's something you might want to check out: [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer).
Going to (what I think is) the real problem (that lead to the question):
- The array is passed to the Fortran subroutine by reference (its buffer start memory address). I don't have an official source to state this (it was an assumption since the beginning), but it's like in C passing a pointer (I guess it has something to do with C bind from the subroutine declaration)
- No metadata (rows, columns) is passed (otherwise the next 2 arguments would be useless)
The (2D) array is stored in memory as an 1D one: 1 row, followed by the 2 one, 3, ..., and the last one.
In order to reach element array[i][j] (i = [0, row_count), j = [0, column_count)), the following formula (pointer logic) is used: array_start_address + array_element_size_in_bytes * (i * column_count + j).
Hoping to clear some of the confusion, here's a small C demo. I replicated (what I thought it was) the Fortran subroutine behavior. I also increased the row count to make things clearer.
main00.c:
#include <stdio.h>
#include <string.h>
#define ROWS 3
#define COLS 6
//#pragma align 4
typedef int Array2D[ROWS][COLS];
/*
void modifyArray(Array2D array, int rows, int cols, int coef) {
for (int i = 0; i < rows; ++i)
for (int j = 0; j < cols; ++j)
array[i][j] *= coef;
}
//*/
void modifyPtr(int *pArray, int rows, int cols, int coef) { // This is the equivalent.
for (int i = 0; i < rows; ++i)
for (int j = 0; j < cols; ++j)
(*(pArray + (i * cols + j))) *= coef;
}
void printArray(Array2D array, int rows, int cols, char *head) {
printf("\n%s:\n", head);
for (int i = 0; i < rows; ++i) {
for (int j = 0; j < cols; ++j)
printf("% 3d ", array[i][j]);
printf("\n");
}
}
void modify(Array2D arr, int rows, int cols, int coef, char *head) {
printf("\nRows: %d, Cols: %d, Coef: %d", rows, cols, coef);
Array2D arr_dup;
memcpy(arr_dup, arr, sizeof(Array2D));
modifyPtr(arr_dup, rows, cols, coef);
printArray(arr_dup, ROWS, COLS, head);
}
int main() {
Array2D arr = { // ROWS X COLS
{ 1, 2, 3, 4, 5, 6 },
{ 7, 8, 9, 10, 11, 12 },
{ 13, 14, 15, 16, 17, 18 },
};
char *modifiedText = "Modified";
//printf("Array size: %d %d %d\n", sizeof(arr), sizeof(arr[0]), sizeof(arr[0][0]));
printArray(arr, ROWS, COLS, "Original");
modify(arr, 3, 6, 2, modifiedText);
printArray(arr, ROWS, COLS, "Original");
modify(arr, 2, 6, 2, modifiedText);
modify(arr, 1, 6, 2, modifiedText);
modify(arr, 3, 5, 2, modifiedText);
modify(arr, 3, 4, 2, modifiedText);
modify(arr, 5, 2, 2, modifiedText);
modify(arr, 7, 1, 2, modifiedText);
printf("\nDone.\n");
return 0;
}
Output:
[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q068314707]> sopr.bat ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [prompt]> "c:\Install\pc032\Microsoft\VisualStudioCommunity\2019\VC\Auxiliary\Build\vcvarsall.bat" x64 > nul [prompt]> dir /b code00.py example.f90 main00.c [prompt]> [prompt]> cl /nologo /MD /W0 main00.c /link /NOLOGO /OUT:main00_pc064.exe main00.c [prompt]> dir /b code00.py example.f90 main00.c main00.obj main00_pc064.exe [prompt]> [prompt]> main00_pc064.exe Original: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Rows: 3, Cols: 6, Coef: 2 Modified: 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 Original: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Rows: 2, Cols: 6, Coef: 2 Modified: 2 4 6 8 10 12 14 16 18 20 22 24 13 14 15 16 17 18 Rows: 1, Cols: 6, Coef: 2 Modified: 2 4 6 8 10 12 7 8 9 10 11 12 13 14 15 16 17 18 Rows: 3, Cols: 5, Coef: 2 Modified: 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 16 17 18 Rows: 3, Cols: 4, Coef: 2 Modified: 2 4 6 8 10 12 14 16 18 20 22 24 13 14 15 16 17 18 Rows: 5, Cols: 2, Coef: 2 Modified: 2 4 6 8 10 12 14 16 18 20 11 12 13 14 15 16 17 18 Rows: 7, Cols: 1, Coef: 2 Modified: 2 4 6 8 10 12 14 8 9 10 11 12 13 14 15 16 17 18 Done.
Back to Fortran (saved your script locally) + Python (created a new one).
code00.py:
#!/usr/bin/env python
import sys
import ctypes as ct
import numpy as np
IntPtr = ct.POINTER(ct.c_int)
DLL_NAME = "./example.{:s}".format("dll" if sys.platform[:3].lower() == "win" else "so")
def modify(arr, rows, cols, coef, modify_func):
print("\nRows: {:d}, Cols {:d}, Coef: {:d}".format(rows, cols, coef))
arr_dup = arr.copy()
arr_ct = np.ctypeslib.as_ctypes(arr_dup)
rows_ct = ct.c_int(rows)
cols_ct = ct.c_int(cols)
coef_ct = ct.c_int(coef)
modify_func(arr_ct, ct.byref(rows_ct), ct.byref(cols_ct), ct.byref(coef_ct))
print("Modified array:\n {:}".format(arr_dup))
def main(*argv):
dll00 = ct.CDLL(DLL_NAME)
func = getattr(dll00, "array_by_ref_modifying")
#func.argtypes = (ct.c_void_p, ct.c_void_p, ct.c_void_p, ct.c_void_p)
func.argtypes = (ct.c_void_p, IntPtr, IntPtr, IntPtr)
func.restype = None
arr = np.array([
[1, 2, 3, 4, 5, 6],
[7, 8, 9, 10, 11, 12],
[13, 14, 15, 16, 17, 18],
], dtype=ct.c_int)
print("Original array:\n {:}".format(arr))
modify(arr, 3, 6 , 2, func)
modify(arr, 2, 6 , 2, func)
modify(arr, 1, 6 , 2, func)
modify(arr, 3, 5 , 2, func)
modify(arr, 3, 4 , 2, func)
modify(arr, 5, 2 , 2, func)
modify(arr, 7, 1 , 2, func)
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.")
sys.exit(rc)
Output:
[prompt]> "f:\Install\pc032\Intel\OneAPI\Version\compiler\2021.3.0\windows\bin\intel64\ifort.exe" /c example.f90 Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.3.0 Build 20210609_000000 Copyright (C) 1985-2021 Intel Corporation. All rights reserved. [prompt]> link /NOLOGO /DLL /OUT:example.dll /LIBPATH:"f:\Install\pc032\Intel\OneAPI\Version\compiler\2021.3.0\windows\compiler\lib\intel64_win" example.obj Creating library example.lib and object example.exp [prompt]> dir /b code00.py example.dll example.exp example.f90 example.lib example.mod example.obj main00.c main00.obj main00_pc064.exe [prompt]> [prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.08.07_test0\Scripts\python.exe" code00.py Python 3.8.7 (tags/v3.8.7:6503f05, Dec 21 2020, 17:59:51) [MSC v.1928 64 bit (AMD64)] 064bit on win32 Original array: [[ 1 2 3 4 5 6] [ 7 8 9 10 11 12] [13 14 15 16 17 18]] Rows: 3, Cols 6, Coef: 2 Modified array: [[ 2 4 6 8 10 12] [14 16 18 20 22 24] [26 28 30 32 34 36]] Rows: 2, Cols 6, Coef: 2 Modified array: [[ 2 4 6 8 10 12] [14 16 18 20 22 24] [13 14 15 16 17 18]] Rows: 1, Cols 6, Coef: 2 Modified array: [[ 2 4 6 8 10 12] [ 7 8 9 10 11 12] [13 14 15 16 17 18]] Rows: 3, Cols 5, Coef: 2 Modified array: [[ 2 4 6 8 10 12] [14 16 18 20 22 24] [26 28 30 16 17 18]] Rows: 3, Cols 4, Coef: 2 Modified array: [[ 2 4 6 8 10 12] [14 16 18 20 22 24] [13 14 15 16 17 18]] Rows: 5, Cols 2, Coef: 2 Modified array: [[ 2 4 6 8 10 12] [14 16 18 20 11 12] [13 14 15 16 17 18]] Rows: 7, Cols 1, Coef: 2 Modified array: [[ 2 4 6 8 10 12] [14 8 9 10 11 12] [13 14 15 16 17 18]] Done.
Conclusions:
- The array is modified in the same way in the 2 examples, confirming my assumption
- No matter what row and column numbers are passed, the array is modified from the beginning, left to right, top to bottom. This might be a bit confusing (when picturing the 2D representation and for example passing a column number that's smaller than the actual one). That's why it works with a dimension greater than the actual one. The only thing to be careful about is not to go beyond the number of elements (row_count * column_count), because that would yield Undefined Behavior (and it might crash)
- I'm not sure about this one, but mentioning it anyway: there might be some other Undefined Behavior cases, e.g. when each row in the array would be padded by compiler in order to be properly aligned (similar to
#pragma pack). Example: a char array with 7 columns (a row would have 7 bytes) might be 8 bytes aligned. Not sure how pointer logic copes with that
Post a Comment for "What Is Numpy.ctypeslib.as_ctypes Exacty Doing"