Quantcast
Channel: Active questions tagged python - Stack Overflow
Viewing all articles
Browse latest Browse all 13981

Why doesn't an error stop a parallel python script [duplicate]

$
0
0

My intension is to raise an error and stop the Python script when certain condition met. Here is the relevant code snippet.

def my_func(some arguments):    # some code    if X.abs().max().max() > 1e50:        print('------------ ERROR! line 287, extremely large X={} ----------------'.format(X.abs().max().max()), file=sys.stderr)        print('selectedFeatures: '+('-'.join(sorted(list(selectedFeatures)))), file=sys.stderr)        print('featurePool: '+('-'.join(sorted(list(featurePool)))), file=sys.stderr)        print('p1: '+p1+',  p2: '+p2, file=sys.stderr)        print('p1: '+p1+',  p2: '+p2, file=sys.stderr)        print('X.shape={}X{}'.format(X.shape[0],X.shape[1]), file=sys.stderr)        tmp = 1/0 # to create an error    # some codedef caller_func():    for-loop:        # some code        my_func(arguments)        # some code

Part of the output is below, and the script keeps running even though there are errors.

# some stdout output content------------ ERROR! line 287, extremely large X=3.130749212136059e+58 ----------------selectedFeatures: <some_string>featurePool: <some_string>p1: <some_string>, p2: <some_string>p1: <some_string>, p2: <some_string>X.shape=133615X41/usr/lib/python3.10/multiprocessing/process.py:-1: ResourceWarning: unclosed file <_io.TextIOWrapper name='my_file_1.csv' mode='w' encoding='UTF-8'>ResourceWarning: Enable tracemalloc to get the object allocation traceback/usr/lib/python3.10/multiprocessing/process.py:-1: ResourceWarning: unclosed file <_io.TextIOWrapper name='my_file_2.csv' mode='w' encoding='UTF-8'>ResourceWarning: Enable tracemalloc to get the object allocation tracebackTraceback (most recent call last):  File "//script_heavy_ga.py", line 332, in caller_func    X, selectedFeatures, featurePool = my_func(p1, p2, X, X_original, selectedFeatures, featurePool)  File "//script_heavy_ga.py", line 293, in my_func    tmp = 1/0 # to create an errorZeroDivisionError: division by zero# some stdout output content------------ ERROR! line 287, extremely large X=1.1940462536258555e+75 ----------------selectedFeatures: <some_string>featurePool: <some_string>p1: <some_string>, p2: <some_string>p1: <some_string>, p2: <some_string>X.shape=183025X41/usr/lib/python3.10/multiprocessing/process.py:-1: ResourceWarning: unclosed file <_io.TextIOWrapper name='my_file_3.csv' mode='w' encoding='UTF-8'>ResourceWarning: Enable tracemalloc to get the object allocation traceback/usr/lib/python3.10/multiprocessing/process.py:-1: ResourceWarning: unclosed file <_io.TextIOWrapper name='my_file_4.csv' mode='w' encoding='UTF-8'>ResourceWarning: Enable tracemalloc to get the object allocation tracebackTraceback (most recent call last):  File "//script_heavy_ga.py", line 332, in caller_func    X, selectedFeatures, featurePool = my_func(p1, p2, X, X_original, selectedFeatures, featurePool)  File "//script_heavy_ga.py", line 293, in my_func    tmp = 1/0 # to create an errorZeroDivisionError: division by zero# some stdout output content

Parallel computing is used in my code from multiprocessing import Pool. Multiple instances of the function caller_func() run on multiple cores.

My question is, why isn't the script stopped by the errors? Some previous posts (and this) don't seem relevant. This might be relevant, but it did not answer why my code did not stop.


Viewing all articles
Browse latest Browse all 13981

Trending Articles