Catch BrokenPipeError in CLI and don't log exception #369

geowurster · 2016-06-30T23:32:56Z

When piping multiline outputs like feature sequences to another utility that does not fully consume the output an exception is raised and logged to stderr. If its easy, handling KeyboardInterrupt would be nice too. BrokenPipeError is ignored by default on Python 3, but Python 2 takes a bit of extra handling like:

try:
    # CLI logic
except BrokenPipeError as e:
    if sys.version_info[0] == 2 and 'broken pipe' in str(e):
        ctx.exit()  # Exit the click context
    else:
        raise
except Exception:
    logger.exception("Exception encountered ...")
    raise

For example:

$ fio cat tests/data/coutwildrnp.shp --precision 1 | head -1
{"bbox": [-111.8, 41.7, -111.6, 42.0], "geometry": {"coordinates": [[[-111.7, 42.0], [-111.7, 42.0], [-111.7, 42.0], [-111.7, 42.0], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.6, 41.9], [-111.6, 41.9], [-111.6, 41.9], [-111.6, 41.9], [-111.6, 41.8], [-111.6, 41.8], [-111.6, 41.8], [-111.6, 41.8], [-111.6, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.8, 41.8], [-111.8, 41.7], [-111.8, 41.8], [-111.8, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.8, 41.8], [-111.8, 41.8], [-111.8, 41.8], [-111.8, 41.8], [-111.8, 41.8], [-111.8, 41.8], [-111.8, 41.8], [-111.8, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.7, 41.8], [-111.8, 41.8], [-111.8, 41.9], [-111.8, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.7, 41.9], [-111.8, 41.9], [-111.8, 41.9], [-111.7, 41.9], [-111.7, 42.0], [-111.7, 42.0], [-111.7, 42.0], [-111.7, 42.0], [-111.7, 42.0], [-111.7, 42.0], [-111.7, 42.0], [-111.7, 42.0], [-111.7, 42.0]]], "type": "Polygon"}, "id": "0", "properties": {"AGBUR": "FS", "AREA": 0.0179264, "FEATURE1": "Wilderness", "FEATURE2": null, "NAME": "Mount Naomi Wilderness", "PERIMETER": 1.22107, "STATE": "UT", "STATE_FIPS": "49", "URL": "http://www.wilderness.net/index.cfm?fuse=NWPS&sec=wildView&wname=Mount%20Naomi", "WILDRNP020": 332}, "type": "Feature"}
ERROR:fio:Exception caught during processing
Traceback (most recent call last):
  File "/Users/wursterk/code/Fiona/fiona/fio/cat.py", line 85, in cat
    click.echo(json.dumps(feat, **dump_kwds))
  File "/Users/wursterk/code/Fiona/venv/lib/python3.5/site-packages/click/utils.py", line 260, in echo
    file.flush()
BrokenPipeError: [Errno 32] Broken pipe
Aborted!
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

sgillies · 2017-08-17T21:54:30Z

@geowurster I'd like to keep punting on this one. The exception goes to stderr (you can verify with $ fio cat tests/data/coutwildrnp.shp --precision 1 2> /dev/null | head -1) and not stdout like the JSON data.

geowurster · 2017-09-03T16:32:12Z

@sgillies The note about the exception being emitted to stdout is a typo. I edited my initial comment. This definitely isn't a blocker and my mail goal was to eliminate confusion.

geowurster added enhancement good first issue labels Jun 30, 2016

sgillies added the wontfix label Aug 17, 2017

geowurster mentioned this issue Oct 27, 2017

Add head option to 'fio cat' #483

Closed

Oct	NOV	Dec
	06
2019	2020	2021

Toblerity / Fiona

Catch BrokenPipeError in CLI and don't log exception #369

Catch BrokenPipeError in CLI and don't log exception #369

geowurster commented Jun 30, 2016 •

edited

sgillies commented Aug 17, 2017

geowurster commented Sep 3, 2017

Toblerity / Fiona

Join GitHub today

Catch BrokenPipeError in CLI and don't log exception #369

Catch BrokenPipeError in CLI and don't log exception #369

Comments

geowurster commented Jun 30, 2016 • edited

sgillies commented Aug 17, 2017

geowurster commented Sep 3, 2017

Essential cookies

Always active

Analytics cookies

geowurster commented Jun 30, 2016 •

edited