flask - Using python's Multiprocessing makes response hang on gunicorn -

first admit there few many keywords in title, indeed trying capture problem in correct manner. issue here can not seem able correctly create sub process using python multiprocessing module without causing webpage response hang. have tried few recent versions of gunicorn , problem persists. interestingly, problem never issue on ubuntu server, moving application rhel6.5 issue has presented itself. here workflow:

-route hit -form submitted hits route , triggers multiprocessing.process() created, work done sleep 30 seconds -the route appears finish, print statement after multiprocessing call printed, browser keeps connection open , not 'finish loading' (show page) until 30 seconds of sleep finished

note form submission not part of issue, helps in viewing issue happen.

here simple route , function produces issue:

def multi():     print 'begin multi'     time.sleep(30)     print 'end multi'  @app.route('/multiprocesstest', methods=['get','post']) def multiprocesstest():      syntaxform = forms.mainsyntaxform()      if syntaxform.validate_on_submit():         print 'before multi call'         th = multiprocessing.process(target=multi)         th.start()         print 'after multi call'         return redirect('multiprocesstest')      return render_template('syntax_main.html', form=syntaxform) 

after extended research , sparse google results problem, have not found conclusive. going try load balancer check problem gunicorn only.

ok, ran on flask/uwsgi/nginx/linux 14.04.

good news future readers: able resolve this. bad news: pretty sure horrible hack.

some interesting test code prove hang forever:

@app.route('/spin-up') def spin_up():   import multiprocessing mp   def spin_forever():     while true:       time.sleep(1)    print('starting new process')   p = mp.process(target=spin_forever)   print('created')   p.start()   print('started--generating result')   return flask.render_template_string('awesome') 

if hit endpoint '/spin-up' spin process , hang forever. pretty awesome, huh?

basic message-queues don't work unless go out-of-process (i.e., use message queue running in already-started different process) or don't validate success (i.e., wait success ack response).

the short answer if attempt validate sub-process succeeded, in trouble. using internal message queue between threads, , if waited "success" response, flask server still hang. e.g.,

@app.route('/spin-up') def spin_up():   put_start_message_on_queue():   while true:     if got_success_response_from_queue():       break     time.sleep(0.1)   return flask.render_template_string('awesome') 

this still hangs (forever), instead added second command message queue called 'restart':

@app.route('/spin-up') def spin_up():   put_start_message_on_queue()   while true:     if got_success_message_from_queue():       break     time.sleep(0.1)   put_restart_message_on_queue()   return flask.render_template_string('awesome') 

you have ensure restart_message, when received kills existing process, minimal amount of work before starting new one, potentially inserting

time.sleep(0) 

in message handler. huge hack far can tell, works consistently, , long process can restarted (sorry if that's not case you...)