Public Types |
Public Attributes |
Private Attributes |
Static Private Attributes |
List of all members
Gambit::SignalData Class Reference Variables for use in signal handlers. More...
Detailed DescriptionVariables for use in signal handlers. Definition at line 74 of file signal_handling.hpp. Member Typedef Documentation◆ void_func
Set cleanup function to run during emergency shutdown. Definition at line 91 of file signal_handling.hpp. Constructor & Destructor Documentation◆ SignalData()
SignalData member functions. Constructor (initialise member variables) Definition at line 142 of file signal_handling.cpp. bool shutdown_due_to_MPI_message Definition: signal_handling.hpp:188 volatile sig_atomic_t shutdownBegun Flag to warn if early shutdown is already in process. Definition: signal_handling.hpp:180 bool POSIX_signal_noticed Flag to indicate if POSIX shutdown signal has been noticed. Definition: signal_handling.hpp:185 bool inside_omp_block Flag to switch signal handling behavior to multithreaded mode (i.e. Definition: signal_handling.hpp:199 volatile sig_atomic_t emergency Flag to warn if the shutdown that is in progress is an emergency shutdown (use to decided whether to ... Definition: signal_handling.hpp:183 int shutdown_attempts Number of times synchronisation for soft shutdown has been attempted;. Definition: signal_handling.hpp:191 Member Function Documentation◆ add_signal()Check if emergency shutdown is in progress. Add signal to record. Add signal to record Definition at line 209 of file signal_handling.cpp. References MAX_SIGNALS, N_signals, and received_signals. Referenced by Gambit::sighandler_soft(). int received_signals[MAX_SIGNALS] Definition: signal_handling.hpp:203 static const int MAX_SIGNALS Array to record received signals (up to max_signals) Definition: signal_handling.hpp:202
Here is the caller graph for this function:
![]() ◆ all_processes_ready()
Attempt to synchronise all processes, but abort if it takes too long. Definition at line 265 of file signal_handling.cpp. References Gambit::EOM, and Gambit::logger(). Referenced by attempt_soft_shutdown(). const Logging::endofmessage EOM Explicit const instance of the end of message struct in Gambit namespace. Definition: logger.hpp:100 EXPORT_SYMBOLS Logging::LogMaster & logger() Function to retrieve a reference to the Gambit global log object. Definition: logger.cpp:95
Here is the call graph for this function:
![]()
Here is the caller graph for this function:
![]() ◆ attempt_soft_shutdown()
Perform soft shutdown if processes can be synchronised. Start counting... First time we see the shutdown signal, we will allow control to return to the scanner at least once, so that it can get its own affairs in order. Definition at line 284 of file signal_handling.cpp. References all_processes_ready(), attempts_since_ff, call_cleanup(), Gambit::EOM, ff_count, ff_loop_count, ff_on, Gambit::logger(), myrank(), and shutdown_attempts. Referenced by Gambit::Scanner::Scanner_Plugin_Function< ret(args...)>::main(), Gambit::Scanner::Multi_Scanner_Plugin_Function< ret(args...)>::main(), and Gambit::Likelihood_Container::main(). 286 const int max_attempts=-1; // Number of extra likelihood evaluations allowed for sync attempts before we declare failure. -1 means "unlimited" 287 const int attempts_before_ff=10; // Number of times to attempt synchronisation before entering a "fast forward" period 291 static std::chrono::time_point<std::chrono::system_clock> start(std::chrono::system_clock::now()); 297 logger() << "Beginning GAMBIT soft shutdown procedure. Control will be returned to the scanner plugin so " 298 << "that it can get its affairs in order in preparation for shutdown (it may cease iterating if " 299 << "it has that capability), and next iteration we will attempt to synchronise all processes and " 300 << "shut them down. If sync fails, we will loop up to "<<max_attempts<<" times (-1 means infinite), attempting to " 306 logger() << "Fast-forward active (loop "<<ff_loop_count<<"); no synchronisation attempted." << EOM; 322 msg << "rank "<<myrank()<<": Tried to synchronise for shutdown (attempt "<<shutdown_attempts<<") but failed. Will now fast-forward through "<<ff_loops<<" iterations in an attempt to 'unlock' possible MPI deadlocks with the scanner."; 333 logger() << "Scanner did not shut down when given the chance; we will therefore assume responsibility for terminating the scan." << EOM; 335 logger() << "Attempting to synchronise for soft shutdown (attempt "<<shutdown_attempts<<")" << EOM; 351 logger() << "Failed to synchronise for soft shutdown! Attempting cleanup anyway, but cannot guarantee safety of the scan output." << EOM; 357 msg << "Soft shutdown failed, emergency shutdown performed instead! (could not synchronise all processes after "<<shutdown_attempts 358 <<" attempts, and after waiting "<<std::chrono::duration_cast<std::chrono::seconds>(time_waited).count() 360 <<" times). Data handled by external scanner codes may have been left in an inconsistent state." << std::endl; 365 logger() << "Attempt to sync for soft shutdown failed (this was attempt "<<shutdown_attempts<<" of "<<max_attempts<<" (-1 means infinite)); " 366 <<std::chrono::duration_cast<std::chrono::seconds>(time_waited).count() <<" seconds have elapsed since " 367 <<"shutdown attempts began). Will allow evaluation to continue and attempt to sync again next iteration." << EOM;
bool all_processes_ready() Attempt to synchronise all processes, but abort if it takes too long. Definition: signal_handling.cpp:265 const Logging::endofmessage EOM Explicit const instance of the end of message struct in Gambit namespace. Definition: logger.hpp:100 EXPORT_SYMBOLS Logging::LogMaster & logger() Function to retrieve a reference to the Gambit global log object. Definition: logger.cpp:95 std::string myrank() Retrieve MPI rank as a string (for log messages etc.) Definition: signal_handling.cpp:175 int shutdown_attempts Number of times synchronisation for soft shutdown has been attempted;. Definition: signal_handling.hpp:191
Here is the call graph for this function:
![]()
Here is the caller graph for this function:
![]() ◆ call_cleanup()
Call cleanup function. Definition at line 202 of file signal_handling.cpp. References cleanup, and cleanup_function_set. Referenced by attempt_soft_shutdown().
Here is the caller graph for this function:
![]() ◆ check_if_shutdown_begun()
Check for signals that early shutdown is required If an MPI message telling us to perform an emergency shutdown is received (which should only happen in the case of an error on some other process) then a shutdown exception is raised. Otherwise, we just return a bool indicating the shutdown status Definition at line 379 of file signal_handling.cpp. References Gambit::LogTags::core, display_received_signals(), emergency, Gambit::EOM, Gambit::LogTags::info, Gambit::logger(), POSIX_signal_noticed, rank, set_shutdown_begun(), shutdown_begun(), and shutdown_due_to_MPI_message. Referenced by Gambit::Printers::check_for_error_messages(). 383 // If shutdown is not known to be in progress, check for MPI messages telling us to initiate shutdown 387 logger() << LogTags::core << LogTags::info << "Doing Iprobe to check for shutdown messages from other processes (with MPI tag "<<signalComm->mytag<<")" << EOM; 402 logger() << LogTags::core << LogTags::info << "Received SOFT shutdown message from process with rank " << msg_status.MPI_SOURCE << EOM; 407 logger() << LogTags::core << LogTags::info << "Received EMERGENCY shutdown message from process with rank " << msg_status.MPI_SOURCE << EOM; 412 ss << "Received UNRECOGNISED shutdown message from process with rank " << msg_status.MPI_SOURCE<<". Performing emergency shutdown, but please note that this indicates a ***BUG*** somewhere in the signal handling code!!!"; 423 logger() << LogTags::core << LogTags::info << "No shutdown message detected; continuing as normal" << EOM; 448 if(not shutdown_due_to_MPI_message) // Don't broadcast another shutdown message if we are shutting down due to an MPI message we received. Assume that all processes will get the first message (otherwise for 1000 process job we will end up with 1000*1000 shutdown messages clogging up the network) std::string display_received_signals() Print to string a list of the signals received so far by this process. Definition: signal_handling.cpp:224 void set_shutdown_begun(const sig_atomic_t emergnc=0) Register that shutdown has begun. Definition: signal_handling.cpp:246 bool shutdown_due_to_MPI_message Definition: signal_handling.hpp:188 Definition: log_tags.hpp:36 const Logging::endofmessage EOM Explicit const instance of the end of message struct in Gambit namespace. Definition: logger.hpp:100 EXPORT_SYMBOLS Logging::LogMaster & logger() Function to retrieve a reference to the Gambit global log object. Definition: logger.cpp:95 bool POSIX_signal_noticed Flag to indicate if POSIX shutdown signal has been noticed. Definition: signal_handling.hpp:185 Definition: log_tags.hpp:47 bool shutdown_begun() Check if (any kind of) shutdown is in progress. Definition: signal_handling.cpp:261 volatile sig_atomic_t emergency Flag to warn if the shutdown that is in progress is an emergency shutdown (use to decided whether to ... Definition: signal_handling.hpp:183
Here is the call graph for this function:
![]()
Here is the caller graph for this function:
![]() ◆ display_received_signals()
Print to string a list of the signals received so far by this process. Definition at line 224 of file signal_handling.cpp. References MAX_SIGNALS, N_signals, received_signals, and Gambit::signal_name(). Referenced by check_if_shutdown_begun(), and main(). 234 myout <<"#"<<i+1<<": Signal "<<received_signals[i]<<" ("<<signal_name(received_signals[i])<<")"<<std::endl; 239 myout << "Another " << (N_signals - MAX_SIGNALS) <<" signals were caught but their values were not recorded (buffer exceeded)"<<std::endl; std::string signal_name(int sig) Translate signal codes to strings. Definition: signal_handling.cpp:35 int received_signals[MAX_SIGNALS] Definition: signal_handling.hpp:203 static const int MAX_SIGNALS Array to record received signals (up to max_signals) Definition: signal_handling.hpp:202
Here is the call graph for this function:
![]()
Here is the caller graph for this function:
![]() ◆ entering_multithreaded_region()
Check if shutdown is in progress and raise appropriate termination exception if so. Absorb any extra shutdown messages that may be unreceived (since every process broadcasts to every other process that it should shut down, so with lots of processess there will be lots of unreceived messages floating around) (to be called by Gambit once it is safe to trigger termination) Only check for emergency shutdown signals (i.e. do not attempt synchronisation) Switch to threadsafe signal handling mode TODO: Thread checking routines are no longer needed due to simplified shutdown method. Can be deleted when functors are updated to no longer call these routines. Switch to threadsafe signal handling mode Definition at line 550 of file signal_handling.cpp. References inside_omp_block. Referenced by Gambit::FunctorHelp::entering_multithreaded_region(). bool inside_omp_block Flag to switch signal handling behavior to multithreaded mode (i.e. Definition: signal_handling.hpp:199
Here is the caller graph for this function:
![]() ◆ inside_multithreaded_region()
Report 'true' if inside a multithreaded region (according to our own flag) Definition at line 568 of file signal_handling.cpp. References Gambit::LogTags::core, Gambit::EOM, Gambit::LogTags::info, inside_omp_block, LOCAL_INFO, Gambit::logger(), rank, and Gambit::utils_error(). bool inside_omp_block Flag to switch signal handling behavior to multithreaded mode (i.e. Definition: signal_handling.hpp:199
Here is the call graph for this function:
![]() ◆ leaving_multithreaded_region()
Exit threadsafe signal handling mode. Definition at line 559 of file signal_handling.cpp. References inside_omp_block. Referenced by Gambit::FunctorHelp::leaving_multithreaded_region(). bool inside_omp_block Flag to switch signal handling behavior to multithreaded mode (i.e. Definition: signal_handling.hpp:199
Here is the caller graph for this function:
![]() ◆ myrank()
Retrieve MPI rank as a string (for log messages etc.) Definition at line 175 of file signal_handling.cpp. References rank. Referenced by attempt_soft_shutdown(), Gambit::FunctorHelp::entering_multithreaded_region(), and Gambit::FunctorHelp::leaving_multithreaded_region().
Here is the caller graph for this function:
![]() ◆ set_cleanup()Set cleanup function. Definition at line 195 of file signal_handling.cpp. References cleanup, cleanup_function_set, and f. Referenced by main().
Here is the caller graph for this function:
![]() ◆ set_shutdown_begun()
Register that shutdown has begun. Definition at line 246 of file signal_handling.cpp. References emergency, and shutdownBegun. Referenced by check_if_shutdown_begun(), main(), and Gambit::sighandler_soft(). volatile sig_atomic_t shutdownBegun Flag to warn if early shutdown is already in process. Definition: signal_handling.hpp:180 volatile sig_atomic_t emergency Flag to warn if the shutdown that is in progress is an emergency shutdown (use to decided whether to ... Definition: signal_handling.hpp:183
Here is the caller graph for this function:
![]() ◆ setjump()
Set jump point;. Definition at line 188 of file signal_handling.cpp. References env, havejumped, and jumppoint_set. ◆ shutdown_begun()
Check if (any kind of) shutdown is in progress. Check if shutdown is in progress. Definition at line 261 of file signal_handling.cpp. References shutdownBegun. Referenced by check_if_shutdown_begun(). volatile sig_atomic_t shutdownBegun Flag to warn if early shutdown is already in process. Definition: signal_handling.hpp:180
Here is the caller graph for this function:
![]() ◆ update_looptime()Extra functions needed in MPI mode. Add a new loop time to internal array used to decide barrier timeout. Add a new loop time to internal array used to decide barrier timeout Definition at line 474 of file signal_handling.cpp. References Gambit::LogTags::core, Gambit::LogTags::debug, Gambit::EOM, Gambit::LogTags::info, LOCAL_INFO, Gambit::logger(), cyclic_include_check::loop, rank, and Gambit::utils_error().
Here is the call graph for this function:
![]() Member Data Documentation◆ attempts_since_ff
Definition at line 192 of file signal_handling.hpp. Referenced by attempt_soft_shutdown(). ◆ cleanup
Definition at line 92 of file signal_handling.hpp. Referenced by call_cleanup(), and set_cleanup(). ◆ cleanup_function_set
Definition at line 93 of file signal_handling.hpp. Referenced by call_cleanup(), and set_cleanup(). ◆ emergency
Flag to warn if the shutdown that is in progress is an emergency shutdown (use to decided whether to attempt MPI synchronisation) Definition at line 183 of file signal_handling.hpp. Referenced by check_if_shutdown_begun(), and set_shutdown_begun(). ◆ env
Saved information on calling environment for longjmp. Definition at line 82 of file signal_handling.hpp. Referenced by setjump(). ◆ ff_count
Definition at line 195 of file signal_handling.hpp. Referenced by attempt_soft_shutdown(). ◆ ff_loop_count
Definition at line 193 of file signal_handling.hpp. Referenced by attempt_soft_shutdown(). ◆ ff_on
Definition at line 194 of file signal_handling.hpp. Referenced by attempt_soft_shutdown(). ◆ havejumped
Definition at line 84 of file signal_handling.hpp. Referenced by setjump(). ◆ inside_omp_block
Flag to switch signal handling behavior to multithreaded mode (i.e. triggers switch to threadsafe emergency shutdown behaviour) Definition at line 199 of file signal_handling.hpp. Referenced by entering_multithreaded_region(), inside_multithreaded_region(), and leaving_multithreaded_region(). ◆ jumppoint_set
Definition at line 83 of file signal_handling.hpp. Referenced by setjump(). ◆ MAX_SIGNALS
Array to record received signals (up to max_signals) Definition at line 202 of file signal_handling.hpp. Referenced by add_signal(), and display_received_signals(). ◆ N_signals
Definition at line 204 of file signal_handling.hpp. Referenced by add_signal(), and display_received_signals(). ◆ POSIX_signal_noticed
Flag to indicate if POSIX shutdown signal has been noticed. Definition at line 185 of file signal_handling.hpp. Referenced by check_if_shutdown_begun(). ◆ rank
Definition at line 177 of file signal_handling.hpp. Referenced by check_if_shutdown_begun(), inside_multithreaded_region(), myrank(), and update_looptime(). ◆ received_signals
Definition at line 203 of file signal_handling.hpp. Referenced by add_signal(), and display_received_signals(). ◆ shutdown_attempts
Number of times synchronisation for soft shutdown has been attempted;. Definition at line 191 of file signal_handling.hpp. Referenced by attempt_soft_shutdown(). ◆ shutdown_due_to_MPI_message
Definition at line 188 of file signal_handling.hpp. Referenced by check_if_shutdown_begun(). ◆ shutdownBegun
Flag to warn if early shutdown is already in process. Definition at line 180 of file signal_handling.hpp. Referenced by set_shutdown_begun(), and shutdown_begun(). The documentation for this class was generated from the following files:
|