I'm guessing this question is a bit over my head, or I misunderstood it, but are you looking for something like
ps auxww and then checking the VSZ column. Then if VSZ hits a certain amount, you execute your SIG on that process? And then just run the command at your favorite interval?
From ps man page
vsz VSZ virtual memory size of the process in KiB (1024-byte units). Device mappings are currently excluded; this is subject to change. (alias vsize).
What you are referring to is process checkpointing. There is some work in the later kernels to offer this (in conjunction with the freezer cgroup) but its not ready yet.
This is actually very difficult to achieve well unfortunately because certain resources which are shared go stale after being unavailable for a fixed period of time (TCP springs to mind, although this may also apply to applications that use a wall clock, or perhaps some shared memory that changes state during a processes offline period).
As for stopping the process when it reaches a certain memory utilization, theres a hack I can think of that will do this.
cgroup.event_control
and set a memory threshold that you do not want to exceed (this is somewhat explained in the kernel documentation.)Note the "freeze" cgroup will not evict pages to a media persistent location, but it will swap the pages out when enough time has passed and the pages are needed for something else.
Even if this does work (its pretty hacky if it did) you need to consider whether or not this is really doing anything to solve your problem.
sched_min_granularity_ns
.Unfortunately, the best solution would be the ability to checkpoint your tasks. Its a shame that most of the implementations are just not that concrete enough yet.
Alternatively, you could wait a couple of years for proper checkpoint/restore to be available in the kernel!