N-Body 1.18
Message boards :
News :
N-Body 1.18
Message board moderation
Previous · 1 · 2 · 3 · 4 · Next
| Author | Message |
|---|---|
|
Send message Joined: 4 Sep 12 Posts: 219 Credit: 456,474 RAC: 0  
|
I've now watched an N-Body task through being pre-empted by BOINC to allow other projects' tasks to run. During suspension (with the app left in memory), one thread - perhaps a watchdog or timer thread - continued to show a Cycles Delta ~300K (trivial), but the worker threads were completely dormant. When BOINC gave the task the resume instruction, all four (in my case) worker threads resumed, with the normal 2 billion Cycles Delta. IMHO, the application's internal multi-threading behaviour is working properly (though I'll try to watch the final wrap-up at the end of this 100k_chisq_alt task, because I suspect the threads may finish out of synch, and wind down CPU usage to a single 'housekeeper' thread in the final stages.) I think the remaining issues are the initial runtime estimates for the tasks, and some rough edges in BOINC's scheduling of MT tasks: that hasn't been given much operational testing since BOINC v7 was launched, which is the primary reason why I'm here. Let's discuss that when Travis has got the server back on an even keel with respect to the GPU applications. BTW, I was recently reminded of a thread I started, and Travis contributed to, with regard to the difficulties of predicting runtimes in advance. I don't think much has happened on this issue in the intervening 18 months. http://www.setiusa.us/showthread.php?2415-BOINC-DA-admits-CreditNew-is-really-a-random-number-generator |
|
Send message Joined: 4 Sep 12 Posts: 219 Credit: 456,474 RAC: 0
|
yes, and but on my computer for mt dark tasks estimated at less then 10 minutes, 6 of 12 CPU's are parked for the entire duration of the task. Also, look at run time verses CPU time +/- equal in the results file. Why does the above group of tasks always have this condition! Bottom line the above referenced group of tasks are executing very ineffectively, perhaps with correct results, and are reserving 1100% more resources meaning only 1 CPU is required, and 12 CPUs are reserved for the entire run time! I think I may have a better idea of this 'dark' task problem now. I was running ps_nbody_06_06_dark_1371001451_18919 when I took this screenshot.  (direct link) BOINC Manager (in the background) shows that the task has been running for 3 minutes already, but it's still showing 0%age progress - and it's only using one thread (left-hand Process Explorer window). A couple of minutes later, it went into full multi-threaded mode, and progress jumped from 0% to 100% in a couple of seconds. Finally, there was a further 5 minutes or so more running after 100% had been reached, during which again only one thread was active. Other types of task seem to spend far less time (both absolutely and relatively) in the single-threaded plateaux at 0% and 100%, and far longer in the multi-threaded stage where the progress %age counter is incrementing steadily. Hope that help the programmers track down what's going on. |
|
Send message Joined: 4 Apr 13 Posts: 3 Credit: 572,285 RAC: 0  
|
This morning after loading with new work a new mt job was stuck at 62.7% so I suspended all the 40 mwah gpu jobs and the 3 seti jobs (1 gpu), mt job now running to completion. |
|
Send message Joined: 4 Apr 13 Posts: 3 Credit: 572,285 RAC: 0
|
And another mt stuck, this time at 46.415% after one minute precisely. Every test batch has one of these problem mt jobs wanting all the resources to enable completion. |
|
Send message Joined: 4 Sep 12 Posts: 219 Credit: 456,474 RAC: 0
|
And another mt stuck, this time at 46.415% after one minute precisely. Every test batch has one of these problem mt jobs wanting all the resources to enable completion. It's not the batch or the WUs wanting anything, it's the bug in the BOINC client scheduler - it's being too assertive when trying to avoid over-committing the CPUs. |
|
Send message Joined: 22 Jun 11 Posts: 32 Credit: 41,852,496 RAC: 0  
|
Richard, I'm not trying to hijack a thread here, but... I wanted to point out that the scheduling policy may not be bugged here. There's a basic "Job Scheduling" section within the BOINC documentation, found here: http://boinc.berkeley.edu/trac/wiki/ClientSched Those 4 bullet points drive the main scheduling. But I think, additionally (per the emails below from April, where I asked David Anderson about it a bit)... the scheduler makes sure never to be commit more than #ncpus, when running a multi-thread (mt) task. Conversely, it won't schedule an mt task if the resulting cpu usage would be more than #ncpus. What's happening, in your case (I think), is that the mt task runs high-priority for a bit (as "Job Scheduling" bullet 2), then at some point BOINC thinks it can finish by deadline, so... it re-evaluates the task list to do scheduling, and schedules a GPU task (as "Job Scheduling" bullet 3), and then when deciding to schedule the mt task (as "Job Scheduling" bullet 4), it does not schedule it, since doing so would result in cpu usage more than #ncpus. If you have enough GPU tasks to always be running at least 1 GPU task, it will never re-schedule the mt task until BOINC puts it in high-priority mode again. Believe it or not... I think it may be working correctly, and I cannot think of a better design. If you can, you might dig up the boinc_alpha thread, and reply to it. Maybe a design that treats mt tasks and gpu tasks as equals, above regular-cpu tasks? I don't run mt tasks, so I don't have behavioral proof, but I'm betting your behavior matches the policies described here; ie: no bug. Regards, Jacob Klein ------------------------------------------------------------------------------------ Date: Tue, 2 Apr 2013 14:49:34 -0700 From: davea@ssl.berkeley.edu To: boinc_alpha@ssl.berkeley.edu Subject: Re: [boinc_alpha] Using app_config.xml <cpu_usage>2</cpu_usageresults in underloading/overloading CPU These are both consistent with the current job-scheduling policy: 1) If a multi-thread job is running, the scheduler won't run a CPU job if doing so would exceed #CPUs. 2) the scheduler will run GPU jobs until the CPU load is #CPUs+1 (but not beyond that). There is a rationale for both of these, thought they are both open to debate. -- David ------------------------------------------------------------------------------------ Date: Thu, 4 Apr 2013 13:48:52 -0700 From: davea@ssl.berkeley.edu To: jacob_w_klein@msn.com CC: boinc_alpha@ssl.berkeley.edu Subject: Re: [boinc_alpha] Using app_config.xml <cpu_usage>2</cpu_usageresults in underloading/overloading CPU Jacob: There are situations where overloading the CPUs, even slightly, can greatly reduce overall throughput. 1) multithread apps. Typically these are tightly coupled, meaning that each thread uses results recently computed by other threads, and if these are not available it has to wait. Multithread apps perform best if all the threads are running all the time. If 1 thread is descheduled (e.g. because there are 8.5 runnable threads on an 8-CPU system) this causes the other threads to sleep and wake up, which has an overhead. If this happens frequently it has a cascading effect and this can reduce the throughput of the app by a large factor (like 25%). 2) GPU apps. If the CPU part of a GPU app is descheduled, the GPU is idle until it's scheduled again. This can be on the order of a second, esp. on Windows. In such cases, throughput can be much higher e.g. with 7.5 threads than with 8.5 threads. So I'm reluctant to make any changes based on our collective intuition, because that intuition may not be correct. Ideally, some computer science graduate student would do a research project to study this issue with a range of real-world applications, and we could use the findings to improve our policies. However, the academic computer science world pretends that volunteer computing doesn't exist, so no such study has been done or is likely to get done. -- David ------------------------------------------------------------------------------------ |
 Jeffery M. Thompson Send message Joined: 23 Sep 12 Posts: 159 Credit: 16,977,106 RAC: 0   |
We are looking to get the wu.rsc_fpops_est to mitigate this issue. We are using the work units reported here and work units with large run times returned to the servers to see if we can get a better estimate to help with priority scheduling issues at the least. We may need more data to make this work and it may be that due to the nature of chaos in some physical systems modeled that there will always be estimates that are larger than the real run time. But on the average it should balance out. Also we do have valid work units that will have long run times because they have to calculate more. There are valid models where the particles spread out requiring more individual calculations. Knowing these cases ahead of the calculation would mean that we would not need to run the calculations we could throw them out before they were run. So there is a middle somewhere. And that is what I am working on. I believe getting a better estimate will help with the scheduler. And mitigate the issues. The other thing to look at is the reporting deadline. As our project is asynchronous, new work units are generated based on the early results from the earlier work units, thus timing is important. So I am trying to find a balance in this. The current form may be causing issues that need resolved. The calculations are done on the server side so there will not be a new binary released at this time. On the positive side it is returning good science, the results make sense for the problem being addressed. .Jeff |
|
Send message Joined: 4 Sep 12 Posts: 219 Credit: 456,474 RAC: 0
|
Richard, I have no query with policy (1) in that email - that seems to be working as designed and intended. But I think the current v7.0.64 scheduler is failing to obey policy (2) - the relationship between MT and GPU tasks. I've been following this one for quite a while: very few projects supply MT apps, so it doesn't get much attention in alpha testing - the ClientSched document you linked doesn't even mention MT jobs, let alone co-scheduling with GPUs, and the change from SVN to GIT has made old code changes unsearchable. But I'm pretty certain I had a hand in: ------------------------------------------------------------------------------------ Revision: 97ee3a38f265653d6b16bd5611df3ece4b2eef91 Author: David Anderson <davea@ssl.berkeley.edu> Date: 22/09/2009 00:23:40 Message: - client: tweak CPU scheduling policy to avoid running multithread apps overcommitted. Actually: allow overcommitment but only a fractional CPU (so that, e.g., we can run a GPU app and a 4-CPU app on a 4-CPU host) svn path=/trunk/boinc/; revision=19126 ---- Modified: checkin_notes Modified: client/cpu_sched.cpp ------------------------------------------------------------------------------------ At the moment, we can only do the highlighted bit some of the time. |
|
Send message Joined: 22 Jun 11 Posts: 32 Credit: 41,852,496 RAC: 0
|
But I think the current v7.0.64 scheduler is failing to obey policy (2) - the relationship between MT and GPU tasks. Interesting. Sorry, I was not aware of that 2009 checkin where the concession was made. Perhaps this will inspire me to take on an MT task to test exactly how it works. Are you saying that, if the MT task is scheduled first, and then GPU tasks are allowed to also be scheduled, then BOINC will schedule both [correctly]? And are you also saying that, if the GPU task is scheduled first, and then the MT task is allowed to also be scheduled, then BOINC will only schedule the GPU task [incorrectly]? If those statements are how it is currently behaving, and that inconsistency exists, then it does indeed sound like a BOINC scheduling bug. [Edit] I just tested with 7.1.15 alpha, and it appears that BOINC is correctly scheduling for me. If the 8-CPU MT task is started first, BOINC will allow 2 GPU tasks that each use 0.5 CPU. If the 2 GPU tasks are started first, BOINC will allow the 8-CPU MT task to also be scheduled. Am I trying to reproduce the bug incorrectly? |
|
Send message Joined: 26 May 11 Posts: 32 Credit: 43,959,896 RAC: 0
|
My observations, it appears to me that the more CPU cores you have the more "ok" it is to overcommit. For reference, I have 12 cores, and the MT tasks can run 2 GPU tasks quite well with total machine CPU loads at +/- 85% total CPU percent used. The scheduler does allow GPU tasks to load / execute while MT tasks are running, with the apparent exception of when a High Priority MT task changes to Normal priority. WHEN THIS HAPPENS, then the SCHEDULER stops the MT tasks puts the task in "wait" status and will not allow any other CPU based tasks to execute until ALL other work - INCLUDIONG ALL GPU tasks are complete, Including GPU tasks in QUE, which can mean if Boinc scheduler is obtaining more GPU tasks, CPU task will never commence. The Scheduler will not allow other any CPU work to start until the MT task is complete. Remember it is in "wait" state, waiting for 100% CPU availability. Essentially all 12 cores are available and only are supporting GPU work. Bottom line, my opinion is that the more cores you have, it is GOOD to allow overcommitting. This allows cores that are underperforming because they have to wait to contribute to GPU activity. I also agree that the Sum GPU CPU core requirements needs to be <= 1 CPU core OR perhaps <= .1 * # of cores (.1*12 cores = 1.2 provided that no GPU task requires a 1, a whole core). I would be glad to record or snapshot total CPU use on this computer with 12 cores. Please advise... There is a number of issues here, the most serious being that the MT task when it changes status from High Priority to normal, then goes into "Wait" and holds all remaining CPU tasks hostage. |
|
Send message Joined: 4 Sep 12 Posts: 219 Credit: 456,474 RAC: 0
|
There is a number of issues here, the most serious being that the MT task when it changes status from High Priority to normal, then goes into "Wait" and holds all remaining CPU tasks hostage. My observation too, and I think it ties in with my conversation with Jacob Klein. Back in 2009, when I was doing similar tests and having similar conversations with the 'AQUA' project, we were still using BOINC v6 versions with separate CPU and GPU scheduling, each resource maintaining a separate 'debt' level with respect to other projects. Now, I'm testing with BOINC v7.0.64, which maintains a single common 'REC' for each project, which is used to schedule both CPU and GPU tasks. I'm seeing this failure to run MT and GPU tasks together after a long - many hours - exclusive occupation of all CPUs by the MT task running in high priority. This means the REC for MW (as a whole) will be high, and runtime prio will be low: BOINC will be reluctant to schedule MW for a while to come. This may result in a different set of scheduling decisions from those seen by Jacob, who - with very little MW running under his belt - will be in the opposite 'low REC, high prio' state. I've seen MT + GPU running properly in that state, too. In order to help the bug-hunt here, here's the message log for the event I posted to boinc_alpha: 10/06/2013 13:53:11 | | [cpu_sched_debug] Request CPU reschedule: periodic CPU scheduling I'm assuming that, somewhere in the switchover from debt-based to REC-based scheduling, a code path was introduced which overlooks the 'allow minimal overcommits for GPUs' rule. It will be interesting to see if this has persisted into v 7.1.15 and later. I haven't seen any alterations in that area, but Jacob has been watching that area of code more closely than I have over the last few weeks. |
|
Send message Joined: 22 Jun 11 Posts: 32 Credit: 41,852,496 RAC: 0
|
Now, THAT is interesting. Look closer. The code has logic to keep a task set as "deadline miss - high priority"... if it has PREVIOUSLY been marked "high priority". I think that's the "thrashing prevention" message you see there. That may be key to getting into the scenario that has the problem. "Previously high priority, to the point where BOINC would keep it high priority to prevent thrashing... and now in a situation where it's no longer a deadline miss." So, serious question, what happens when this occurs on your system? Does it just run the GPU task, and leave the other 3.96 CPUs completely idle? |
|
Send message Joined: 26 May 11 Posts: 32 Credit: 43,959,896 RAC: 0
|
Richard, great information. Thank you. Now what happens? Do we have to wait for the next version of Boinc? Would a temporary band aid fix be to modify the MT task to be Number of available core processors - 1 (My case 12-1=11)? |
|
Send message Joined: 26 May 11 Posts: 32 Credit: 43,959,896 RAC: 0
|
So, serious question, what happens when this occurs on your system? Does it just run the GPU task, and leave the other 3.96 CPUs completely idle? On my system, all CPU cores other than those that support GPU's go idle, EVEN if there are other CPU tasks available to run. |
|
Send message Joined: 26 May 11 Posts: 32 Credit: 43,959,896 RAC: 0
|
FYI, I have the conditions currently executing so that in the next 30 minutes +/- I believe I will be in the "hung" position as being discussed... Advise if there is anyway I can assist.... |
|
Send message Joined: 4 Sep 12 Posts: 219 Credit: 456,474 RAC: 0
|
So, serious question, what happens when this occurs on your system? Does it just run the GPU task, and leave the other 3.96 CPUs completely idle? Yes. That was part of my first "extended run" MT task, and I wanted to finish it and report it to see what happened next - so (and since there only appeared to this human to be a few more minutes to run), I suspended all other CPU work hoping that the MT task would run. It didn't, until I suspend the GPU task as well. I'll have to run some more tasks - may be difficult this weekend - and get it into the same state with more debug log flags active - I think I'm going to need both priority and work_fetch, in addition to that cpu_sched. @ jdzukley, extended logging as I've just suggested would be helpful, if you're familiar with setting that up in time - but unfortunately, I've got to go out now, so won't be around to guide you. |
|
Send message Joined: 26 May 11 Posts: 32 Credit: 43,959,896 RAC: 0
|
Are these the log flags wanted: <log_flags> <coproc_debug>1</coproc_debug> <cpu_sched>1</cpu_sched> <cpu_sched_debug>1</cpu_sched_debug> <priority_debug>1</priority_debug> </log_flags> I have these running at the moment, advise if I should have others... |
|
Send message Joined: 26 May 11 Posts: 32 Credit: 43,959,896 RAC: 0
|
A snapshots of the debugs, first here is the job in execution: 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] scheduling de_separation_21_2s_sscon_1_1371083750_657599_0 (coprocessor job, FIFO) (prio -1.000000) 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] reserving 1.000000 of coproc NVIDIA 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] scheduling de_separation_79_DR8_rev_3_1371083750_657610_0 (coprocessor job, FIFO) (prio -1.020216) 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] reserving 1.000000 of coproc NVIDIA 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] scheduling ps_nbody_06_11_dark_4_1371083750_62813_0 (CPU job, priority order) (prio -1.040431) 6/14/2013 3:44:39 PM | | [cpu_sched_debug] enforce_schedule(): start 6/14/2013 3:44:39 PM | | [cpu_sched_debug] preliminary job list: 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] 0: de_separation_21_2s_sscon_1_1371083750_657599_0 (MD: no; UTS: yes) 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] 1: de_separation_79_DR8_rev_3_1371083750_657610_0 (MD: no; UTS: no) 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] 2: ps_nbody_06_11_dark_4_1371083750_62813_0 (MD: no; UTS: no) 6/14/2013 3:44:39 PM | | [cpu_sched_debug] final job list: 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] 0: de_separation_21_2s_sscon_1_1371083750_657599_0 (MD: no; UTS: yes) 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] 1: de_separation_79_DR8_rev_3_1371083750_657610_0 (MD: no; UTS: no) 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] 2: ps_nbody_06_11_dark_4_1371083750_62813_0 (MD: no; UTS: no) 6/14/2013 3:44:39 PM | Milkyway@Home | [coproc] NVIDIA instance 1: confirming for de_separation_21_2s_sscon_1_1371083750_657599_0 6/14/2013 3:44:39 PM | Milkyway@Home | [coproc] Assigning NVIDIA instance 0 to de_separation_79_DR8_rev_3_1371083750_657610_0 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] scheduling de_separation_21_2s_sscon_1_1371083750_657599_0 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] scheduling de_separation_79_DR8_rev_3_1371083750_657610_0 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] avoid MT overcommit: skipping ps_nbody_06_11_dark_4_1371083750_62813_0 6/14/2013 3:44:39 PM | | [cpu_sched_debug] using 0.79 out of 12 CPUs 6/14/2013 3:44:39 PM | climateprediction.net | [cpu_sched_debug] hadcm3n_o3e8_1940_40_008382078_1 sched state 1 next 1 task state 9 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] ps_nbody_06_11_dark_4_1371083750_62813_0 sched state 1 next 1 task state 9 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] de_separation_21_2s_sscon_1_1371083750_657599_0 sched state 2 next 2 task state 1 6/14/2013 3:44:39 PM | Milkyway@Home | [cpu_sched_debug] de_separation_79_DR8_rev_3_1371083750_657610_0 sched state 0 next 2 task state 0 6/14/2013 3:44:39 PM | Milkyway@Home | Starting task de_separation_79_DR8_rev_3_1371083750_657610_0 using milkyway version 102 (opencl_nvidia) in slot 0 6/14/2013 3:44:39 PM | | [cpu_sched_debug] enforce_schedule: end Now is the shot where the priority changed and the job was suspended: 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] scheduling de_separation_21_2s_sscon_1_1371083750_657599_0 (coprocessor job, FIFO) (prio -1.000000) 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] reserving 1.000000 of coproc NVIDIA 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] scheduling de_separation_79_DR8_rev_3_1371083750_657610_0 (coprocessor job, FIFO) (prio -1.020216) 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] reserving 1.000000 of coproc NVIDIA 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] scheduling ps_nbody_06_11_dark_4_1371083750_62813_0 (CPU job, priority order) (prio -1.040431) 6/14/2013 3:44:55 PM | | [cpu_sched_debug] enforce_schedule(): start 6/14/2013 3:44:55 PM | | [cpu_sched_debug] preliminary job list: 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] 0: de_separation_21_2s_sscon_1_1371083750_657599_0 (MD: no; UTS: yes) 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] 1: de_separation_79_DR8_rev_3_1371083750_657610_0 (MD: no; UTS: no) 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] 2: ps_nbody_06_11_dark_4_1371083750_62813_0 (MD: no; UTS: no) 6/14/2013 3:44:55 PM | | [cpu_sched_debug] final job list: 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] 0: de_separation_21_2s_sscon_1_1371083750_657599_0 (MD: no; UTS: yes) 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] 1: de_separation_79_DR8_rev_3_1371083750_657610_0 (MD: no; UTS: yes) 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] 2: ps_nbody_06_11_dark_4_1371083750_62813_0 (MD: no; UTS: no) 6/14/2013 3:44:55 PM | Milkyway@Home | [coproc] NVIDIA instance 1: confirming for de_separation_21_2s_sscon_1_1371083750_657599_0 6/14/2013 3:44:55 PM | Milkyway@Home | [coproc] NVIDIA instance 0: confirming for de_separation_79_DR8_rev_3_1371083750_657610_0 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] scheduling de_separation_21_2s_sscon_1_1371083750_657599_0 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] scheduling de_separation_79_DR8_rev_3_1371083750_657610_0 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] avoid MT overcommit: skipping ps_nbody_06_11_dark_4_1371083750_62813_0 6/14/2013 3:44:55 PM | | [cpu_sched_debug] using 0.79 out of 12 CPUs 6/14/2013 3:44:55 PM | climateprediction.net | [cpu_sched_debug] hadcm3n_o3e8_1940_40_008382078_1 sched state 1 next 1 task state 9 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] ps_nbody_06_11_dark_4_1371083750_62813_0 sched state 1 next 1 task state 9 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] de_separation_21_2s_sscon_1_1371083750_657599_0 sched state 2 next 2 task state 1 6/14/2013 3:44:55 PM | Milkyway@Home | [cpu_sched_debug] de_separation_79_DR8_rev_3_1371083750_657610_0 sched state 2 next 2 task state 1 6/14/2013 3:44:55 PM | | [cpu_sched_debug] enforce_schedule: end Unfortunately for this log capture, I did not have other CPU tasks ready for execution (the one job was suspended. Next, I allowed the 2 GPU tasks in execution to finish, and suspended all remaining GPU tasks que.. this is the log right after the last GPU task finished showing the start of the MT tasks: 6/14/2013 3:51:49 PM | Milkyway@Home | [cpu_sched_debug] scheduling ps_nbody_06_11_dark_4_1371083750_62813_0 (CPU job, priority order) (prio -1.000000) 6/14/2013 3:51:49 PM | | [cpu_sched_debug] enforce_schedule(): start 6/14/2013 3:51:49 PM | | [cpu_sched_debug] preliminary job list: 6/14/2013 3:51:49 PM | Milkyway@Home | [cpu_sched_debug] 0: ps_nbody_06_11_dark_4_1371083750_62813_0 (MD: no; UTS: no) 6/14/2013 3:51:49 PM | | [cpu_sched_debug] final job list: 6/14/2013 3:51:49 PM | Milkyway@Home | [cpu_sched_debug] 0: ps_nbody_06_11_dark_4_1371083750_62813_0 (MD: no; UTS: no) 6/14/2013 3:51:49 PM | Milkyway@Home | [cpu_sched_debug] scheduling ps_nbody_06_11_dark_4_1371083750_62813_0 6/14/2013 3:51:49 PM | climateprediction.net | [cpu_sched_debug] hadcm3n_o3e8_1940_40_008382078_1 sched state 1 next 1 task state 9 6/14/2013 3:51:49 PM | Milkyway@Home | [cpu_sched_debug] ps_nbody_06_11_dark_4_1371083750_62813_0 sched state 1 next 2 task state 9 6/14/2013 3:51:49 PM | Milkyway@Home | [cpu_sched] Resuming ps_nbody_06_11_dark_4_1371083750_62813_0 6/14/2013 3:51:49 PM | Milkyway@Home | Resuming task ps_nbody_06_11_dark_4_1371083750_62813_0 using milkyway_nbody version 118 (mt) in slot 3 6/14/2013 3:51:49 PM | | [cpu_sched_debug] enforce_schedule: end Advise if you folks want a copy of the full log file for this time frame. |
|
Send message Joined: 26 May 11 Posts: 32 Credit: 43,959,896 RAC: 0
|
To help avoid High Priority MT Tasks and the slew of problems presented below, consider a quick temporary fix on the task generator server to update the estimated run time for the task to be no greater than (required completion date & time - (now() + say 24hours)). This way the task most often should not become High Priority. estimated task hours = min (system generated estimated task hours, (required completion date - (now() + 24 hours))) And but, I do not understand why a task that start out as High Priority, after it become Normal Priority, at no time can a GPU task can run until the specific MT task has completed... Any other MT task will allow GPU's to come and go... Alternately, MT tasks can be set / sent to be # of processors - 1 and then this whole discussion thread is mute... The CC config file for Boinc probably needs an optional setting for how many cores can be made available for MT processing. My third long run High Priority MT task has just started (consecutively, no other short runs this afternoon after the first long started). I will be setting No New Tasks on Milkyway as I have better things to do on a Friday night than baby sit long run MT tasks that suspend most work on my system... |
Overtonesinger Send message Joined: 15 Feb 10 Posts: 63 Credit: 1,836,010 RAC: 0  
|
Interesting WU, which crashed the 1.18 (mt) Win-x64 application on my very very stable mobile Core i7 CPU "720QM": -1073741571 (0xffffffffc00000fd) Unknown error number link to WU ID 382354923 is: http://milkyway.cs.rpi.edu/milkyway/workunit.php?wuid=382354923 Err: [code]<core_client_version>7.0.64</core_client_version> <
©2024 Astroinformatics Group