Welcome to MilkyWay@home

N00b here. Multi-threading, scheduling and stuff I just don't understand

Message boards : Number crunching : N00b here. Multi-threading, scheduling and stuff I just don't understand
Message board moderation

To post messages, you must log in.

AuthorMessage
Bryan Price

Send message
Joined: 19 Apr 09
Posts: 2
Credit: 453,228
RAC: 3,896
Message 77192 - Posted: 26 Jul 2024, 15:32:45 UTC

I used to run BOINC for SETI years ago. I've gotten back into it, although SETI is now shutdown.

Current system is a 10th gen I7, 8 cores, 16 threads. 64GB memory.

I'm running Milkyway and Einstein now.

Milkyway sends me 16 thread units, which for some reason, only consume 8 threads. (shrug) And because it thinks it is using 16 threads, there are no threads for Einstein to run. Not a huge problem, I'm just not processing as fast as I could be.The problem I have is that when I don't get Milkyway units fast enough, Einstein, which had been pulling only GPU units for me to work, also picks up 16 one CPU threads, and starts those. Then, when Milkyway gives me units, BOINC proceeds to execute those, even though they have a later deadline than Einstein. And I've seen it where my Einstein units have been flushed, merely because they were past deadline. It probably wouldn't have happened if I had been away on vacation. With the Crowdstrike WTF hitting, evidently Milkyway stopped sending me units, but Einstein was still working, and BOINC (I'm guessing it's BOINCs doing) then got me CPU as well as GPU units to run. And since the deadline for those units that BOINC now refused to run, I suspended Milkyway, I stopped getting new units from Einstein, and let Einstein finish those units close to the deadline. I then unsuspended Milkyway, got some new units, and then set Einstein to get new units. Now that I've got a grasp on what I need to do to handle this manually, who do I need to talk to to change somebody's behavior so it shouldn't be a problem? Or if BOINC sees that there are still threads available, despite what it thinks it should be using, then process those other CPU units and get them out of the way? I guess the 16 threads using only 8 threads is something I can report here. Would BOINC be the place to see if they can run the CPU deadlines sooner than later? Things are now running with Milkyway giving me all CPU units and Einstein giving me GPU units to run.

I think this is my first post here. I certainly don't remember being on any message board for this. And I'm confused. And I really don't know who to talk to. And I'm not even sure I'm making sense. :/

Or is it a Shut up Bryan, just handle it manually.

Thanks.
ID: 77192 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3339
Credit: 524,010,781
RAC: 14
Message 77194 - Posted: 27 Jul 2024, 9:57:52 UTC - in response to Message 77192.  

I used to run BOINC for SETI years ago. I've gotten back into it, although SETI is now shutdown.

Current system is a 10th gen I7, 8 cores, 16 threads. 64GB memory.

I'm running Milkyway and Einstein now.

Milkyway sends me 16 thread units, which for some reason, only consume 8 threads. (shrug) And because it thinks it is using 16 threads, there are no threads for Einstein to run. Not a huge problem, I'm just not processing as fast as I could be.The problem I have is that when I don't get Milkyway units fast enough, Einstein, which had been pulling only GPU units for me to work, also picks up 16 one CPU threads, and starts those. Then, when Milkyway gives me units, BOINC proceeds to execute those, even though they have a later deadline than Einstein. And I've seen it where my Einstein units have been flushed, merely because they were past deadline. It probably wouldn't have happened if I had been away on vacation. With the Crowdstrike WTF hitting, evidently Milkyway stopped sending me units, but Einstein was still working, and BOINC (I'm guessing it's BOINCs doing) then got me CPU as well as GPU units to run. And since the deadline for those units that BOINC now refused to run, I suspended Milkyway, I stopped getting new units from Einstein, and let Einstein finish those units close to the deadline. I then unsuspended Milkyway, got some new units, and then set Einstein to get new units. Now that I've got a grasp on what I need to do to handle this manually, who do I need to talk to to change somebody's behavior so it shouldn't be a problem? Or if BOINC sees that there are still threads available, despite what it thinks it should be using, then process those other CPU units and get them out of the way? I guess the 16 threads using only 8 threads is something I can report here. Would BOINC be the place to see if they can run the CPU deadlines sooner than later? Things are now running with Milkyway giving me all CPU units and Einstein giving me GPU units to run.

I think this is my first post here. I certainly don't remember being on any message board for this. And I'm confused. And I really don't know who to talk to. And I'm not even sure I'm making sense. :/

Or is it a Shut up Bryan, just handle it manually.

Thanks.


The first thing is WELCOME BACK TO BOINC CRUNCHING!!

The 2nd thing is to stop getting the multi-task units and go to the single threaded tasks by setting in the Preferences venues to max number of cpu's to 1, that way both MilkyWay and Einstein can co-exist without battling for the same resources. The 2nd thing is to put an app_config.xml file in both the MilkyWay and Einstein folders in C:\Program Data\Boinc\Projects\

The app_config.file should look like this:
<app_config>
<project_max_concurrent>7</project_max_concurrent>
</app_config>

Use Notepad to copy and paste and save it in each folder, this will ensure both Einstein and MilkyWay each use 7 cpu cores and no more, ie they won't hog all the resources
not letting the other project run any tasks.

Next I would also check your cache settings in the Boinc Manager so that you are not using the default of 10 days for your settings, I use 1 day for the main cache size and then 0.5 days for the additional cache size. This usually ends up being about 2 days worth of tasks for me.

As for Einstein itself if you don't want gpu, or cpu, tasks then you need to uncheck them in the Preferences section there, it's here:
https://einsteinathome.org/account/prefs/project Be sure to ensure you are in the correct venue, ie default, home, school or work, when making any changes. Also if you decide to only get gpu tasks from Einstein, MilkyWay no longer has any gpu tasks, then be sure to change the app_config.xml file for Einstein so only 1 task runs at a time and the app_config.xml file at MilkyWay so it can do more cpu tasks.

Whatever you do though do not set the app config file to use all 16 cpu cores of your pc unless you also change the default setting about how much cpu time your pc uses or you won't be able to use your pc for anything because it will be always crunching and not listening for your mouse or keyboard clicks. Personally I always leave at least one cpu core free for the gpu to use, yes it needs to get the tasks info sent to it when it needs a new task and also needs it when it finishes the existing task.
ID: 77194 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Bryan Price

Send message
Joined: 19 Apr 09
Posts: 2
Credit: 453,228
RAC: 3,896
Message 77202 - Posted: 4 Aug 2024, 23:06:23 UTC - in response to Message 77194.  

The first thing is WELCOME BACK TO BOINC CRUNCHING!!

Thanks!

It seems that BOINC is actually smarter than I was giving it!

I managed to scoop up two Einstein CPU threads. I watched to see what BOINC was going to do with them. When it came to two days left before needing to be completed, BOINC ran both of them, while executing 16 thread units from MilkyWay. I don't know why it didn't do that sooner, but OK. By the time the units were done, I still had one day of room before they expired.

The 2nd thing is to stop getting the multi-task units and go to the single threaded tasks by setting in the Preferences venues to max number of cpu's to 1, that way both MilkyWay and Einstein can co-exist without battling for the same resources.

I don't see that under Computing Preferences. :/

The 2nd thing is to put an app_config.xml file in both the MilkyWay and Einstein folders in C:\Program Data\Boinc\Projects\

I will do that. And see what happens.

Next I would also check your cache settings in the Boinc Manager so that you are not using the default of 10 days for your settings,

The default was set at .25 days and up to .25 days of additional work. I bumped the default up to .50 days. I've got the room on the drive, so I'm not worried about that.

As for Einstein itself if you don't want gpu, or cpu, tasks then you need to uncheck them in the Preferences section there, it's here:
https://einsteinathome.org/account/prefs/project Be sure to ensure you are in the correct venue, ie default, home, school or work, when making any changes. Also if you decide to only get gpu tasks from Einstein, MilkyWay no longer has any gpu tasks, then be sure to change the app_config.xml file for Einstein so only 1 task runs at a time and the app_config.xml file at MilkyWay so it can do more cpu tasks.

I'm running local prefs for Einstein. I'm just running on one machine right now, so it doesn't really matter. I'll checkout the web prefs and figure out what I want to do.

Whatever you do though do not set the app config file to use all 16 cpu cores of your pc

I had zero problems with that when I was running Einstein by itself. Yeah, procexp was showing 100%, but it was responsive with both keyboard and mouse. This beast wasn't breathing hard regardless of what I did with it before, now it's actually using much more of its potential. And this one is a hand me down from my son. He got a faster system. But he now drools over the 4K monitor I have set up.Along with the second 1080p monitor that I have. I just won't talk about the lag as the 4K monitor actually takes its time responding when it changes modes (like turning HDR on or exiting the screensaver, which one would think wouldn't be that big of a deal, but it is).
ID: 77202 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3339
Credit: 524,010,781
RAC: 14
Message 77203 - Posted: 5 Aug 2024, 10:50:26 UTC - in response to Message 77202.  

The first thing is WELCOME BACK TO BOINC CRUNCHING!!

Thanks!

It seems that BOINC is actually smarter than I was giving it!

I managed to scoop up two Einstein CPU threads. I watched to see what BOINC was going to do with them. When it came to two days left before needing to be completed, BOINC ran both of them, while executing 16 thread units from MilkyWay. I don't know why it didn't do that sooner, but OK. By the time the units were done, I still had one day of room before they expired.

The 2nd thing is to stop getting the multi-task units and go to the single threaded tasks by setting in the Preferences venues to max number of cpu's to 1, that way both MilkyWay and Einstein can co-exist without battling for the same resources.

I don't see that under Computing Preferences. :/

The 2nd thing is to put an app_config.xml file in both the MilkyWay and Einstein folders in C:\Program Data\Boinc\Projects\

I will do that. And see what happens.

Next I would also check your cache settings in the Boinc Manager so that you are not using the default of 10 days for your settings,

The default was set at .25 days and up to .25 days of additional work. I bumped the default up to .50 days. I've got the room on the drive, so I'm not worried about that.

As for Einstein itself if you don't want gpu, or cpu, tasks then you need to uncheck them in the Preferences section there, it's here:
https://einsteinathome.org/account/prefs/project Be sure to ensure you are in the correct venue, ie default, home, school or work, when making any changes. Also if you decide to only get gpu tasks from Einstein, MilkyWay no longer has any gpu tasks, then be sure to change the app_config.xml file for Einstein so only 1 task runs at a time and the app_config.xml file at MilkyWay so it can do more cpu tasks.

I'm running local prefs for Einstein. I'm just running on one machine right now, so it doesn't really matter. I'll checkout the web prefs and figure out what I want to do.

Whatever you do though do not set the app config file to use all 16 cpu cores of your pc

I had zero problems with that when I was running Einstein by itself. Yeah, procexp was showing 100%, but it was responsive with both keyboard and mouse. This beast wasn't breathing hard regardless of what I did with it before, now it's actually using much more of its potential. And this one is a hand me down from my son. He got a faster system. But he now drools over the 4K monitor I have set up.Along with the second 1080p monitor that I have. I just won't talk about the lag as the 4K monitor actually takes its time responding when it changes modes (like turning HDR on or exiting the screensaver, which one would think wouldn't be that big of a deal, but it is).


Each Boinc Project uses different programmers to design their tasks, meaning your pc won't use the same resources to crunch with when going from project to project. Boinc normally runs tasks on a first in first out(fifo) basis but will change if you get into what it thinks is too close to a deadline but yours is doing exactly what it's supposed to do.

Changing your cache size is a good thing as it will allow you to get thru temporary outages that all projects have now and then. It will also allow you to get slightly bigger tasks if the project sends them out, ie Einstein did that a while back bundling more tasks into one downloaded task.

Changing the number of cpu's to use for each task is done with this line : Max # of threads for each MilkyWay@home task [__] at the bottom of this page:
https://milkyway.cs.rpi.edu/milkyway/prefs_edit.php?subset=project&tnow=1722854267&ttok=56d0048d7bc4ea87f2f79b560030661c

If you want to crunch faster turn the screen saver off and use the power button on the monitor instead, to run the screen saver requires cpu time, not alot but some, and it will cost you slightly in crunching time.

As for web versus local setting for Boinc I have 17 pc's running so always use local settings because what works for this pc doesn't work for that pc, with only one pc though you can do either. Just try not to do both as Boinc could get very confused and not do what you want it to do. When you into gpu crunching at Einstein, and some other Projects like PrimeGrid, I find it easier to use the website settings to do multiple tasks at the same time as they make it really easy.

I'm glad things are working well for you, if you have any other questions there is an active group of people that can help you.
ID: 77203 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Number crunching : N00b here. Multi-threading, scheduling and stuff I just don't understand

©2024 Astroinformatics Group