I recently saw that my GTS 240 GPU was processing a "Gamma-ray pulsar search #2 1.12 (FGRPopencl-nvidia)" task, but the GPU load was 6%. After restarting BOINC, the GPU load wouldn't go above 0%. The task is slowly progressing, but I don't think it's getting any help from the GPU.

I've seen 1 report in the nVidia Forums, on the 331.58 driver feedback thread, where users were complaining about OpenCL performance, but... my situation here seems different.

Is anyone else noticing OpenCL performance issues on the latest drivers, possibly on older GPUs?

Windows 8.1 x64

Time to test 331.65 drivers on that same task!

331.65 drivers are exhibiting the same behavior - only 0-12% GPU usage.

What I need to know is:
Is this an Albert application issue, an Albert task issue, or is this this an nVidia driver issue?

The task that completed successfully had a bunch of error info in the Std error output portion.

See:
http://albert.phys.uwm.edu/result.php?resultid=1179842

There were a lot of these lines:
Error in OpenCL context: CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_WRITE_BUFFER on GeForce GTS 240 (Device 0).
Error during OpenCL host->device transfer (error: -4)

Any ideas?

I wanted to chime in to mention that, today, I spent about 4 hours testing every released beta/whql driver version back to 314.22, on my Windows 8.1 x64 machine. I was testing OpenCL performance of the Albert@Home "Gamma-ray pulsar search #2 1.12 (FGRPopencl-nvidia)" task, on my GTS 240, for each driver version.

The performance results are:
331.65: Bad
331.58: Bad
331.40: Bad
327.23: Very Bad
326.80: Very Bad
326.41: Very Bad
326.19: Very Bad
320.49: Very Bad
320.18: Very Bad
320.14: Very Bad
320.00: Very Bad
314.22: Very Bad

... where "Bad" means GPU Load % fluctuating between 0-25%, and "Very Bad" means GPU Load % fluctuating between 0-9%.

So, if this is a regression, it's not recent. It's entirely possible that my issue is with the task itself, and not within the drivers.

Thoughts?

Thanks,
Jacob

A GTS 240 GPU (G92a or G92b chip, depending on version) is a very old and slow GPU (Q4 2009).

Most of the available performance will have been squeezed out of those years ago. Newer drivers will still be trying to leverage more performance out of:

GFxxx (Fermi) chips - 2010 onwards
GKxxx (Kepler) chips - 2012 onwards
GK110 (Titan-class) chips - 2013 onwards

And remember the huge driver architecture changes between Windows XP and Vista/7/8 (WDDM model).

Your GTS 240 would probably be happiest with Windows XP and a legacy driver - but in that system (and depending on the application - wait for the CUDA 6 tests), even a baby Kepler should still be showing improvement as the drivers are refined.

Let me put it another way:

What is the expected behavior of an "FGRPopencl-nvidia" Albert@Home task... on my GTX 660 Ti? What GPU Load % should I expect on that beefy GPU?

Here is what I'm currently seeing on the 331.65 drivers (monitoring with eVGA Precision-X at a 100ms polling interval):
GTX 660 Ti: Super-quick flickers, going from 0% to 33% back to 0%, about 4 times a second.
GTX 460: 23% most of the time, but brief surges downward to 15% for usually less than 2 seconds.
GTS 240: Bouncing around about once a second, between being at 13% and being at 26%.

Does this sound like correct behavior for those architectures, for that task type? :)

My CUDA tasks are usually 90% constant on the GPUs, which is why I thought the GPU Loads reported here looked suspicious. (And yes, I know CUDA is way different than OpenCL, but... are the low loads in this post really the expected loads for this task type?)

I did some testing a little while back and determined that a 660Ti Win7 x64 needs to run 4 concurrent tasks w/ 1 cpu core for each GPU task to keep properly fed. I forget how long they took but IIRC they only garner 70 pts. each ... call me a pt hore but I'm crunch other Einstein\ Albert GPU tasks until we see something new here.

I only have selected Perseus ARM only but occasionally get an FRGP.
Is this by design or an oddity of running beta?

Run only the selected applications
Binary Radio Pulsar Search: no
Binary Radio Pulsar Search (Arecibo, GPU): no
Binary Radio Pulsar Search (single DM): no
Binary Radio Pulsar Search (Perseus Arm Survey): yes
Gravitational Wave S6 Directed Search (CasA): no
Gamma-ray pulsar search #2: no
 
Run beta/test application versions?
This helps us develop applications, but it may cause jobs to fail on your computer: no 
Run CPU versions of applications for which GPU versions are available: no

TIA,
Steve

When this happens, getting the wrong kind of work, see if you can get the server log for that contact. To do this go to your list of computers and then in the rightmost column click on the datestamp for the last contact to display the log, post it here and maybe that can shed some light on why you are allocated work for an application you have not selected.

---

I apologize in advance for the wall of text about to ensue as I'm not sure what precisely is relevant and there are references from the beginning to the end for 2 GAMMA WU buried in this. I do move hardware between rigs and I have aborted a couple of GAMMA WUs in the past. I did let these 2 run with mixed results. The first ran on a 7950 concurrently with 2 Milkyway WUS and experienced a computation error which is likely more to do with my card OC than anything else. The second ran to completion on a 7850 with an additional concurrent Perseus Arm task and is now pending validation. One last side note, the message board removes duplicate space characters but in the original the 31st line there was only a single space before the [CRITICAL] tag, all other lines have 4 spaces (likely a tab).

I'm not concerned about getting unselected gamma WU as I'll crunch what I get, I promise no more aborts, but maybe something in the scheduler needs a tweak?

- Steve

2013-11-29 11:21:39.5693 [PID=11032]   Request: [USER#xxxxx] [HOST#9649] [IP xxx.xxx.xxx.139] client 7.2.5
2013-11-29 11:21:39.5705 [PID=11032]    [send] Not using matchmaker scheduling; Not using EDF sim
2013-11-29 11:21:39.5705 [PID=11032]    [send] CPU: req 0.00 sec, 0.00 instances; est delay 0.00
2013-11-29 11:21:39.5705 [PID=11032]    [send] ATI: req 159130.28 sec, 0.00 instances; est delay 0.00
2013-11-29 11:21:39.5705 [PID=11032]    [send] work_req_seconds: 0.00 secs
2013-11-29 11:21:39.5705 [PID=11032]    [send] available disk 95.30 GB, work_buf_min 172800
2013-11-29 11:21:39.5705 [PID=11032]    [send] active_frac 0.997831 on_frac 0.974682
2013-11-29 11:21:39.5706 [PID=11032]    [send] p_vm_extensions_disabled: no
2013-11-29 11:21:39.5706 [PID=11032]    [send] CPU features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss htt tm pni ssse3 cx16 sse4_1 sse4_2 popcnt aes syscall lm vmx tm2 pbe
2013-11-29 11:21:39.5743 [PID=11032]    [send] [HOST#9649] app version 728 is reliable
2013-11-29 11:21:39.5743 [PID=11032]    [send] set_trust: random choice for cons valid 15: yes
2013-11-29 11:21:39.5743 [PID=11032]    [send] [AV#729] not reliable; cons valid 7 < 10
2013-11-29 11:21:39.5743 [PID=11032]    [send] set_trust: cons valid 7 < 10, don't use single replication
2013-11-29 11:21:39.5743 [PID=11032]    [send] [AV#768] not reliable; cons valid 0 < 10
2013-11-29 11:21:39.5743 [PID=11032]    [send] set_trust: cons valid 0 < 10, don't use single replication
2013-11-29 11:21:39.5743 [PID=11032]    [mixed] sending locality work first
2013-11-29 11:21:39.5744 [PID=11032]    [locality] [HOST#9649] removing file rand_PAS.bank.v3 from file_infos list
2013-11-29 11:21:39.5744 [PID=11032]    [locality] [HOST#9649] removing file JPLEPH.405 from file_infos list
2013-11-29 11:21:39.6231 [PID=11032]    [version] get_app_version(): getting app version for WU#503247 (LATeah0069U_48.0_500_-4.01e-10) appid:25
2013-11-29 11:21:39.6232 [PID=11032]    [version] looking for version of hsgamma_FGRP2
2013-11-29 11:21:39.6232 [PID=11032]    [version] Checking plan class 'FGRPopencl-ati'
2013-11-29 11:21:39.6241 [PID=11032]    [version] reading plan classes from file '/BOINC/projects/AlbertAtHome/plan_class_spec.xml'
2013-11-29 11:21:39.6241 [PID=11032]    [version] host_flops: 4.242098e+09, 	speedup: 3.00, 	projected_flops: 1.202914e+10, 	peak_flops: 4.242098e+09, 	peak_flops_factor: 0.21
2013-11-29 11:21:39.6242 [PID=11032]    [version] Checking plan class 'FGRPopencl-nvidia'
2013-11-29 11:21:39.6242 [PID=11032]    [version] No NVidia devices found
2013-11-29 11:21:39.6242 [PID=11032]    [version] [AV#766] app_plan() returned false
2013-11-29 11:21:39.6242 [PID=11032]    [version] [AV#768] (FGRPopencl-ati) adjusting projected flops based on PFC avg: 9.63G
2013-11-29 11:21:39.6242 [PID=11032]    [version] Best version of app hsgamma_FGRP2 is [AV#768] (9.63 GFLOPS)
2013-11-29 11:21:39.6242 [PID=11032]    [send] est delay 0, skipping deadline check
2013-11-29 11:21:39.6274 [PID=11032]    [send] Sending app_version hsgamma_FGRP2 7 112 FGRPopencl-ati; projected 9.63 GFLOPS
2013-11-29 11:21:39.6275 [PID=11032] [CRITICAL]   No filename found in [WU#503247 LATeah0069U_48.0_500_-4.01e-10]
2013-11-29 11:21:39.6275 [PID=11032]    [send] est. duration for WU 503247: unscaled 1557.59 scaled 1601.52
2013-11-29 11:21:39.6275 [PID=11032]    [HOST#9649] Sending [RESULT#1212496 LATeah0069U_48.0_500_-4.01e-10_1] (est. dur. 1601.52s (0h26m41s52)) (max time 31151.75s (8h39m11s74))
2013-11-29 11:21:39.6300 [PID=11032]    [locality] send_old_work(LATeah0069U_48.0_500_-4.01e-10_1) sent result created 347.4 hours ago [RESULT#1212496]
2013-11-29 11:21:39.6300 [PID=11032]    [locality] Note: sent NON-LOCALITY result LATeah0069U_48.0_500_-4.01e-10_1
2013-11-29 11:21:39.6300 [PID=11032]    [locality] send_new_file_work(): try to send old work
2013-11-29 11:21:39.6312 [PID=11032]    [version] get_app_version(): getting app version for WU#509715 (h1_0999.40_S6Direct__S6CasAf40_999.75Hz_277) appid:28
2013-11-29 11:21:39.6312 [PID=11032]    [version] looking for version of einstein_S6CasA
2013-11-29 11:21:39.6312 [PID=11032]    [version] Checking plan class 'SSE2'
2013-11-29 11:21:39.6312 [PID=11032]    [version] host_flops: 4.242098e+09, 	speedup: 1.60, 	projected_flops: 6.446067e+09, 	peak_flops: 4.028792e+09, 	peak_flops_factor: 1.00
2013-11-29 11:21:39.6312 [PID=11032]    [version] [AV#707] Skipping CPU version - user prefs say no CPUs
2013-11-29 11:21:39.6312 [PID=11032]    [version] returning NULL; platforms:
2013-11-29 11:21:39.6312 [PID=11032]    [version] windows_x86_64
2013-11-29 11:21:39.6312 [PID=11032]    [version] windows_intelx86
2013-11-29 11:21:39.6313 [PID=11032]    [mixed] sending non-locality work second
2013-11-29 11:21:39.6373 [PID=11032]    [version] get_app_version(): getting app version for WU#510419 (PA0087_002B1_366) appid:27
2013-11-29 11:21:39.6374 [PID=11032]    [version] looking for version of einsteinbinary_BRP5
2013-11-29 11:21:39.6374 [PID=11032]    [version] Checking plan class 'BRP5-opencl-ati'
2013-11-29 11:21:39.6374 [PID=11032]    [version] parsed project prefs setting 'gpu_util_brp' : true : 0.500000
2013-11-29 11:21:39.6374 [PID=11032]    [version] host_flops: 4.242098e+09, 	speedup: 15.00, 	projected_flops: 5.665771e+10, 	peak_flops: 4.242098e+09, 	peak_flops_factor: 0.21
2013-11-29 11:21:39.6374 [PID=11032]    [version] Checking plan class 'BRP5-opencl-intel_gpu'
2013-11-29 11:21:39.6374 [PID=11032]    [version] parsed project prefs setting 'gpu_util_brp' : true : 0.500000
2013-11-29 11:21:39.6375 [PID=11032]    [version] host_flops: 4.242098e+09, 	speedup: 15.00, 	projected_flops: 5.665771e+10, 	peak_flops: 4.242098e+09, 	peak_flops_factor: 0.21
2013-11-29 11:21:39.6375 [PID=11032]    [version] [AV#729] (BRP5-opencl-intel_gpu) setting projected flops based on host elapsed time avg: 40.00G
2013-11-29 11:21:39.6375 [PID=11032]    [version] Best version of app einsteinbinary_BRP5 is [AV#729] (40.00 GFLOPS)
2013-11-29 11:21:39.6375 [PID=11032]    [send] est. duration for WU 510419: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.6375 [PID=11032]    [send] [WU#510419] meets deadline: 800.76 + 11565.94 < 1209600
2013-11-29 11:21:39.6416 [PID=11032]    [send] Sending app_version einsteinbinary_BRP5 7 139 BRP5-opencl-intel_gpu; projected 40.00 GFLOPS
2013-11-29 11:21:39.6418 [PID=11032]    [send] est. duration for WU 510419: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.6418 [PID=11032]    [HOST#9649] Sending [RESULT#1230913 PA0087_002B1_366_1] (est. dur. 11565.94s (3h12m45s93)) (max time 224973.24s (62h29m33s23))
2013-11-29 11:21:39.6422 [PID=11032]    [version] get_app_version(): getting app version for WU#510036 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2496) appid:29
2013-11-29 11:21:39.6422 [PID=11032]    [version] looking for version of einsteinbinary_BRP4G
2013-11-29 11:21:39.6422 [PID=11032]    [version] Checking plan class 'BRP4G-opencl-ati'
2013-11-29 11:21:39.6423 [PID=11032]    [version] parsed project prefs setting 'gpu_util_brp' : true : 0.500000
2013-11-29 11:21:39.6423 [PID=11032]    [version] host_flops: 4.242098e+09, 	speedup: 15.00, 	projected_flops: 5.665771e+10, 	peak_flops: 4.242098e+09, 	peak_flops_factor: 0.21
2013-11-29 11:21:39.6423 [PID=11032]    [version] [AV#721] (BRP4G-opencl-ati) adjusting projected flops based on PFC avg: 72.94G
2013-11-29 11:21:39.6423 [PID=11032]    [version] Best version of app einsteinbinary_BRP4G is [AV#721] (72.94 GFLOPS)
2013-11-29 11:21:39.6423 [PID=11032]    [version] get_app_version(): getting app version for WU#505541 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_135) appid:21
2013-11-29 11:21:39.6423 [PID=11032]    [version] looking for version of einsteinbinary_BRP4
2013-11-29 11:21:39.6423 [PID=11032]    [version] Checking plan class 'BRP4X64'
2013-11-29 11:21:39.6424 [PID=11032]    [version] parsed project prefs setting 'also_run_cpu' : true : 1.000000
2013-11-29 11:21:39.6424 [PID=11032]    [version] project prefs setting 'also_run_cpu' (1.000000) prevents using plan class.
2013-11-29 11:21:39.6424 [PID=11032]    [version] [AV#588] app_plan() returned false
2013-11-29 11:21:39.6424 [PID=11032]    [version] Checking plan class 'BRP4SSE'
2013-11-29 11:21:39.6424 [PID=11032]    [version] parsed project prefs setting 'also_run_cpu' : true : 1.000000
2013-11-29 11:21:39.6424 [PID=11032]    [version] project prefs setting 'also_run_cpu' (1.000000) prevents using plan class.
2013-11-29 11:21:39.6424 [PID=11032]    [version] [AV#598] app_plan() returned false
2013-11-29 11:21:39.6424 [PID=11032]    [version] returning NULL; platforms:
2013-11-29 11:21:39.6424 [PID=11032]    [version] windows_x86_64
2013-11-29 11:21:39.6424 [PID=11032]    [version] windows_intelx86
2013-11-29 11:21:39.6424 [PID=11032]    [version] get_app_version(): getting app version for WU#505544 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_138) appid:21
2013-11-29 11:21:39.6425 [PID=11032]    [version] get_app_version(): getting app version for WU#503250 (LATeah0069U_48.0_500_-4.04e-10) appid:25
2013-11-29 11:21:39.6425 [PID=11032]    [version] looking for version of hsgamma_FGRP2
2013-11-29 11:21:39.6425 [PID=11032]    [version] Checking plan class 'FGRPopencl-ati'
2013-11-29 11:21:39.6425 [PID=11032]    [version] host_flops: 4.242098e+09, 	speedup: 3.00, 	projected_flops: 1.202914e+10, 	peak_flops: 4.242098e+09, 	peak_flops_factor: 0.21
2013-11-29 11:21:39.6425 [PID=11032]    [version] Checking plan class 'FGRPopencl-nvidia'
2013-11-29 11:21:39.6425 [PID=11032]    [version] No NVidia devices found
2013-11-29 11:21:39.6425 [PID=11032]    [version] [AV#766] app_plan() returned false
2013-11-29 11:21:39.6425 [PID=11032]    [version] [AV#768] (FGRPopencl-ati) adjusting projected flops based on PFC avg: 9.63G
2013-11-29 11:21:39.6425 [PID=11032]    [version] Best version of app hsgamma_FGRP2 is [AV#768] (9.63 GFLOPS)
2013-11-29 11:21:39.6426 [PID=11032]    [version] get_app_version(): getting app version for WU#510430 (PA0087_002B1_388) appid:27
2013-11-29 11:21:39.6426 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.6426 [PID=11032]    [send] est. duration for WU 510430: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.6426 [PID=11032]    [send] [WU#510430] meets deadline: 3692.25 + 11565.94 < 1209600
2013-11-29 11:21:39.6456 [PID=11032]    [send] Sending app_version einsteinbinary_BRP5 7 139 BRP5-opencl-intel_gpu; projected 40.00 GFLOPS
2013-11-29 11:21:39.6459 [PID=11032]    [send] est. duration for WU 510430: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.6459 [PID=11032]    [HOST#9649] Sending [RESULT#1230934 PA0087_002B1_388_0] (est. dur. 11565.94s (3h12m45s93)) (max time 224973.24s (62h29m33s23))
2013-11-29 11:21:39.6465 [PID=11032]    [version] get_app_version(): getting app version for WU#510030 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2400) appid:29
2013-11-29 11:21:39.6465 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.6465 [PID=11032]    [version] get_app_version(): getting app version for WU#505536 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_130) appid:21
2013-11-29 11:21:39.6465 [PID=11032]    [version] get_app_version(): getting app version for WU#505545 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_139) appid:21
2013-11-29 11:21:39.6466 [PID=11032]    [version] get_app_version(): getting app version for WU#503251 (LATeah0069U_48.0_500_-4.05e-10) appid:25
2013-11-29 11:21:39.6466 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.6466 [PID=11032]    [version] get_app_version(): getting app version for WU#510426 (PA0087_002B1_380) appid:27
2013-11-29 11:21:39.6466 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.6466 [PID=11032]    [send] est. duration for WU 510426: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.6466 [PID=11032]    [send] [WU#510426] meets deadline: 6583.73 + 11565.94 < 1209600
2013-11-29 11:21:39.6469 [PID=11032]    [send] [USER#346066] already has 1 result(s) for [WU#510426]
2013-11-29 11:21:39.6470 [PID=11032]    [version] get_app_version(): getting app version for WU#510031 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2416) appid:29
2013-11-29 11:21:39.6470 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.6470 [PID=11032]    [version] get_app_version(): getting app version for WU#505537 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_131) appid:21
2013-11-29 11:21:39.6470 [PID=11032]    [version] get_app_version(): getting app version for WU#505545 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_139) appid:21
2013-11-29 11:21:39.6471 [PID=11032]    [version] get_app_version(): getting app version for WU#503251 (LATeah0069U_48.0_500_-4.05e-10) appid:25
2013-11-29 11:21:39.6471 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.6471 [PID=11032]    [version] get_app_version(): getting app version for WU#510421 (PA0087_002B1_370) appid:27
2013-11-29 11:21:39.6471 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.6471 [PID=11032]    [send] est. duration for WU 510421: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.6471 [PID=11032]    [send] [WU#510421] meets deadline: 6583.73 + 11565.94 < 1209600
2013-11-29 11:21:39.6496 [PID=11032]    [send] Sending app_version einsteinbinary_BRP5 7 139 BRP5-opencl-intel_gpu; projected 40.00 GFLOPS
2013-11-29 11:21:39.6497 [PID=11032]    [send] est. duration for WU 510421: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.6497 [PID=11032]    [HOST#9649] Sending [RESULT#1230917 PA0087_002B1_370_1] (est. dur. 11565.94s (3h12m45s93)) (max time 224973.24s (62h29m33s23))
2013-11-29 11:21:39.6499 [PID=11032]    [version] get_app_version(): getting app version for WU#510031 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2416) appid:29
2013-11-29 11:21:39.6499 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.6499 [PID=11032]    [version] get_app_version(): getting app version for WU#505538 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_132) appid:21
2013-11-29 11:21:39.6500 [PID=11032]    [version] get_app_version(): getting app version for WU#505546 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_140) appid:21
2013-11-29 11:21:39.6500 [PID=11032]    [version] get_app_version(): getting app version for WU#503252 (LATeah0069U_48.0_500_-4.06e-10) appid:25
2013-11-29 11:21:39.6500 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.6500 [PID=11032]    [version] get_app_version(): getting app version for WU#510421 (PA0087_002B1_370) appid:27
2013-11-29 11:21:39.6501 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.6501 [PID=11032]    [send] [HOST#9649] [WU#510421 PA0087_002B1_370] WU is infeasible: Already in reply
2013-11-29 11:21:39.6501 [PID=11032]    [version] get_app_version(): getting app version for WU#510037 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2512) appid:29
2013-11-29 11:21:39.6501 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.6501 [PID=11032]    [version] get_app_version(): getting app version for WU#505539 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_133) appid:21
2013-11-29 11:21:39.6501 [PID=11032]    [version] get_app_version(): getting app version for WU#505546 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_140) appid:21
2013-11-29 11:21:39.6502 [PID=11032]    [version] get_app_version(): getting app version for WU#503252 (LATeah0069U_48.0_500_-4.06e-10) appid:25
2013-11-29 11:21:39.6502 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.6502 [PID=11032]    [version] get_app_version(): getting app version for WU#510430 (PA0087_002B1_388) appid:27
2013-11-29 11:21:39.6502 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.6502 [PID=11032]    [version] get_app_version(): getting app version for WU#510032 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2432) appid:29
2013-11-29 11:21:39.6503 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.6503 [PID=11032]    [version] get_app_version(): getting app version for WU#505547 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_141) appid:21
2013-11-29 11:21:39.6503 [PID=11032]    [version] get_app_version(): getting app version for WU#505547 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_141) appid:21
2013-11-29 11:21:39.6504 [PID=11032]    [version] get_app_version(): getting app version for WU#503253 (LATeah0069U_48.0_500_-4.07e-10) appid:25
2013-11-29 11:21:39.6504 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.6504 [PID=11032]    [version] get_app_version(): getting app version for WU#510422 (PA0087_002B1_372) appid:27
2013-11-29 11:21:39.6504 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.6504 [PID=11032]    [send] est. duration for WU 510422: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.6504 [PID=11032]    [send] [WU#510422] meets deadline: 9475.21 + 11565.94 < 1209600
2013-11-29 11:21:39.7796 [PID=11032]    [send] Sending app_version einsteinbinary_BRP5 7 139 BRP5-opencl-intel_gpu; projected 40.00 GFLOPS
2013-11-29 11:21:39.7800 [PID=11032]    [send] est. duration for WU 510422: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7800 [PID=11032]    [HOST#9649] Sending [RESULT#1230918 PA0087_002B1_372_0] (est. dur. 11565.94s (3h12m45s93)) (max time 224973.24s (62h29m33s23))
2013-11-29 11:21:39.7810 [PID=11032]    [version] get_app_version(): getting app version for WU#510036 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2496) appid:29
2013-11-29 11:21:39.7811 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.7811 [PID=11032]    [version] get_app_version(): getting app version for WU#505548 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_142) appid:21
2013-11-29 11:21:39.7811 [PID=11032]    [version] get_app_version(): getting app version for WU#505548 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_142) appid:21
2013-11-29 11:21:39.7811 [PID=11032]    [version] get_app_version(): getting app version for WU#503253 (LATeah0069U_48.0_500_-4.07e-10) appid:25
2013-11-29 11:21:39.7811 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.7812 [PID=11032]    [version] get_app_version(): getting app version for WU#510411 (PA0087_002B1_350) appid:27
2013-11-29 11:21:39.7812 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.7812 [PID=11032]    [send] est. duration for WU 510411: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7812 [PID=11032]    [send] [WU#510411] meets deadline: 12366.70 + 11565.94 < 1209600
2013-11-29 11:21:39.7820 [PID=11032]    [RESULT#1230897] expected to be unsent; instead, state is 4
2013-11-29 11:21:39.7821 [PID=11032]    [version] get_app_version(): getting app version for WU#510032 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2432) appid:29
2013-11-29 11:21:39.7821 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.7821 [PID=11032]    [version] get_app_version(): getting app version for WU#505549 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_143) appid:21
2013-11-29 11:21:39.7821 [PID=11032]    [version] get_app_version(): getting app version for WU#505549 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_143) appid:21
2013-11-29 11:21:39.7822 [PID=11032]    [version] get_app_version(): getting app version for WU#503248 (LATeah0069U_48.0_500_-4.02e-10) appid:25
2013-11-29 11:21:39.7822 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.7822 [PID=11032]    [version] get_app_version(): getting app version for WU#510428 (PA0087_002B1_384) appid:27
2013-11-29 11:21:39.7822 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.7822 [PID=11032]    [send] est. duration for WU 510428: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7822 [PID=11032]    [send] [WU#510428] meets deadline: 12366.70 + 11565.94 < 1209600
2013-11-29 11:21:39.7843 [PID=11032]    [send] Sending app_version einsteinbinary_BRP5 7 139 BRP5-opencl-intel_gpu; projected 40.00 GFLOPS
2013-11-29 11:21:39.7845 [PID=11032]    [send] est. duration for WU 510428: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7845 [PID=11032]    [HOST#9649] Sending [RESULT#1230931 PA0087_002B1_384_1] (est. dur. 11565.94s (3h12m45s93)) (max time 224973.24s (62h29m33s23))
2013-11-29 11:21:39.7847 [PID=11032]    [version] get_app_version(): getting app version for WU#510035 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2480) appid:29
2013-11-29 11:21:39.7847 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.7847 [PID=11032]    [version] get_app_version(): getting app version for WU#505550 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_144) appid:21
2013-11-29 11:21:39.7847 [PID=11032]    [version] get_app_version(): getting app version for WU#505550 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_144) appid:21
2013-11-29 11:21:39.7847 [PID=11032]    [version] get_app_version(): getting app version for WU#503254 (LATeah0069U_48.0_500_-4.08e-10) appid:25
2013-11-29 11:21:39.7847 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.7848 [PID=11032]    [version] get_app_version(): getting app version for WU#510429 (PA0087_002B1_386) appid:27
2013-11-29 11:21:39.7848 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.7848 [PID=11032]    [send] est. duration for WU 510429: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7848 [PID=11032]    [send] [WU#510429] meets deadline: 15258.18 + 11565.94 < 1209600
2013-11-29 11:21:39.7850 [PID=11032]    [send] [USER#346066] already has 1 result(s) for [WU#510429]
2013-11-29 11:21:39.7851 [PID=11032]    [version] get_app_version(): getting app version for WU#510033 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2448) appid:29
2013-11-29 11:21:39.7851 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.7851 [PID=11032]    [version] get_app_version(): getting app version for WU#505551 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_145) appid:21
2013-11-29 11:21:39.7851 [PID=11032]    [version] get_app_version(): getting app version for WU#505551 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_145) appid:21
2013-11-29 11:21:39.7852 [PID=11032]    [version] get_app_version(): getting app version for WU#503254 (LATeah0069U_48.0_500_-4.08e-10) appid:25
2013-11-29 11:21:39.7852 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.7852 [PID=11032]    [version] get_app_version(): getting app version for WU#510427 (PA0087_002B1_382) appid:27
2013-11-29 11:21:39.7852 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.7852 [PID=11032]    [send] est. duration for WU 510427: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7852 [PID=11032]    [send] [WU#510427] meets deadline: 15258.18 + 11565.94 < 1209600
2013-11-29 11:21:39.7871 [PID=11032]    [send] Sending app_version einsteinbinary_BRP5 7 139 BRP5-opencl-intel_gpu; projected 40.00 GFLOPS
2013-11-29 11:21:39.7873 [PID=11032]    [send] est. duration for WU 510427: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7873 [PID=11032]    [HOST#9649] Sending [RESULT#1230929 PA0087_002B1_382_1] (est. dur. 11565.94s (3h12m45s93)) (max time 224973.24s (62h29m33s23))
2013-11-29 11:21:39.7875 [PID=11032]    [version] get_app_version(): getting app version for WU#510028 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2368) appid:29
2013-11-29 11:21:39.7875 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.7875 [PID=11032]    [version] get_app_version(): getting app version for WU#505552 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_146) appid:21
2013-11-29 11:21:39.7875 [PID=11032]    [version] get_app_version(): getting app version for WU#505518 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_112) appid:21
2013-11-29 11:21:39.7875 [PID=11032]    [version] get_app_version(): getting app version for WU#503229 (LATeah0069U_48.0_500_-3.83e-10) appid:25
2013-11-29 11:21:39.7876 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.7876 [PID=11032]    [version] get_app_version(): getting app version for WU#510424 (PA0087_002B1_376) appid:27
2013-11-29 11:21:39.7876 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.7876 [PID=11032]    [send] est. duration for WU 510424: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7876 [PID=11032]    [send] [WU#510424] meets deadline: 18149.67 + 11565.94 < 1209600
2013-11-29 11:21:39.7897 [PID=11032]    [send] Sending app_version einsteinbinary_BRP5 7 139 BRP5-opencl-intel_gpu; projected 40.00 GFLOPS
2013-11-29 11:21:39.7898 [PID=11032]    [send] est. duration for WU 510424: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7899 [PID=11032]    [HOST#9649] Sending [RESULT#1230923 PA0087_002B1_376_1] (est. dur. 11565.94s (3h12m45s93)) (max time 224973.24s (62h29m33s23))
2013-11-29 11:21:39.7901 [PID=11032]    [version] get_app_version(): getting app version for WU#510017 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2192) appid:29
2013-11-29 11:21:39.7901 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.7901 [PID=11032]    [version] get_app_version(): getting app version for WU#505518 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_112) appid:21
2013-11-29 11:21:39.7901 [PID=11032]    [version] get_app_version(): getting app version for WU#505510 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_104) appid:21
2013-11-29 11:21:39.7901 [PID=11032]    [version] get_app_version(): getting app version for WU#503229 (LATeah0069U_48.0_500_-3.83e-10) appid:25
2013-11-29 11:21:39.7901 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.7902 [PID=11032]    [version] get_app_version(): getting app version for WU#510514 (PA0080_024D1_2) appid:27
2013-11-29 11:21:39.7902 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.7902 [PID=11032]    [send] est. duration for WU 510514: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.7902 [PID=11032]    [send] [WU#510514] meets deadline: 21041.15 + 11565.94 < 1209600
2013-11-29 11:21:39.9290 [PID=11032]    [send] Sending app_version einsteinbinary_BRP5 7 139 BRP5-opencl-intel_gpu; projected 40.00 GFLOPS
2013-11-29 11:21:39.9298 [PID=11032]    [send] est. duration for WU 510514: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.9298 [PID=11032]    [HOST#9649] Sending [RESULT#1232397 PA0080_024D1_2_0] (est. dur. 11565.94s (3h12m45s93)) (max time 224973.24s (62h29m33s23))
2013-11-29 11:21:39.9315 [PID=11032]    [version] get_app_version(): getting app version for WU#510035 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2480) appid:29
2013-11-29 11:21:39.9316 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.9316 [PID=11032]    [version] get_app_version(): getting app version for WU#505531 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_125) appid:21
2013-11-29 11:21:39.9316 [PID=11032]    [version] get_app_version(): getting app version for WU#505534 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_128) appid:21
2013-11-29 11:21:39.9316 [PID=11032]    [version] get_app_version(): getting app version for WU#503231 (LATeah0069U_48.0_500_-3.85e-10) appid:25
2013-11-29 11:21:39.9317 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.9317 [PID=11032]    [version] get_app_version(): getting app version for WU#510514 (PA0080_024D1_2) appid:27
2013-11-29 11:21:39.9317 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.9317 [PID=11032]    [version] get_app_version(): getting app version for WU#510034 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2464) appid:29
2013-11-29 11:21:39.9317 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:39.9318 [PID=11032]    [version] get_app_version(): getting app version for WU#505540 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_134) appid:21
2013-11-29 11:21:39.9318 [PID=11032]    [version] get_app_version(): getting app version for WU#505540 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_134) appid:21
2013-11-29 11:21:39.9319 [PID=11032]    [version] get_app_version(): getting app version for WU#503247 (LATeah0069U_48.0_500_-4.01e-10) appid:25
2013-11-29 11:21:39.9319 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.9319 [PID=11032]    [version] get_app_version(): getting app version for WU#505542 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_136) appid:21
2013-11-29 11:21:39.9320 [PID=11032]    [version] get_app_version(): getting app version for WU#503249 (LATeah0069U_48.0_500_-4.03e-10) appid:25
2013-11-29 11:21:39.9320 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:39.9320 [PID=11032]    [version] get_app_version(): getting app version for WU#510513 (PA0080_024D1_0) appid:27
2013-11-29 11:21:39.9320 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:39.9320 [PID=11032]    [send] est. duration for WU 510513: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:39.9320 [PID=11032]    [send] [WU#510513] meets deadline: 23932.64 + 11565.94 < 1209600
2013-11-29 11:21:40.0090 [PID=11032]    [send] Sending app_version einsteinbinary_BRP5 7 139 BRP5-opencl-intel_gpu; projected 40.00 GFLOPS
2013-11-29 11:21:40.0091 [PID=11032]    [send] est. duration for WU 510513: unscaled 11248.66 scaled 11565.94
2013-11-29 11:21:40.0092 [PID=11032]    [HOST#9649] Sending [RESULT#1232396 PA0080_024D1_0_1] (est. dur. 11565.94s (3h12m45s93)) (max time 224973.24s (62h29m33s23))
2013-11-29 11:21:40.0094 [PID=11032]    [version] get_app_version(): getting app version for WU#510029 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2384) appid:29
2013-11-29 11:21:40.0094 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:40.0094 [PID=11032]    [version] get_app_version(): getting app version for WU#505542 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_136) appid:21
2013-11-29 11:21:40.0094 [PID=11032]    [version] get_app_version(): getting app version for WU#505543 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_137) appid:21
2013-11-29 11:21:40.0094 [PID=11032]    [version] get_app_version(): getting app version for WU#503249 (LATeah0069U_48.0_500_-4.03e-10) appid:25
2013-11-29 11:21:40.0094 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:40.0095 [PID=11032]    [version] get_app_version(): getting app version for WU#510513 (PA0080_024D1_0) appid:27
2013-11-29 11:21:40.0095 [PID=11032]    [version] returning cached version: [AV#729]
2013-11-29 11:21:40.0095 [PID=11032]    [version] get_app_version(): getting app version for WU#510029 (p2030.20130202.G203.30-00.11.N.b0s0g0.00000_2384) appid:29
2013-11-29 11:21:40.0095 [PID=11032]    [version] returning cached version: [AV#721]
2013-11-29 11:21:40.0095 [PID=11032]    [version] get_app_version(): getting app version for WU#505543 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_137) appid:21
2013-11-29 11:21:40.0096 [PID=11032]    [version] get_app_version(): getting app version for WU#505544 (p2030.20130202.G203.30-00.11.N.b3s0g0.00000_138) appid:21
2013-11-29 11:21:40.0096 [PID=11032]    [version] get_app_version(): getting app version for WU#503250 (LATeah0069U_48.0_500_-4.04e-10) appid:25
2013-11-29 11:21:40.0096 [PID=11032]    [version] returning cached version: [AV#768]
2013-11-29 11:21:40.0119 [PID=11032]    Sending reply to [HOST#9649]: 10 results, delay req 60.00
2013-11-29 11:21:40.0124 [PID=11032]    Scheduler ran 0.449 seconds

2013-11-29 11:21:39.6275 [PID=11032] [HOST#9649] Sending [RESULT#1212496 LATeah0069U_48.0_500_-4.01e-10_1] (est. dur. 1601.52s (0h26m41s52)) (max time 31151.75s (8h39m11s74))
2013-11-29 11:21:39.6300 [PID=11032] [locality] send_old_work(LATeah0069U_48.0_500_-4.01e-10_1) sent result created 347.4 hours ago [RESULT#1212496]
2013-11-29 11:21:39.6300 [PID=11032] [locality] Note: sent NON-LOCALITY result LATeah0069U_48.0_500_-4.01e-10_1
2013-11-29 11:21:39.6300 [PID=11032] [locality] send_new_file_work(): try to send old work

I think the reason that you get tasks from the Gamma Ray search is that the scheduler is resending lost work, that is work that's been allocated but for some reason is not present on your machine. Task selection via prefs don't apply when resending work so you'll keep getting them until there are no more missing tasks, but you should not get any new ones.

I checked host #9649 (the posted log is from a scheduler contact from this host) and there are no in progress Gamma Ray tasks so that host should not get any more of them unless you opt into that search again.

---