Job Serving Diagnostics

Is the service running?

If no, check the service logs and windows log for information

Does the server state in the database mirror the actual state of the service?

If the service is running but the database records it as being stopped, the service will not be served jobs.

    select ServerID, convert(nvarchar(50), Hostname) as Hostname, ss.Name as ServerState, MaxQueueLength, MaxActiveJobs, IsQueueProcessingEnabled 
      from config.t_Server s
     inner join config.t_ServerState ss on s.StateID = ss.ServerStateID

Target Set Expansions

Have all targets sets been expanded fully...

select ipar.IPAddressRangeId, ipar.Name, ipart.Name as TargetSetType, 
       StartIPAddress, EndIPAddress, 
       jobs.convertBinaryToIP(aip.Start) as ActiveIPRangesStart,
	   jobs.convertBinaryToIP(aip.Finish) as ActiveIPRangesFinish,
       aip.LocationID,
       ProjectID,
       case jobs.convertBinaryToIP(aip.Finish) when EndIPAddress then 'Fully Expanded'
                                               else 'In Progress'
       end as [State]
 from [jobs].[t_ActiveIPRanges] aip with (nolock)
inner join config.t_IPAddressRange ipar with (nolock) on aip.IPAddressRangeID = ipar.IPAddressRangeID
inner join config.t_IPAddressRangeType ipart with (nolock) on ipart.IPAddressRangeTypeID = ipar.IPAddressRangeTypeID

If yes, scan engine believes it has expanded everything it needs to.

Are there Jobs Queued?

Run the Job Stats script here.

Does the Queue By State query show queued items?
Does the Queue By State and Project show that there are items queued for projects which are running?

Are the Queue Processing procs being executed.

If all targets sets have not been fully expanded and there are no queued jobs, check that the queue step procs are being executed.

Open SQL Profiler
Select the File > New Trace menu option and login to the server. There are some specific permissions/securables (ALTER TRACE maybe). A sys admin user will definitely have access.
In the Trace Properties dialog which appears, select the Events Selection tab.
Click the Show all events and Show all columns checkboxes.
In the event list, ensure only the following items are ticked..
- Stored Prodedures
  - RPC: Completed
  - RPC: Starting
  - SP: Completed
  - SP: Starting
Click the Column Filters button.
Add Like filters for the following properties (if there is a pre-existing filter on ApplicationName, leave it there)…
1. DatabaseName: Name of iQSonar database. (Normally iQSonarSE)
2. HostName: Hostname of the scanning server
3. TextData: %jobs%
Start the trace

We expect to see regular executions of jobs.QueueStep1_ExpandRanges and jobs.QueueStep2_SetJobStatus.

If yes - and nothing is being queued - find out why?

What states are the queued items in - do they indicate what's happening? (e.g. project not started, exclusion scan window in effect).
Review the jobs.QueueStep2_SetJobStatus code to determine why jobs aren't reaching a state of Queued.

If yes - but the rate of job throughput is slow...

Run the Queued Job Analysis script here.
This will record how many jobs are being added and removed from the queue.
Do the scan engine servers have free capacity?
Can the queued items only be picked up by a server which is currently maxed out.

Is the job serving proc running...

If items are queued and the queue processing procs are being executed, follow the steps above to create a database trace and check for regular executions of jobs.Targets_Serve.

Is the server in breach...

Queue processing is suspended when the average resource usage over 5 minutes on the scanning server exceeds the levels configured in Administration > Scanning Servers. It will only resume when average resource usage has fallen below the configured limits for 5 minutes. To determine if a server is in breach of these limits execute the following query. The output results are SQL Statements to check the performance limits for each individual server. Copy and paste the results into SQL Management Studio and execute that. If the value in any of the Is<X>InBreach columns is 1 then processing queue processing and job serving have been suspended. Also check for the following text in the logs: Queue processing is suspended.

select '/* Server: ' + s.HostName + ' */   exec [config].[Performance_Check] ''' + convert(nvarchar(50), InstallationID) + '''   -- Server: ' + s.Hostname
  from config.t_Server s

Is the server actually requesting jobs...

Check the service logs for message indicating Polled X new targets. If X is -1 then the server has no further capacity to serve additional jobs

How many jobs are currently in progress...

How many jobs are currently in progress.
For the longest running jobs are the target logs being updated - if not, the job may have died or not been recovered after a restart/crash.

    select s.ServerID,  convert(nvarchar(50), s.Hostname) as Hostname, tp.Name as JobPhase, ts.Name as JobStatus, j.JobId, ParentJobID, convert(nvarchar(50), j.Hostname) as Hostname, j.IPAddressBinary, IsComplete, StartDate, EndDate, DATEDIFF(hh, StartDate, getDate()) as 'Duration {Hours)',   q.JobStatusID, 
           '            QUEUE DATA  -->          ', js.name as QueueStatus, DisabledUntil, [Priority], Source, QueueId, IgnoreConfiguredIPs
      from jobs.t_Job j
     inner join config.t_Server s on j.ServerID = s.ServerID
     inner join jobs.t_TargetStatus ts on j.TargetStatusID= ts.TargetStatusID
     inner join jobs.t_TargetPhase tp on ts.TargetPhaseID = tp.TargetPhaseID
     inner join jobs.t_Queue q on j.JobId = q.JobID
     inner join jobs.t_JobStatus js on q.JobStatusID = js.JobStatusID
     where IsComplete = 0

Anything in the logs?

Search the Service logs for the text iQSonar Job Poll. How much information you see here will depend on the level of logging (configuration for this not in scope here). Look for errors or warnings - but also look for messages similar to any of the following...

No queue capacity.
Queue processing is suspended
Polled N new targets
No new targets available
Failed to process target
Job acquisition stopped
iQSonar license has expired - scanning is suspended
iQSonar Server not activated - scanning is suspended.

.