Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »


Server Configuration

Is the service running?


If yes, does the database think the service is running?


    select ServerID, convert(nvarchar(50), Hostname) as Hostname, ss.Name as ServerState, MaxQueueLength, MaxActiveJobs, IsQueueProcessingEnabled 
      from config.t_Server s
     inner join config.t_ServerState ss on s.StateID = ss.ServerStateID

.


Target Set Expansions

Have all targets sets been expanded fully...

select ipar.IPAddressRangeId, ipar.Name, ipart.Name as TargetSetType, 
       StartIPAddress, EndIPAddress, 
       jobs.convertBinaryToIP(aip.Start) as ActiveIPRangesStart,
	   jobs.convertBinaryToIP(aip.Finish) as ActiveIPRangesFinish,
       aip.LocationID,
       ProjectID,
       case jobs.convertBinaryToIP(aip.Finish) when EndIPAddress then 'Fully Expanded'
                                               else 'In Progress'
       end as [State]
 from [jobs].[t_ActiveIPRanges] aip with (nolock)
inner join config.t_IPAddressRange ipar with (nolock) on aip.IPAddressRangeID = ipar.IPAddressRangeID
inner join config.t_IPAddressRangeType ipart with (nolock) on ipar.IPAddressRangeTypeID = ipar.IPAddressRangeTypeID


Location Configuration

Each location must be enabled and linked to to a scanning server. The MaxScanningCount dictates how many targets in the location can be scanned at one time.

select l.Name as Location, l.IsEnabled, l.MaxScanningCount,
       s.ServerId, s.Version, s.Hostname
  from config.t_Location l
  left join config.t__Location_Server ls on l.LocationId = ls.LocationID
  left join config.t_Server s on ls.ServerID = s.ServerID
  left join config.t_ServerState ss on s.StateID = ss.ServerStateID


If there is a scan window for the location, jobs will only be served during this time. If the scan window is an exclusion, there needs to be a non-exclusion scan window for jobs to be served.

select sw.*, swwd.WeeklyDays, l.LocationId, l.Name
  from config.t_ScanWindow sw
  left join config.t_ScanWindowWeeklyDays swwd on sw.ScanWindowID = swwd.ScanWindowID
  left join config.t_Location l on sw.LocationID = l.LocationID

.

Project Configuration

The project should be running...

select p.Name as Project, ps.Name as [Status]
  from config.t_Project p 
 inner join config.t_ProjectState ps on p.ProjectStateID = ps.ProjectStateID

.


Current Snapshot of Incomplete Jobs

    select s.ServerID,  convert(nvarchar(50), s.Hostname) as Hostname, tp.Name as JobPhase, ts.Name as JobStatus, j.JobId, ParentJobID, convert(nvarchar(50), j.Hostname) as Hostname, j.IPAddressBinary, IsComplete, StartDate, EndDate, DATEDIFF(hh, StartDate, getDate()) as 'Duration {Hours)',   q.JobStatusID, 
           '            QUEUE DATA  -->          ', js.name as QueueStatus, DisabledUntil, [Priority], Source, QueueId, IgnoreConfiguredIPs
      from jobs.t_Job j
     inner join config.t_Server s on j.ServerID = s.ServerID
     inner join jobs.t_TargetStatus ts on j.TargetStatusID= ts.TargetStatusID
     inner join jobs.t_TargetPhase tp on ts.TargetPhaseID = tp.TargetPhaseID
     inner join jobs.t_Queue q on j.JobId = q.JobID
     inner join jobs.t_JobStatus js on q.JobStatusID = js.JobStatusID
     where tp.TargetPhaseID = @PHASE_IN_PROGRESS
        or IsComplete = 0

.


Confirm Job Serving Threads are Running

  1. Open SQL Profiler
  2. Select the File > New Trace menu option and login to the server.  There are some specific permissions/securables (ALTER TRACE maybe). A sys admin user will definitely have access.
  3. In the Trace Properties dialog which appears, select the Events Selection tab.
  4. Click the Show all events and Show all columns checkboxes.
  5. In the event list, ensure only the following items are ticked..
    • Stored Prodedures
      • RPC: Completed 
      • RPC: Starting
      • SP: Completed
      • SP: Starting
  6. Click the Column Filters button.
  7. Add Like filters for the following properties (if there is a pre-existing filter on ApplicationName, leave it there)…
  8. DatabaseName:   Name of iQSonar database. (Normally iQSonarSE)
  9. HostName: Hostname of the scanning server
  10. TextData: %jobs%
  11. Start the trace


Let it run for about 5 minutes. You should see repeated entries for jobs.QueueStep2_SetJobStatus and jobs.Targets_Serve. If you don't see this, it's likely the threads have fallen over and a service re-start may be required.

.


Has Queue Processing Been Suspended?

Queue processing is suspended when the average resource usage over 5 minutes on the scanning server exceeds the levels configured in Administration > Scanning Servers.  It will only resume when average resource usage has fallen below the configured limits for 5 minutes. To determine if a server is in breach of these limits execute the following query. The output results are SQL Statements to check the performance limits for each individual server. Copy and paste the results into SQL Management Studio and execute that. If the value in any of the Is<X>InBreach columns is 1 then processing queue processing and job serving have been suspended.  Also check for the following text in the logs: Queue processing is suspended.

select '/* Server: ' + s.HostName + ' */   exec [config].[Performance_Check] ''' + convert(nvarchar(50), InstallationID) + '''   -- Server: ' + s.Hostname
  from config.t_Server s


Anything in the logs?

Search the Service logs for the text iQSonar Job Poll. How much information you see here will depend on the level of logging (configuration for this not in scope here). Look for errors or warnings - but also look for messages similar to any of the following...

  • No queue capacity.
  • Queue processing is suspended
  • Polled N new targets
  • No new targets available
  • Failed to process target
  • Job acquisition stopped
  • iQSonar license has expired - scanning is suspended
  • iQSonar Server not activated - scanning is suspended.

.

Is Anything Blocking the Queue?

Some builds (pre Brendan R3 RC2) have an issue where targets from non-running projects are allowed to remain in the Queued state. These targets will never be picked up by job serving so they block the addition of new items to the queue.

declare @PROJ_STATE_RUNNING int             = (select ProjectStateID from config.t_ProjectState where Name = 'Running')
declare @JOB_STATUS_QUEUED  int             = (select JobStatusID from jobs.t_JobStatus js where Constant = 'JOB_STATUS_QUEUED')

select * 
  from jobs.t_Queue q
 inner join config.t_Project p on q.ProjectId = p.ProjectId
 inner join config.t_ProjectState ps on p.ProjectStateID = ps.ProjectStateID
 where JobStatusID       = @JOB_STATUS_QUEUED
   and p.ProjectStateID != @PROJ_STATE_RUNNING

To fix this, run:

declare @PROJ_STATE_RUNNING int             = (select ProjectStateID from config.t_ProjectState where Name = 'Running')
declare @JOB_STATUS_QUEUED  int             = (select JobStatusID from jobs.t_JobStatus js where Constant = 'JOB_STATUS_QUEUED')
declare @JOB_STATUS_PROJECT_NOT_RUNNING int = (select JobStatusID from jobs.t_JobStatus js where Constant = 'JOB_STATUS_PROJECT_NOT_RUNNING')

update q
   set q.JobStatusID = @JOB_STATUS_PROJECT_NOT_RUNNING
  from jobs.t_Queue q
 inner join config.t_Project p on q.ProjectId = p.ProjectId
 inner join config.t_ProjectState ps on p.ProjectStateID = ps.ProjectStateID
 where JobStatusID       = @JOB_STATUS_QUEUED
   and p.ProjectStateID != @PROJ_STATE_RUNNING


There was also an issue where targets in the queue can not be linked back to Targets. The root cause is still under investigation. Th e following statement identifies queue items affected by this problem - again they block spaces in the queue as job serving will not pick them up.

    declare @JOB_STATUS_QUEUED  int             = (select JobStatusID from jobs.t_JobStatus js where Constant = 'JOB_STATUS_QUEUED')

    select *
      from jobs.t_Queue q
      left join ( 
                  select p.projectID, ipar.StartIPAddressBinary, ipar.EndIPAddressBinary, ipar.Hostname, ipar.IPAddressRangeID
                    from config.t_Project p
                   inner join config.t__Project_Location pl ON p.ProjectID   = pl.ProjectID
                   inner join config.t_Location l           ON pl.LocationID = l.LocationID
                   inner join config.t_IPAddressRange ipar  ON l.LocationID  = ipar.LocationID
                   where ipar.IsExclusion = 0
                ) projectTargets on q.ProjectId = projectTargets.ProjectID
                                AND ( ( q.IPAddressBinary BETWEEN projectTargets.StartIPAddressBinary AND EndIPAddressBinary ) 
                                          OR 
                                      ( q.IPAddressBinary = projectTargets.StartIPAddressBinary AND EndIPAddressBinary IS NULL) 
                                          OR 
                                      ( q.Hostname = projectTargets.Hostname ) 
                                          OR 
                                      ( q.IgnoreConfiguredIPs = 1 ) )
     WHERE q.JobStatusID = @JOB_STATUS_QUEUED
       and projectTargets.IPAddressRangeID is null



  • No labels