This guide can be used to evaluate the
performance of a laboratory with regards to its adherence to
established laboratory quality practices for the essential elements
of managing a well-performing laboratory. The suggested scoring
system can be used to identify the laboratory areas which need
improvement in performance.
The assessment guide (Table 1) should be
adjusted or modified to reflect the specific laboratory quality
system that the laboratory follows. This guide is based on Practice
D6792.
5.3 Similar but more generic schemes can
be found in other ASTM standards such as Guide E548, E882, E994,
and E1323. But this guide is the first one to attempt a numerical
evaluation for the petroleum products and lubricants testing
laboratories in the oil industry.
|
Number |
Issue |
Comment |
Maximum Score |
1.0 PERSONNEL &
TRAINING |
|
|
1.1 |
Is there an
organization chart available for the lab? |
|
/ 5 |
1.2 |
Is there a
management approved policy, directive, endorsement or the like for
the goals and operations of the laboratory? |
|
/ 10 |
1.3 |
Are job
descriptions provided for all laboratory associates? |
|
/ 5 |
1.4 |
Are personnel
qualification records including training maintained for all lab
personnel? |
|
/ 5 |
1.5 |
Are technicians
trained in all pertinent tests? |
|
/ 10 |
1.6 |
Are newly hired
technicians trained by approved trainers? |
|
/ 10 |
1.7 |
Before new
technicians perform routine analysis, are they evaluated to produce
correct results by analyzing quality control standards? |
|
/ 10 |
1.8 |
Are technicians
provided instrument training courses where necessary? |
|
/ 10 |
1.9 |
Are there back-up
technicians for each test? |
|
/ 5 |
1.10 |
Is there a
documented system for training and is it being followed? |
|
/ 5 |
1.11 |
Is there a system
to recognize innovative ideas from technicians? |
|
/ 5 |
1.12 |
Are there quality
improvement teams organized in the lab? |
|
/ 5 |
1.13 |
Are results of
these teams’ work readily
available? |
|
/ 5 |
1.14 |
Does management
review the work of these quality teams? |
|
/ 5 |
|
|
TOTAL |
/195 |
2.0 SAMPLE PREPARATION |
|
|
2.1A |
Has a representative
sample been obtained per Practice D4057 or D4177 protocols? |
|
/ 5 |
2.2B |
If the sample
analysis involves LPG, does the sampling and storage follow
instructions in Practice D6849? |
|
/ 5 |
2.3B |
Are LPG samples
obtained according to Practice D3700 if using a floating piston
cylinder? |
|
/ 5 |
2.4B |
Are LPG samples
collected using manual method Practice D1265? |
|
/ 5 |
2.5B |
For the analysis of
calcined petroleum coke, is Practice D6969 followed for sample
preparation? |
|
/ 5 |
2.6B |
For collection of
calcined petroleum coke, is Practice D6970 followed? |
|
/ 5 |
2.7B |
For ampulization and
storage of gasoline is Practice D6596 being followed? |
|
/ 5 |
2.8B |
If volatility
measurements are conducted on fuel samples, is Practice D5842 being
followed? |
|
/ 5 |
2.9B |
Is Practice D4296
being followed in sampling of pitch? |
|
/ 5 |
2.10B |
Are aviation fuel
samples stored in containers suitable according to Practice
D4306 |
|
/ 5 |
2.11B |
Are gas turbine
fuels handled in accordance with Practice D4418? |
|
/ 5 |
2.12B |
If in-service
monitoring of fluids is done, is Practice D4378 being
followed? |
|
/ 5 |
2.13 |
Are appropriate
sample containers being used, checked for their integrity,
cleanliness, and compatibility? |
|
/ 5 |
2.14B |
Is liquid sample
mixing done using Practice D5854 protocol? |
|
/ 5 |
|
|
TOTAL |
/
175 |
3.0 TEST METHOD COMPLIANCE |
|
|
3.1 |
Are all test
methods used in the lab available to all staff members? |
|
/ 10 |
3.2 |
Is a current site
specific lab manual available? |
|
/ 10 |
3.3 |
Are the methods
specified in the product specifications used for analysis? |
|
/ 10 |
3.4 |
Are all tests
performed in the lab following the exact language of the written
test methods? |
|
/ 30 |
3.5C |
Has the lab
modified any of the specified test methods? |
|
? |
3.6 |
If so, has it been
shown that the modified method gives results equivalent to those
obtained by the specified method? |
|
/ 20 |
3.7 |
If a method is
modified, is the customer informed about this change? |
|
/ 5 |
3.8 |
Are all tests
specified in the product specifications performed? |
|
/ 10 |
3.9 |
If not, has
approval been given in writing by the process leaders and/or
customers not to carry out all tests? |
|
/ 5 |
3.10 |
Does the lab have
appropriate instrument for the analysis? |
|
/ 10 |
3.11 |
Is an equipment
inventory list available and is it up-to-date? |
|
/ 5 |
3.12 |
Are major
instruments included in service contracts? |
|
/ 5 |
3.13 |
If not, has
arrangement been made to get the instrument speedily repaired or
use of a back-up lab for analysis? |
|
/ 5 |
3.14 |
Are logs kept of
all downtime and service problems of all instruments? |
|
/ 5 |
3.15 |
Is there a system
of backup in case of instrument failure? |
|
/ 5 |
3.16 |
Is there a long
range plan of upgrading and replacing older instrumentation? |
|
/ 5 |
3.17 |
Has a maintenance
schedule been established for all equipment in the lab? |
|
/ 10 |
3.18 |
Is a record of special
or routine or preventive maintenance kept and is it
up-to-date? |
|
/ 5 |
3.19 |
If a duplicate
analysis is done for some reason, are the results checked to see
that they meet Practice D3244 or ISO 4259 criteria for replicate
testing? |
|
/ 10 |
3.20 |
In reporting the
results, are protocols given in Practice E29 and standards followed
regarding rounding of the test results? |
|
/ 10 |
|
|
TOTAL |
/
175 |
4.0 CALIBRATION OF INSTRUMENTS |
|
|
4.1 |
Are all pertinent
calibration standards available in the lab? |
|
/ 10 |
4.2 |
Are they all stored
in clean, safe, and contamination-free environment? |
|
/ 10 |
4.3B |
Is Practice D4307
followed for preparing liquid blends for use as analytical
standards? |
|
/ 5 |
4.4B |
Is Practice D4051
used for the preparation of low-pressure gas blends? |
|
/ 5 |
4.5 |
Are the calibration
standards traceable to national or international standards? |
|
/ 5 |
4.6 |
Where appropriate,
are the values for reference materials produced by following the
appropriate NIST or other standards issuing body certification
protocol? |
|
/ 10 |
4.7B |
If used, are crude
oil samples for mercury analysis handled as per Practice
D7482? |
|
/ 5 |
4.8 |
Is relevant
calibration done on all instruments before sample analysis? |
|
/ 10 |
4.9 |
Has a calibration
schedule been established for lab equipment? |
|
/ 5 |
4.10 |
Are calibration
procedures documented and available to all lab personnel? |
|
/ 10 |
4.11 |
If the calibration
is done through a vendor, is it checked that it meets the
requirements? |
|
/ 10 |
4.12 |
Are all calibration
records maintained containing all necessary information? |
|
/ 10 |
4.13 |
When found to be
out of calibration, is the instrument taken out of operation until
the problem is fixed? |
|
/ 10 |
4.14B |
Was the performance
of process stream analyzer, if used, validated using Practice D3764
or D6122, whichever is applicable? |
|
/ 5 |
4.15B |
If used, are
moisture analyzers calibrated per Practice D4178? |
|
/ 5 |
4.16B |
If analyzers are
used has a linear correlation been established between the analyzer
and primary ASTM test methods using Guide D7235? |
|
/ 15 |
4.17B |
If metals are
analyzed using ICP-AES, is Practice D7260 followed for ICP-AES
operation? |
|
/ 10 |
4.18B |
If metals are
analyzed using XRF, is Practice D7343 followed for XRF
operation? |
|
/ 10 |
4.19B |
If balances are used
for analytical purposes, are they being calibrated using Test
Method E898? |
|
/ 10 |
4.20B |
If thermometric
measuring devices are being used, are they calibrated using Test
Method E77? |
|
/ 10 |
4.21B |
If timers are used
for measurement purposes, are they being calibrated or verified
using the relevant standard test method? |
|
/ 10 |
|
|
TOTAL |
/
180 |
5.0 STATISTICAL QUALITY CONTROL |
|
|
5.1 |
Is a QC program set
up for each routine test performed in the lab, if pertinent? |
|
/ 10 |
5.2 |
Has a schedule of
QC frequency been established and followed? |
|
/ 10 |
5.3 |
What is the
frequency of QC testing? |
|
/ 10 |
5.4D |
Is a QC samples
analyzed with each “batch” of samples? |
|
/ 10 |
5.5 |
Is a list available
showing tests with QC program, standards, frequency, and the
analyst responsible for running it? |
|
/ 10 |
5.6 |
Are control charts
used in the lab for all appropriate tests? |
|
/ 10 |
5.7 |
Are the control
charts being plotted in real time? |
|
/ 10 |
5.8E |
Are the control
charts plotted manually or electronically? |
|
/ 5 |
5.9F |
Are the control
charts plotted by technicians or supervisors? |
|
/ 10 |
5.10G |
Are the control
charts displayed near the test stations or filed in cabinets? |
|
/ 5 |
5.11 |
Are all relevant
data recorded on the QC charts (e.g., analyst name or initial,
date, numerical value, etc.)? |
|
/ 10 |
5.12 |
Are the values
reported on the control chart consistent with the reporting
resolution in the test method? |
|
/ 5 |
5.13 |
Are the mean
± standard deviation values assigned
to the QC sample based on at least 20 replicate measurements? |
|
/10 |
5.14 |
If starting the
chart with 15 or less data points, is it revised after obtaining 20
-30 data points? (D6299) |
|
/10 |
5.15H |
How many
statistical run rules are used for control charts? |
|
/ 10 |
5.16 |
Have any run rules
been violated? |
|
/ 10 |
5.17 |
What action was
taken in such cases? Is the action taken documented? |
|
/ 0 –10 |
5.18 |
If a QC data point
is found to be out-ofcontrol, are the samples analyzed between the
last good QC data point and the first bad data point
reanalyzed? |
|
/ 10 |
5.19 |
Are appropriate QC
samples being used for specific tests and relevant to the matrices
being analyzed? |
|
/ 5 |
5.20 |
Are QC samples
prepared from stable, homogenous and well-characterized materials
similar to the samples being analyzed? |
|
/ 5 |
5.21I |
Is the same
material used both for calibration and QC? |
|
/ 10 |
5.22 |
Is QC material
available in sufficiently large quantities? |
|
/ 10 |
5.23 |
Is a program in
place to replace the depleting QC material with a new lot of
material? |
|
/ 10 |
5.24 |
Is action taken when
the control chart displays an upset? |
|
/ 10 |
5.25 |
Is the action taken
noted on the control chart? |
|
/ 10 |
5.26 |
Does it appear that
the corrective action taken has improved the chart? |
|
/ 5 |
5.27 |
Does every test
have a control chart? -If not, why not? |
|
/ 10 |
5.28 |
Are sigmas
calculated and updated from the control chart data? (D6299) |
|
/ 10 |
5.29 |
Are they equivalent
or better than the ones quoted in the standard test methods? |
|
/ 10 |
5.30 |
If TPIs calculated
are below the expected level in the standard methods, is any action
taken to improve this situation? |
|
/ 10 |
5.31J |
Are the sample
results reported to the customers if the QC analysis is found to be
out of control? |
|
/ 0 – 10 |
5.32 |
Does the laboratory
use random and/or blind testing to evaluate the performance? |
|
/ 5 – 0 |
5.33 |
Are procedures in
place to revise or replace the QC charts with new ones? (Practice
D6792) |
|
/ 10 |
|
|
TOTAL |
/
295 |
6.0 QUALITY MANAGEMENT PROCESS |
|
|
6.1 |
Does the laboratory
have or is it a part of site ISO 9000 registration? |
|
/ 10 |
6.2 |
Does the laboratory
have ISO 17025 accreditation? |
|
/ 10 |
6.3 |
Does the laboratory
have any other quality accreditation? Specify. |
|
/ 10 |
6.4 |
Is there a specific
quality manager (or other designation) overseeing the quality
activities in the laboratory? |
|
/ 10 |
6.5 |
Is there a written
quality manual? |
|
/ 10 |
6.6 |
Are all staff
trained in the quality principles and manual? |
|
/ 10 |
6.7 |
Is a sample
Analysis schedule available? |
|
/ 10 |
6.8 |
Are all test method
files up-to-date with current version? |
|
/ 10 |
6.9 |
Does the laboratory
conduct periodic internal audits to check that the expected quality
systems are working? |
|
/ 20 |
6.10 |
Are audits of test
methods conducted to confirm adherence to documented test
methods? |
|
/ 20 |
6.11 |
Are the results
promptly documented and action, if necessary, taken to correct the
deficiencies? |
|
/ 10 |
6.12 |
If deficiencies are
observed in external audits (e.g., ISO 9000), are prompt corrective
actions taken and documented? |
|
/ 20 |
6.13 |
Is there a
procedure for following up on any customer complaints and
documenting the results? |
|
/ 10 |
6.14 |
Has the laboratory
established continuous improvement goal and teams to implement
them? |
|
/ 10 |
6.15 |
Are the activity
reports of such teams available showing continuous
improvement? |
|
/ 10 |
|
|
TOTAL |
/180 |
7.0 PARTICIPATION IN PROFICIENCY
TESTING |
|
|
7.1 |
Does the laboratory
take part in pertinent round robins or crosschecks? |
|
/ 10– 0 |
7.2 |
What is the
frequency of the laboratory’s
participation in such round robins? |
|
? |
7.3 |
How many outliers
did the lab have in the last three proficiency test programs?
- How many tests were found with consecutive outliers in them? |
|
/0 –10 |
7.4 |
What corrective
actions, if any, are taken to follow up on the unsatisfactory
proficiency testing results? |
|
/0 – 10 |
7.5 |
Does the lab
participate in ASTM ILCP cross checks? |
|
/ 0 – 10 |
7.6 |
Are the Z-scores
assigned to lab results satisfactory? |
|
/ 10 |
7.7B |
Does the lab take
part in TMC surveillance panel analysis? |
|
/ 0 – 10 |
7.8B |
If the lab takes
part in TMC surveillance, how well was the lab rated by TMC? / |
|
NA-10 |
|
|
TOTAL |
/
70 |
8.0 INFORMATION MANAGEMENT SYSTEM |
|
|
8.1 |
Are all samples
submitted to lab logged in before the analysis? |
|
/ 10 |
8.2 |
Does each submitted
sample receive an unique identification number? |
|
/ 5 |
8.3 |
Does the system
produce backlog and turnaround time reports? |
|
/ 5 |
8.4 |
Are all analyzed
samples logged out? |
|
/ 5 |
8.5 |
Are certificates of
analysis printed out when the analysis is complete? |
|
/ 5 |
8.6 |
Do all COAs have
all necessary information (e.g., analyst name or initials,
day/time, sample identification name or number, test method used,
numerical or other results)? |
|
/ 10 |
8.7 |
Does the system
plot quality control charts? |
|
/ 10 |
8.8 |
Does the system
flag out-of-statistical control data? |
|
/ 10 |
8.9 |
Are all staff
members familiar with the IMS? |
|
/ 10 |
8.10 |
Does the lab have a
documented system for sample retention? |
|
/ 5 |
8.11 |
Are all data (e.g.,
weights, volumes, dilutions, analyst, results, etc.) produced in
the lab permanently recorded? |
|
/ 10 |
8.12 |
Are all required
records kept for required period of time? |
|
/ 5 |
8.13 |
Is there a document
control system established in the lab? |
|
/ 5 |
8.14 |
If some analyses
are done by subcontracting to another lab is this clearly indicated
in the COA? |
|
/ 5 |
8.15 |
Before using an
outside lab for analysis, has it been audited for its suitability
to do the analysis? |
|
/ 5 |
8.16 |
Are all various
records kept in a secure place with only authorized personnel being
admitted therein? |
|
/5 |
|
|
TOTAL |
/
110 |
9.0 NON-CONFORMANCE AND CORRECTIVE
ACTIONS |
|
|
9.1 |
Does the lab have a
documented procedure for corrective actions and non-conformances,
including Corrective Action Report (CAR)? Were they documented?
Approximate number of CARs per year? |
|
/ 20 |
9.2 |
What items are
addressed by the corrective action system? |
|
/ 10 |
9.3 |
Are roles and
responsibilities for initiation, investigation, root cause
analysis, corrective action, review, and approval identified and
documented? |
|
/ 20 |
9.4 |
Is reviewer and/or
approver separate from the initiator/investigator if possible? |
|
/ 10 |
9.5 |
Is there a target
date for CAR closure? |
|
/ 10 |
9.6 |
Are CARs closed in
a timely manner? |
|
/ 10 |
9.7 |
Do CAR forms have
all appropriate signatures? |
|
/ 10 |
9.8 |
How does the lab
tract status of active CARs? |
|
/ 10 |
9.9 |
Activities that
demonstrate review or continuous improvement. |
|
/ 20 |
|
|
TOTAL |
/
120 |
10.0 SAFETY PRACTICES |
|
|
10.1 |
Are all lab
personnel trained in safety and emergency procedures? |
|
|
10.2 |
Do all personnel
have and use personal protective equipment (e.g., safety glasses,
lab coats, gloves, safety shoes, etc.)? |
|
|
10.3 |
Are fire
extinguishers, safety showers, eye washers working properly? |
|
|
10.4 |
Is there a
regularly scheduled safety inspection? |
|
|
10.5 |
Are chemicals and
solvents stored in proper safety cabinets? |
|
|
10.6 |
Are hoods properly
ventilated? |
|
|
10.7 |
Are gas cylinders
properly secured and safety valves inspected? |
|
|
10.8 |
Are MSDS sheets or
other safety information on chemicals in the lab readily
available? |
|
|
10.9 |
Are used chemicals
and samples properly disposed off in accordance with government
regulations? |
|
|
10.10 |
Does the lab have a
clean and neat appearance? |
|
|
10.11 |
Does the lab
environment meet regulations regarding temperature range, dust,
noise, radiation, drafts, etc.? |
|
|