Background. We conducted a validation study of digital pathology for the quantitative assessment of tissue-based biomarkers with immunohistochemistry. Objective. To examine observer agreement as a function of viewing modality (digital versus optical microscopy), whole slide versus tissue microarray (TMA) review, biomarker type (HER2 incorporating membranous staining and Ki-67 with nuclear staining), and data type (continuous and categorical). Methods. Eight pathologists reviewed 50 breast cancer whole slides (25 stained with HER2 and 25 with Ki-67) and 2 TMAs (1 stained with HER2, 1 with Ki-67, each containing 97 cores), using digital and optical microscopy. Results. Results showed relatively high overall interobserver and intermodality agreement, with different patterns specific to biomarker type. For HER2, there was better interobserver agreement for optical compared to digital microscopy for whole slides as well as better interobserver and intermodality agreement for TMAs. For Ki-67, those patterns were not observed. Conclusions. The differences in agreement patterns when examining different biomarkers and different scoring methods and reviewing whole slides compared to TMA stress the need for validation studies focused on specific pathology tasks to eliminate sources of variability that might dilute findings. The statistical uncertainty observed in our analyses calls for adequate sampling for each individual task rather than pooling cases. 1. Introduction Digital pathology (DP) is an image-based environment that enables the acquisition, management, and interpretation of pathology information generated from whole slide images (WSI). The potential advantages of DP include telepathology, digital consultation and slide sharing, pathology education, indexing and retrieval of cases, and the use of automated image analysis [1–4]. Digital pathology was enabled by recent technological advances in WSI systems, which can digitize microscope slides at high resolution in an automated manner. The FDA has determined that DP is subject to regulatory oversight. Validation studies can identify possible limitations for specified intended use of DP and provide necessary information for regulatory approval of DP devices for that intended use. Recent guidelines for validation of WSI from the College of American Pathologists state that “validation is recommended to determine that a pathologist can use a WSI system to render an accurate diagnosis with the same or better level of ease as with a traditional microscope and without interfering artifacts or technological risks
References
[1]
L. Pantanowitz, P. N. Valenstein, A. J. Evans, et al., “Review of the current state of whole slide imaging in pathology,” Journal of Pathology Informatics, vol. 2, pp. 1–10, 2011.
[2]
R. S. Weinstein, A. R. Graham, L. C. Richter et al., “Overview of telepathology, virtual microscopy, and whole slide imaging: prospects for the future,” Human Pathology, vol. 40, no. 8, pp. 1057–1069, 2009.
[3]
S. Al-Janabi, A. Huisman, and P. J. Van Diest, “Digital pathology: current status and future perspectives,” Histopathology, vol. 61, no. 1, pp. 1–9, 2012.
[4]
P. N. Furness, “The use of digital images in pathology,” Journal of Pathology, vol. 183, pp. 253–263, 1997.
[5]
L. Pantanowitz, J. H. Sinard, W. H. Henricks, et al., “Validating whole slide imaging for diagnostic purposes in pathology: guideline from the College of American Pathologists Pathology and Laboratory Quality Center,” Archives of Pathology & Laboratory Medicine, vol. 137, pp. 1710–1722, 2013.
[6]
K. E. Brick, J. C. Sluzevich, M. A. Cappel, D. J. Dicaudo, N. I. Comfere, and C. N. Wieland, “Comparison of virtual microscopy and glass slide microscopy among dermatology residents during a simulated in-training examination,” Journal of Cutaneous Pathology, vol. 40, no. 9, pp. 807–811, 2013.
[7]
S. S. P. Costello, D. J. Johnston, P. A. Dervan, and D. G. O'Shea, “Development and evaluation of the virtual pathology slide: a new tool in telepathology,” Journal of Medical Internet Research, vol. 5, no. 2, article e11, 2003.
[8]
J. R. Gilbertson, J. Ho, L. Anthony, D. M. Jukic, Y. Yagi, and A. V. Parwani, “Primary histologic diagnosis using automated whole slide imaging: A Validation Study,” BMC Clinical Pathology, vol. 6, article 4, 2006.
[9]
H. O. Helin, M. E. Lundin, M. Laakso, J. Lundin, H. J. Helin, and J. Isola, “Virtual microscopy in prostate histopathology: simultaneous viewing of biopsies stained sequentially with hematoxylin and eosin, and α-methylacyl-coenzyme A racemase/p63 immunohistochemistry,” Journal of Urology, vol. 175, no. 2, pp. 495–499, 2006.
[10]
B. McCullough, X. Ying, T. Monticello, and M. Bonnefoi, “Digital microscopy imaging and new approaches in toxicologic pathology,” Toxicologic Pathology, vol. 32, no. 2, pp. 49–58, 2004.
[11]
B. Molnar, L. Berczi, C. Diczhazy et al., “Digital slide and virtual microscopy based routine and telepathology evaluation of routine gastrointestinal biopsy specimens,” Journal of Clinical Pathology, vol. 56, no. 6, pp. 433–438, 2003.
[12]
G. Puppa, C. Senore, K. Sheahan et al., “Diagnostic reproducibility of tumour budding in colorectal cancer: a multicentre, multinational study using virtual microscopy,” Histopathology, vol. 61, no. 4, pp. 562–575, 2012.
[13]
D. S. A. Sanders, H. Grabsch, R. Harrison et al., “Comparing virtual with conventional microscopy for the consensus diagnosis of Barrett's neoplasia in the AspECT Barrett's chemoprevention trial pathology audit,” Histopathology, vol. 61, no. 5, pp. 795–800, 2012.
[14]
D. S. Weinberg, F. A. Allaert, P. Dusserre et al., “Telepathology diagnosis by means of digital still images: An International Validation Study,” Human Pathology, vol. 27, no. 2, pp. 111–118, 1996.
[15]
A. R. Jara-Lazaro, T. P. Thamboo, M. Teh, and P. H. Tan, “Digital pathology: exploring its applications in diagnostic surgical pathology practice,” Pathology, vol. 42, no. 6, pp. 512–518, 2010.
[16]
B. H. Hall, M. Ianosi-Irimie, P. Javidian, W. Chen, S. Ganesan, and D. J. Foran, “Computer-assisted assessment of the Human Epidermal Growth Factor Receptor 2 immunohistochemical assay in imaged histologic sections using a membrane isolation algorithm and quantitative analysis of positive controls,” BMC Medical Imaging, vol. 8, article 11, 2008.
[17]
A. S. Joshi, G. M. Sharangpani, K. Porter et al., “Semi-automated imaging system to quantitate Her-2/neu membrane receptor immunoreactivity in human breast cancer,” Cytometry Part A, vol. 71, no. 5, pp. 273–285, 2007.
[18]
I. Skaland, I. ？vestad, E. A. M. Janssen et al., “Comparing subjective and digital image analysis HER2/neu expression scores with conventional and modified FISH scores in breast cancer,” Journal of Clinical Pathology, vol. 61, no. 1, pp. 68–71, 2008.
[19]
H.-A. Lehr, T. W. Jacobs, H. Yaziji, S. J. Schnitt, and A. M. Gown, “Quantitative evaluation of HER-2/neu status in breast cancer by fluorescence in situ hybridization and by immunohistochemistry with image analysis,” American Journal of Clinical Pathology, vol. 115, no. 6, pp. 814–822, 2001.
[20]
K. A. Matkowskyj, D. Schonfeld, and R. V. Benya, “Quantitative immunohistochemistry by measuring cumulative signal strength using commercially available software Photoshop and Matlab,” Journal of Histochemistry and Cytochemistry, vol. 48, no. 2, pp. 303–311, 2000.
[21]
Y. Hatanaka, K. Hashizume, Y. Kamihara et al., “Quantitative immunohistochemical evaluation of HER2/neu expression with HercepTest in breast carcinoma by image analysis,” Pathology International, vol. 51, no. 1, pp. 33–36, 2001.
[22]
H. Masmoudi, S. M. Hewitt, N. Petrick, K. J. Myers, and M. A. Gavrielides, “Automated quantitative assessment of HER-2/neu immunohistochemical expression in breast cancer,” IEEE Transactions on Medical Imaging, vol. 28, no. 6, pp. 916–925, 2009.
[23]
R. Mofidi, R. Walsh, P. F. Ridgway et al., “Objective measurement of breast cancer oestrogen receptor status through digital image analysis,” European Journal of Surgical Oncology, vol. 29, no. 1, pp. 20–24, 2003.
[24]
K. A. DiVito, A. J. Berger, R. L. Camp, M. Dolled-Filhart, D. L. Rimm, and H. M. Kluger, “Automated quantitative analysis of tissue microarrays reveals an association between high Bcl-2 expression and improved outcome in melanoma,” Cancer Research, vol. 64, no. 23, pp. 8773–8777, 2004.
[25]
A. S. Elhafey, J. C. Papadimitriou, M. S. El-Hakim, A. I. El-Said, B. B. Ghannam, and S. G. Silverberg, “Computerized image analysis of p53 and proliferating cell nuclear antigen expression in benign, hyperplastic, and malignant endometrium,” Archives of Pathology and Laboratory Medicine, vol. 125, no. 7, pp. 872–879, 2001.
[26]
B. Keller, W. Chen, and M. A. Gavrielides, “Quantitative assessment and classification of tissue-based biomarker expression with color content analysis,” Archives of Pathology and Laboratory Medicine, vol. 136, no. 5, pp. 539–550, 2012.
[27]
S. Wang, M. H. Saboorian, E. P. Frenkel et al., “Assessment of HER-2/neun status in breast cancer: Automated Cellular Imaging System (ACIS)-assisted quantitation of immunohistochemical assay achieves high accuracy in comparison with fluorescence in situ hybridization assay as the standard,” American Journal of Clinical Pathology, vol. 116, no. 4, pp. 495–503, 2001.
[28]
A. Ciampa, B. Xu, G. Ayata et al., “HER-2 status in breast cancer: correlation of gene amplification by FISH with immunohistochemistry expression using advanced cellular imaging system,” Applied Immunohistochemistry and Molecular Morphology, vol. 14, no. 2, pp. 132–137, 2006.
[29]
K. Bloom and D. Harrington, “Enhanced accuracy and reliability of HER-2/neu immunohistochemical scoring using digital microscopy,” American Journal of Clinical Pathology, vol. 121, no. 5, pp. 620–630, 2004.
[30]
O. W. Tawfik, B. F. Kimler, M. Davis et al., “Comparison of immunohistochemistry by automated cellular imaging system (ACIS) versus fluorescence in-situ hybridization in the evaluation of HER-2/neu expression in primary breast carcinoma,” Histopathology, vol. 48, no. 3, pp. 258–267, 2006.
[31]
D. Lüftner, P. Henschke, A. Kafka et al., “Discordant results obtained for different methods of HER-2/neu testing in breast cancer—a question of standardization, automation and timing,” International Journal of Biological Markers, vol. 19, no. 1, pp. 1–13, 2004.
[32]
M. Cregger, A. J. Berger, and D. L. Rimm, “Immunohistochemistry and quantitative analysis of protein expression,” Archives of Pathology and Laboratory Medicine, vol. 130, no. 7, pp. 1026–1030, 2006.
[33]
A. Nassar, C. Cohen, S. S. Agersborg et al., “Trainable immunohistochemical HER2/neu image analysis: A Multisite Performance Study using 260 breast tissue specimens,” Archives of Pathology and Laboratory Medicine, vol. 135, no. 7, pp. 896–902, 2011.
[34]
C. M. Conway, D. O'Shea, S. O'Brien et al., “The development and validation of the Virtual Tissue Matrix, a software application that facilitates the review of tissue microarrays on line,” BMC Bioinformatics, vol. 7, article 256, 2006.
[35]
C. Conway, L. Dobson, A. O'Grady, E. Kay, S. Costello, and D. O'Shea, “Virtual microscopy as an enabler of automated/quantitative assessment of protein expression in TMAs,” Histochemistry and Cell Biology, vol. 130, no. 3, pp. 447–463, 2008.
[36]
A. Nassar, C. Cohen, M. Albitar, et al., “Reading immunohistochemical slides on a computer monitor—A Multisite Performance Study using 180 HER2-stained breast carcinomas,” Applied Immunohistochemistry & Molecular Morphology, vol. 19, no. 3, pp. 212–217, 2011.
[37]
Y. Kondo, T. Iijima, and M. Noguchi, “Evaluation of immunohistochemical staining using whole-slide imaging for HER2 scoring of breast cancer in comparison with real glass slides,” Pathology International, vol. 62, no. 9, pp. 592–599, 2012.
[38]
J. L. Fine, D. M. Grzybicki, R. Silowash et al., “Evaluation of whole slide image immunohistochemistry interpretation in challenging prostate needle biopsies,” Human Pathology, vol. 39, no. 4, pp. 564–572, 2008.
[39]
P. J. Van Diest, P. Van Dam, S. C. Henzen-Logmans et al., “A scoring system for immunohistochemical staining: consensus report of the task force for basic research of the EORTC-GCCG,” Journal of Clinical Pathology, vol. 50, no. 10, pp. 801–804, 1997.
[40]
M. A. Gavrielides, B. D. Gallas, P. Lenz, A. Badano, and S. M. Hewitt, “Observer variability in the interpretation of HER2/neu immunohistochemical expression with unaided and computer-aided digital microscopy,” Archives of Pathology and Laboratory Medicine, vol. 135, no. 2, pp. 233–242, 2011.
[41]
S. V. Makkink-Nombrado, J. P. A. Baak, L. Schuurmans, J.-W. Theeuwes, and T. Van der Aa, “Quantitative immunohistochemistry using the CAS 200/486 image analysis system in invasive breast carcinoma: A Reproducibility Study,” Analytical Cellular Pathology, vol. 8, no. 3, pp. 227–245, 1995.
[42]
S. M. Hewitt, “Design, construction, and use of tissue microarrays,” in Protein Arrays: Methods and Protocols, E. T. Fung, Ed., pp. 61–72, Humana Press, 2004.
[43]
J. Kononen, L. Bubendorf, A. Kallioniemi et al., “Tissue microarrays for high-throughput molecular profiling of tumor specimens,” Nature Medicine, vol. 4, no. 7, pp. 844–847, 1998.
[44]
M. Lejeune, J. Jaén, L. Pons et al., “Quantification of diverse subcellular immunohistochemical markers with clinicobiological relevancies: validation of a new computer-assisted image analysis procedure,” Journal of Anatomy, vol. 212, no. 6, pp. 868–878, 2008.
[45]
A. C. Wolff, M. E. H. Hammond, J. N. Schwartz et al., “American Society of Clinical Oncology/College of American Pathologists guideline recommendations for human epidermal growth factor receptor 2 testing in breast cancer,” Journal of Clinical Oncology, vol. 25, no. 1, pp. 118–145, 2007.
[46]
T. Hasegawa, Standardization of Ki-67 Immunohistochemical Staining for Diagnosing Grade of Gastrointestinal Stromal Tumor (GIST), 2008.
[47]
J. Gerdes, U. Schwab, H. Lemke, and H. Stein, “Production of a mouse monoclonal antibody reactive with a human nuclear antigen associated with cell proliferation,” International Journal of Cancer, vol. 31, no. 1, pp. 13–20, 1983.
[48]
M. Dowsett, T. O. Nielsen, R. A'Hern et al., “Assessment of Ki67 in Breast Cancer: recommendations from the international Ki67 in breast cancer working Group,” Journal of the National Cancer Institute, vol. 103, no. 22, pp. 1656–1664, 2011.
[49]
E. D. Hsi and R. R. Tubbs, “Guidelines for HER2 testing in the UK,” Journal of Clinical Pathology, vol. 57, no. 3, pp. 241–242, 2004.
[50]
M. Kendall, Rank Correlation Methods, Charles Griffin and Co. Limited, London, UK, 1948.
[51]
R. F. Woolson and W. R. Clarke, Statistical Methods for the Analysis of Biomedical Data, John Wiley & Sons, New York, NY, USA, 1987.
[52]
T. W. Bauer, L. Schoenfield, R. J. Slaw, L. Yerian, Z. Sun, and W. H. Henricks, “Validation of whole slide imaging for primary diagnosis in surgical pathology,” Archives of Pathology and Laboratory Medicine, vol. 137, no. 4, pp. 518–524, 2013.
[53]
W. S. Campbell, S. M. Lele, W. W. West, A. J. Lazenby, L. M. Smith, and S. H. Hinrichs, “Concordance between whole-slide imaging and light microscopy for routine surgical pathology,” Human Pathology, vol. 43, no. 10, pp. 1739–1744, 2012.
[54]
W. H. Henricks, “Evaluation of whole slide imaging for routine surgical pathology: looking through a broader scope,” Journal of Pathology Informatics, vol. 3, article 39, 2012.
