Devices and Methods of Measurements 2017, vol. 8, no. 3, pp. 271–278 Sutkowski M., Saukova Ya. Приборы и методы измерений 2017. – Т. 8, № 3. – С. 271–278 Сутковски М., Савкова Е.Н. Адрес для переписки: Савкова Е.Н. Белорусский национальный технический университет, пр. Независимости, 65, г. Минск 220013, Беларусь e-mail: evgeniya-savkova@yandex.ru Address for correspondence: Saukova Ya. Belarusian National Technical University, Nezavisimosty Ave., 65, Minsk 220013, Belarus e-mail: evgeniya-savkova@yandex.ru Для цитирования: Sutkowski M., Saukova Ya. Research of digital camera dynamic range on the imaging processing basis. Приборы и методы измерений. 2017. – Т. 8, № 3. С. 271–278. DOI: 10.21122/2220-9506-2017-8-3-271-278 For citation: Sutkowski M., Saukova Ya. [Research of Digital Camera Dynamic Range on the Imaging Processing Basis]. Devices and Methods of Measurements. 2017, vol. 8, no. 3, рр. 271–278. DOI: 10.21122/2220-9506-2017-8-3-271-278 Research of Digital Camera Dynamic Range on the Imaging Processing Basis Sutkowski M.1, Saukova Ya.2 1Institute of Microelectronics and Optoelectronics, Warsaw University of Technology, 15/19 Nowowiejska Str., Warsaw 00-665, Poland 2Belarusian National Technical University, Nezavisimosty Ave., 65, Minsk 220013, Belarus Received 02.02.2017 Accepted for publication 23.08.2017 Abstract Digital images provide to determine photometric and colorimetric properties of objects subject to valida- tion all elements of a measuring channel (digital camera, software, display) and solve the problem of their limited dynamic ranges. The aim of the study was to explore the dynamic range of a digital camera for use in photometric and colorimetric measurements. The Laboratory of Photonics at the Institute of Microelectronics and Optoelectronics (Warsaw Technical University, Poland) conducted a comparative experiment to determine the threshold of sensitivity, linearity and range of application the digital camera. Color target sets with certified brightness and chromaticity were created at the terminals and recorded with a digital camera with different exposure times. The authors propose a method to extend the dynamic range of a digital camera for red, green and blue color channel of intensities by pairing the calibration dependencies, and determine the true brightness and color of a point on the object by calculation. Calibration dependencies (triads) of digital camera for red, green and blue color channels intensities were constructed. These dependences allow determining lower and upper bounds of the dynamic range. Each triad has a form of the hysteresis loop. The experiment showed that the accuracy of this method is ± 3–5 %. Keywords: digital camera, dynamic range, imaging. DOI: 10.21122/2220-9506-2017-8-3-271-278 271 Адрес для переписки: Савкова Е.Н. Белорусский национальный технический университет, пр. Независимости, 65, г. Минск 220013, Беларусь e-mail: evgeniya-savkova@yandex.ru Address for correspondence: Saukova Ya. Belarusian National Technical University, Nezavisimosty Ave., 65, Minsk 220013, Belarus e-mail: evgeniya-savkova@yandex.ru Для цитирования: Sutkowski M., Saukova Ya. Research of digital camera dynamic range on the imaging processing basis. Приборы и методы измерений. 2017. – Т. 8, № 3. С. 271–278. DOI: 10.21122/2220-9506-2017-8-3-271-278 For citation: Sutkowski M., Saukova Ya. [Research of Digital Camera Dynamic Range on the Imaging Processing Basis]. Devices and Methods of Measurements. 2017, vol. 8, no. 3, рр. 271–278. DOI: 10.21122/2220-9506-2017-8-3-271-278 Devices and Methods of Measurements 2017, vol. 8, no. 3, pp. 271–278 Sutkowski M., Saukova Ya. Приборы и методы измерений 2017. – Т. 8, № 3. – С. 271–278 Сутковски М., Савкова Е.Н. Исследование динамического диапазона цифровой камеры на основе технологий обработки цифровых изображений Сутковский М.1, Савкова Е.Н.2 1Институт микроэлектроники и оптоэлектроники, Варшавский политехнический университет, ул. Нововейская 15/19, Варшава 00-665, Польша 2Белорусский национальный технический университет, пр. Независимости, 65, г. Минск 220013, Беларусь Поступила 02.02.2017 Принята к печати 23.08.2017 Цифровые изображения позволяют определять фотометрические и колориметрические свойства объектов при условии валидации всех элементов измерительного канала (цифровой камеры, про- граммного обеспечения, дисплея) и решения проблемы ограничения их динамических диапазонов. Целью работы являлось исследование динамического диапазона цифровой камеры для ее дальнейше- го использования в фотометрических и колориметрических измерениях. Лаборатория фотоники в Институте микроэлектроники и оптоэлектроники (Варшавский техниче- ский университет, Польша) проводила сличительный эксперимент для определения порога чувстви- тельности, линейности и диапазона применения цифровой камеры. Цветовые мишени были созданы на дисплее в виде аттестованных цветовых однородных полей и регистрировались с помощью цифро- вой камеры с пошагово увеличивающимся временем экспозиции, а затем осуществлялась обработка изображений и калибровка камеры в красном, зеленом и синем цветовых каналах для получения ка- либровочных зависимостей. Авторами предложен метод расширения динамического диапазона циф- ровой камеры для красного, зеленого и синего цветовых каналов интенсивностей посредством сопря- жения градуировочных зависимостей и определения истинной яркости точки на объекте расчетным путем. Полученные калибровочные зависимости (триады) имели форму петли гистерезиса и позволили расчетным путем расширить динамический диапазон. Эксперименты показали, что точность данно- го метода составляет ± 3–5 %. Ключевые слова: цифровая камера, диапазон, обработка изображений. DOI: 10.21122/2220-9506-2017-8-3-271-278 272 Devices and Methods of Measurements 2017, vol. 8, no. 3, pp. 271–278 Sutkowski M., Saukova Ya. Приборы и методы измерений 2017. – Т. 8, № 3. – С. 271–278 Сутковски М., Савкова Е.Н. Introduction Professional and semi-professional digital cam- eras including thermal imagers can be used as the universal receivers of photonic information general- ly in visible range. At the same time the current level of digital technology provides opportunities to apply digital cameras during the control and diagnostics of objects at all stages of their life cycle. Since the characteristics of the cameras are not the object of obligatory acknowledgement of conformity manu- facturers may indicate exaggerated values. Therefore the need arises to obtain reliable and accurate results of control and testing. As it is known digital cameras have limited dynamic range to reproduce the inten- sity (about 60 dB) while it is necessary to record the brightness over a wider range. Standard methods for solving this problem are based on the dynamic range compression of the plots. Currently compression procedures are used to reproduce image details that are situated in the re- gion closed to the saturation matrix photodetector [1]. So it is necessary to move the point A into the region of the inflection of light characteristics of the signal, which produces a brighter light for compress- ing the amplitude (interval E1–E2, Figure 1a). The point A limits a dynamic range of a camera. A linear dependence of Uout(E) is modified by reducing the slope of the transfer characteristics in the upper part. Inflection points and slope angle affect the light flux of the brightest objects. Therefore, the possibility exists to compress the excess dynamic range (bright areas don’t merge) re- ducing the slope of the transfer characteristics. Also the nonlinear compression can be applied. It is nec- essary to make a plot with the non-linear inflection point in the transition mode overload more smoothly (Figure 1b). In this case, the slope of characteristics varies in proportion to the number of parts of the light phase that would be detected. Also more careful study of details in blown highlights can be achieved by shifting the point of inflection at a constant pole tilt, keeping the signal from vivid seats below the limitation level (Figure 1c). These methods allow only a subjective improve the reproduction of bright details of the image. However, when it is necessary to determine illuminance (brightness) in different points of the object simultaneously a principally new approach is needed. Aspects of cameras range compression are dis- cussed in [2] and [3]. Contrast extension leads to a «splitting» of quantization levels that appear in the histogram as a comb. The combined method of dynamic range controlling by software tools in [4] is described. The method is based on the dynamic range stretching in dark areas and compression in the blown highlights area of image. However these technologies are used only for visual improvement of images quality and don’t allow determining the intensity significantly. They consider the patterns and effects of visual perceptions such as light and color adaptation, the contrast and the contours of fine details etc. [5, 6]. a b c Figure 1 – Electronic methods of dynamic range com- pression: a – linear compression with the displacement of the point of saturation down; b – nonlinear compression after the inflection point for different contrast; c – options linear compression with the same contrast This method is based on repeated digital re- garding of extended objects and digital images 273 Devices and Methods of Measurements 2017, vol. 8, no. 3, pp. 271–278 Sutkowski M., Saukova Ya. Приборы и методы измерений 2017. – Т. 8, № 3. – С. 271–278 Сутковски М., Савкова Е.Н. 274 processing. The brightness of the investigated area of the digital image is compared with the bright- ness of the certified sources available in the long term picture. The problem of determining the il- lumination at the object is solved by expanding the dynamic range of optical measurement software. The method developed in the Scientific Research Laboratory of optics and electronics instrument making (Belarusian National Technical University, Minsk, Belarus) [7] and tested in the Laboratory of Photonics of the Institute of Optoelectronics and Microelectronics (Warsaw University of Technol- ogy, Warsaw, Poland) allows doing photometric measurements in real time with the specified char- acteristics error. The main idea of this method is to do series of digital images of the same object and reference samples with different exposures and process the images in RGB and RAW for- mats. Format RGB provides the ability to define intensity in red, green, and blue color channels of images [8]. There are different technologies of ob- jects multiple registrations with increasing the time of exposure for creating of HRDI images [9, 10]. But they apply in computer games and design. And the technology which is described provides an op- portunity to obtain quantitative data of photometric and colorimetric parameters of the object. The experimental research of a digital cam- era dynamic range To ensure metrological traceability the sets of achromatic and chromatic samples were done on the colorimetric and calibrated display (look Table). The values was applied as five different values repre- sented in standard 8-bit digital number. The samples were then characterized by use of spectrophotometer as real photometric values. The readings were done for 3 different areas and then were averaged. In the Figure 2 there are shown chromacity values of the samples (x, y) plotted on the CIE 1931 chart. Imaging system callibration procedure is neces- sary to know real lighting parameters of the examined area of the light panel. The real photometric param- eters was determined by Minolta CS-100A non-con- tact chroma meter. Each measurement was achieved by averaging of three readings done at three neigh- bouring points of the examined area of the panel. Ev- ery measurement was done not more than 10 s after respective image acquisition (see further paragraph). Table Measured photometric values of the total 17 samples: Y – luminance, x, y – chrominance The hardware set-up of the measuring sys- tem proposed in this paper is targeted to propose portable and flexible color measurement system. It assumes to use the best possible commercially available image acquisition system with best pos- sible image quality. For measuring application it should offer possibility to control all steps of the image registration procedure, including flex- ible and fully controlled image data processing. The next assumption is to use area sensor instead of widely used spot measurement units. It should can work quite everywhere, even there where en- vironmental conditions elliminates use of special- ized spectrophotometers. Fullfilling these factors we can offer a system which allow to examine light panels or large illuminating surfaces com- prehensively. Regarding to mentioned reasons we decided to use commercial photographic camera with high-quality imaging sensor as the image ac- quision unit. The characterization of the imaging hardware for measurement requirements is highly needed. Also we have to use image processing and analysis without (or as low as possible) influence on the data taken by the sensor. Devices and Methods of Measurements 2017, vol. 8, no. 3, pp. 271–278 Sutkowski M., Saukova Ya. Приборы и методы измерений 2017. – Т. 8, № 3. – С. 271–278 Сутковски М., Савкова Е.Н. 275 Figure 2 – Chromacity values of the samples used in the experiment plotted on the CIE 1931 graph. The luminosity is ommitted As the axquisition unit a Nikon D610 camera was chosen. It has a CMOS (Complementary Metal Oxide Semicondutor) large area imaging sensor of- fering active area of 35,9 × 24 mm and total pixel count 24,7 million. It is equipped with standard color filter array of Bayer RGB type. The imaging data can be represented as a digital number with 14-bit depth. The final image can be recorded at resolution of 6016 × 4016 pixels [http:/ HYPERLINK «http:// imaging.nikon.com/lineup/dslr/d610/spec.htm»/ imaging.nikon.com/lineup/dslr/d610/spec.htm]. It works with standard imaging lens AF-S Nikkor 24- 85mm f/3,5−4,5G ED VR. It offers maximum aper- ture of f/3,5 at f = 24 mm and f/4,5 at 85 mm. It's optical construction consists of 16 optical elemnts in 11 groups including 1 ED (low dispertion) glass and 3 aspherical lens elements. This lens was chosen due to the very good response at the center of the image. Its MTF (modulation transfer function) curves show near perfect characteristics for low spatial frequen- cies (10 l/mm for both tangential and sagital lines) at the circle with 10 mm diameter. It has very flat characteristic which minimum value is approx. 0,98 [http://imaging.nikon.com/lineup/lens/zoom/nor- malzoom/af-s_24-85mmf_35-45g_ed_vr/index.htm]. The goal of the present experiment is to find an answer of the camera for different exposure values to find a linear part of the registration curve. To perform these tests a reference light panel is needed. In prac- tice the ligh parameters of the image applied to the reference panel utilized for these measurements has to be well known. As a reference light panel a colori- metric wide gamut LCD monitor was used, in detail Eizo ColorEdge CG245W offering 100 % of sRGB space coverage. It was driven by AMD HD6990 graphic card with correction LUT applied. It is achieved by color callibration procedure with use of datacolor Spyder 3 Elite callibrator. For the tests we used only a small part of the panel at the center to avoid (or minimalise) any angular dependancies and backlight illumination non-uniformity. Measured area had a dimensions of approx. 100 × 100 mm. Measuring set-up is shown on Figure 3. Object distance was fixed at the value of 700 mm, optical axis was perpemdicular to the panel plane. Imag- ing lens was set to the focal distance f = 50 mm and middle aperture f/8. The acquired images were saved as a NED (RAW-type) file with 14-bit depth. All in- camera processing was ommitted and conversion to tri-chromatic form (demosaicing) was performed with «zero» parameters in commercial Adobe Light- room 4.4 software. The signal amplification in cam- era was set to eqivalent of ISO 100. Exposure was measured for middle-gray (128, 128, 128) values and correcponds to exposure time t = 1/4 s. The experi- ment consists of series exposures within the range of relative EV from −5 to +4 with step od 1/3 what correspons exposure time range from 1/125 to 4 s. Figure 3 – Experimental set-up for characterization of the digital camera The goal of the experiment was to determine linear part of the camera characteristic. The tar- get was changed in luminosity and in chromacity in five steps from value 0 to 255 (noted as Gr for theoretically neutral grays, R, G, B – for the basic red, green, blue chroma respectively and numbered from 1 to 5 – as in the Table 1). For each constant target the series of vary exposure is done. Acquired image was converted with use of Adobe Camera RAW from raw data (a 14-bit uncompressed NEF file saved by the camera) to tri-chromatic form and Devices and Methods of Measurements 2017, vol. 8, no. 3, pp. 271–278 Sutkowski M., Saukova Ya. Приборы и методы измерений 2017. – Т. 8, № 3. – С. 271–278 Сутковски М., Савкова Е.Н. saved as 16-bit TIFF file. Measurements of the im- age parameters was done by Image J software af- ter cropping of the ROI (Region Of Interest) with the size 585 × 585 pixels at the center of the image. As result an averaged luminance values of the pixels in three channels are taken. The experimental results are shown oh Figure 4 for gray channel (Gr), R, G, B channels respectively. 276 Figure 4 – Experimental curves of the digital camera range obtained by recording the self-luminous objects with differ- ent exposure time: red colour − R; green − G; blue − B The figure 4 shows that the triad of characteristic dependences have the shape of the hysteresis loop. Every dependence is not necessarily linear within the active range area. Dependences have different angles of inclination. The basic concept of measurements In information systems the process of data con- vertion assumes the operations of filtering, sampling, quantization, encoding, decoding, postfiltration etc. For realization of measurements it is necessary to ad- here to the principle of comparison with a measure. This is a difficult task because there are many diffi- culties with ensuring of the metrological traceability of results, given the diversity of hardware and soft- ware, as well as the use of computer ranking scales for the measurement of the intensities of digital im- ages. In addition, the differences and limitations of transmitting devices gamuts prevent the obtaining of reliable measuring information. There are many dif- ficulties arise in solving measurement problems as- sociated with determining the luminance and color characteristics of objects based on processing their digital images. These difficulties are associated with objective limitations of color coverage, and almost all economic ranges of the transmission device, and establishing the corresponding nominal level of quantization in the color channels as a result of the normalization of the digital image at the brightest point. These problems can be solved by the correct application of available hardware and software ac- cording to the proposed method of measurements. The proposed method is based on digital registration of standard objects with different values of exposure time and digital images processing. The metrological traceability of measurement results is provided by reference to the initial samples (called by authors as a line sources with equal brightness) which are non- point sources of light acting as the measures in mea- surements [2]. Within each of the line light sources should vary in brightness with a step depending on the specific measuring task. Thus we can construct a set of calibration dependencies for each source showing the dependence of numerical representation of intensity values (y-axis) of the color channels from the exposure H (x-axis) – R(t), G(t), B(t), an example of which is given in Figure 5 [3]. Further accord- Devices and Methods of Measurements 2017, vol. 8, no. 3, pp. 271–278 Sutkowski M., Saukova Ya. Приборы и методы измерений 2017. – Т. 8, № 3. – С. 271–278 Сутковски М., Савкова Е.Н. ing to mated for each consecutive pairs of sources for example one line for the one time. The values of intensities of R, G, B for exposure H pending solid lines in the figure on the y-axis. The exposure H is understand as a product of incident beam intensity and the exposure time (time when photon interact with the imager structure): (1) where I – denotes intensity of the photon beam inci- dent onto imager; texp – denotes exposure time. Figure 5 – The theoretical calibration curves of digital im- ages color channels As can be seen in Figure 5, in the red channel, the intensity of Rn lies in the region of saturation. This is due to limitations in the dynamic ranges of imagers of digital cameras and therefore, makes it impossible to obtain valid measurement information. The offset of intensity in the green channel G01 in the area of imager's noise may occur when switching to the image with very low exposure level. Therefore to base on the principle of linearity of the calibration characteristics of the measurement method we can determine on the y-axis conditional point R'n outside physical dynamic range and carry out the normaliza- tion of the intensities of R, G, B, adapting them to new values, introducing appropriate broadening co- efficients directly in a standardized model transfor- mation color spaces RGB→XYZ [4]: (2) (3) (4) where X, Y, Z – coordinates of the color space XYZ; anm – the standardized weighting of intensities R, G, B, kR, kG, kB – expanding the coefficients calculated by the formulas [4]: (5) The highest value of the digital intensity image is given in the denominators. It is dependant on the actual digital quantization level, the highest number is repesented by the 2n where n is the number of bits used. There is 255 gradations of intensity in the case of 8-bit digital representation. New transformed lines of intensity in the color channel and G are shown in Figure 6 by the stipple, but at the saturation level all data is lost. The calculations of the chromaticity coordinates produced by the formulas (2)–(4). Thus it was established experimentally that color coordi- nates triads of the same reference point on the object in images made with different exposures with a giv- en confidence level correspond to the same chroma. If at least one channel reach its saturation level we can predict linear continuity of the constant chroma value and to make reading over this level. It was also found that the greatest variance of the intensity val- ues observed in the blue channel. The averaged vari- ance of determination of chromaticity coordinates is from 2.8 to 3.1 units approximately. Reliable de- tection of object real color coordinates depends on the nominal values of quantization steps, playing re- quired scales conventional digital image. The nomi- nal level of quantization represent the ellipsoids of different volumes in three-dimensional color spaces, and the minimum (definitely) uncertainty ellipses is determined by the Mac-Adam [5]. However the con- stant value of the nominal level of quantization can be maintained in the RGB color space by the math- ematical methods of the increment intensity. There is a possibility to define the color characteristics of self-luminous objects and to enter correction factors for secondary emitters by applying models of chro- matic adaptation Von-Crease, CIECAM02, Fairchild and others [5, 6]. Data conversion must be performed for each color channel. It should be consider surface proper- ties (reflectance) and parameters of shooting in the calculations. The Figure 6 explains the essence of the developed model. Different signals N’01, N’02, N ’’ 01, N’’02, etc. correspond to the reference brightness val- ues L1, L2 on the abscissa. The linear dependences can be built on control points of the photodetector dynamic range before the transition to the stage of saturation. By pairing the dependency mathematical- ly with sufficient accuracy for practice it is possible to calculate intensity values of points on the object in three color channels, thus broadening the range of two-dimensional colorimetric measurements. 277 H I ty = ⋅ exp , X a k R a k G a k B R G B = + +11 12 13 , Y a k R a k G a k B R G B = + +21 22 23 , Z a k R a k G a k B R G B = + +31 32 33 , k RR = ' , 255 k GG = ' , 255 k BB = ' . 255 Devices and Methods of Measurements 2017, vol. 8, no. 3, pp. 271–278 Sutkowski M., Saukova Ya. Приборы и методы измерений 2017. – Т. 8, № 3. – С. 271–278 Сутковски М., Савкова Е.Н. Figure 6 – The extension of detected brightness dynamic range of the images made with different exposures (differ- ent exposure time: t1 < t2 < t3 when other parameters are constant) The goal of the experiment was to study char- acteristics of the digital cameras dynamic range for their validation in the information channel. Lower and upper limits of the range, the linearity of color channels were studied during the experiment. The experiment plan included the following stages: 1) to do the sets of standard sources (achromatic and chromatic samples) on the display; 2) to measure the brightness and color settings of samples with the help of spectrophotometer; 3) to take digital images of samples with differ- ent exposure times; 4) to do image processing for determination the intensities in RGB channels and the chromaticity co- ordinates; 5) to obtain the dependences (calibration charac- teristics of camera); 6) to investigate the characteristics of dynamic range. Conclusion 1. The resulting diagrams (curves) are represent- ing the characteristic of the registration unit (a cam- era with the lens). Each registration curve has a linear section where the aerial measurements are possible. 2. General shape of the registration curve show strong nonlinearity where it cannot be callibrated for measurements. 3. The chroma responses of the camera shows significat differencies in spectral characteristics of the RGB Bayer filters used in the image detector of the camera in comparision to reference values (mea- sured by chroma meter). 4. On the B channel there are some errors on the curve visible. This is due to the very low level of lu- mimance of the blue image displayed on the monitor used in this experiment. 5. With use of aerial sensor we can make averag- ing measurements of the area light panels or to mea- sure few different sources simultaneously. 6. Due to the possibility of exposure adjuste- ments we can offer a very wide range of light inten- sity measurements. 7. In the paper the dedicated demosaicing pro- cess for the camera we used is ommitted. When ap- plied it can allow to produce more precise results and the chroma response of the camera can be calibrated with use of reference source. This technology can be used as a basis for cali- bration and validation methods of cameras, displays and other controlled objects too. References 1. Ward P. Working with digital cameras: lessons learned operator skills. Moscow, Mir Publ., 2001, 301 p. (in Russian). 2. Lynch R. The Usage of Adobe Photoshop 6. Spe- cial edition, Moscow, Publishing house «Williams», 2001, 880 p. (in Russian). 3. Reihard E., Ashikhmin M., Gooch B. and Shirley P. «Colour Transfer between Images». YIEEE Computer Graphics and Application, Appied perception, September/ October 2001, pp. 34–41. 4. Goden J. Colorimetric in the rendering perfor- mance. Moscow, Technosphere Publ., 2008, 328 p. (in Russian). 5. Rudakov P.I., Safonov V.I. Processing of signals and images. MATLAB 5х. Moscow, Technosphere Publ., 1998, 503 p. (in Russian). 6. Margulis D. Photoshop for professionals. Minsk, Popurry Publ., 1999, 304 p. (in Russian). 7. Zuikov I., Saukova Y. The Method of measuring the intensity of the light source. Patent G 01 J 1/00 By 10453. Application № а20051233/ Registered in the State register. 8. Süsstrunk S., Buckley R., Swen St. Standard RGB Color Spaces. Laboratory of audio-visual Communica- tion (EPFL), Xerox Architecture Center, Apple Computer Lausanne, Switzerland; Webster, NY; Cupertino, CA. Available at: https://infoscience.epfl.ch/record/34089/ files/SusstrunkBS99.pdf 9. Starovoitov V. V., Golub Yu. I. Digital image, from receipt to processing. Minsk, United Institute of Informat- ics problems, National Academy of Sciences of Belarus, 2014, 202 p. (in Russian). 10. Dvorkovich V.P., Dvorkovich A.V. Digital vid- eo information systems (Theory and Practice), Moscow, Technosphere Publ., 2012, 1008 p. (in Russian). 278