Image Processing

• Type of physical science: Mathematical methods
• Field of study: Signal processing

Image processing (computer vision) provides digital signal processing of two- or higher-dimensional signals. Image processing can range from simple filtering and enhancement of an individual image to more complicated image-data comparisons and pattern recognition. Image analysis is the converse of computer graphics, since its input is a digital image of a scene as it would be seen from a given viewing location, and its output is a true scene description, whereas computer graphics takes a model of a scene to generate a visual image.

Overview

Image processing or analysis is a multidisciplinary field with methods and jargon often overlapping with signal processing. An image in this context is a two-dimensional representation of a measured or observed object or quantity, obtained by means such as photography, X-ray radiography and tomography, ultrasound, or radar. Although different for different applications, image processing entails a sequence of generic computer-processing operations: data digitization and compression, image enhancement and reconstruction, and image segmentation, matching, and pattern recognition.

In most cases, image processing converts measured density levels on a sensor or film to numbers enterable into computer memory. The number of picture elements, or pixels, needed to represent a given image accurately depends on the overall image size, the number and kind of fine details, and structural complexity. This input digitization comprises both sampling and quantization. In sampling, the two-dimensional grid spacing must be sufficiently fine to pick up all important image details. Employing too few discrete sample points can give rise to poorly separated or smeared image features. Image compression standards now include learning-based image coding, including JPEG artificial intelligence (AI), gaining international-standard publication status in 2025.

The Nyquist sampling theorem requires a grid sampling size of d to reconstruct all image components of period 2d. In many images, point sampling (quantization) can be coarse in regions with (slow) gray-level variation; these facts can be used in an adaptive sampling/quantization algorithm that varies the number and size of data samples from image point to point. This sampling generates an integer for each pixel representing intensity or amplitude at that position; some multispectral images from satellites, for example, can have more than one local signal property defined at each point. Quantization represents the main image variable (height, brightness, and so on) at a given (x,y) point by a given z value. When sampling and quantization are completed for all pixels, the image is digitally represented by a rectangular array of numbers, which can subsequently be processed using digital signal and related image-processing techniques.

The choice of specific image-analysis techniques in any given case depends strongly on the data quality and particular application. Practical solutions to many image-processing problems frequently result from using a combination of techniques in a proper sequence. Most image-processing techniques seek to reduce complicated object shapes to as few descriptive parameters as possible. Topographic roughness as measured by radar or satellite mapping, for example, is frequently specified by storing values of feature location and shape, local root-mean-square height, autocorrelation length, and Fourier transform power spectral coefficients.

Aspects of the more general inverse problem often arise in image processing, since there are usually many possible objects that could give rise to a given image. In most images, both shape and intensity properties are considered. Early in the image-processing sequence, simple statistical measures, such as histograms of the digitized intensity distribution within an image, are invaluable for performing image intensity and contrast comparisons for optimal display.

Frequently, the discrete fast Fourier transform is employed to decompose a complex image pattern quantitatively into a sum of simple shapes of different spatial frequencies. Here, large amplitudes at high spatial frequencies indicate highly curved boundaries or abrupt edges; low-frequency components are interpreted as overall gradual trends. Filters in general suppress much of the random image noise and fluctuations, as well as clarify image details and reduce some of the computational noise of subsequent digital image-processing sequences. High (low) pass filtering removes slowly (rapidly) varying components and retains only high-frequency local (low-frequency global) variations. As in signal processing, high and low pass filtering is usually performed computationally by a local moving average statistic.

Image enhancement is vital because object illumination is usually too high or too low, and thus suboptimal in gray level, to show the full image against the background. Image enhancement is implemented in either the time or frequency (transform) domain. In the time domain, smoothing and image-sharpening operators are frequently used. One basic image enhancer is thus grayscale modification, accomplished by spreading the image gray levels over a wider range. Another related technique is to map background gray levels into a greater number of colors.

In the transform domain, low-pass and homomorphic filtering are typically used. Since image illumination (low frequency) and reflectivity (high frequency) vary slowly and rapidly, respectively, in homomorphic filtering, their independent variations can be separated by taking the natural logarithm of each point’s gray level. Then the fast Fourier transform of the log-scaled image is taken, and high-frequency enhancing filters are applied, where the result is antilogged and inverse Fourier transformed. The general goal in all image restoration is removing or undoing unwanted effects in the image that result from geometric and/or noise degradation. More advanced image restoration frequently employs inversion techniques, as well as Kalman filtering, constrained optimum filtering, and recursive (Wiener) filtering. Least square Wiener inverse filtering uses the standard convolutional model g = h x f + n, where f is the desired true image, h the measuring system or medium response, n statistically governed noise, and g the actually measured image. Kalman filtering estimates each image pixel value as a linear combination of estimates at nearby pixels, such that the weighting coefficients have minimum squared error.

Image reconstruction uses various so-called projection techniques to sum gray/intensity levels along specific groups or sections to form a more correct total image than the previous image estimates. The one-dimensional Fourier transform of a projection of an image is the cross-section of a two-dimensional image transform. The full Fourier transform can be approximated by interpolating from available cross sections and finally reconstructing the image by inverse Fourier transformation. Another well-known image-reconstruction technique is that of back-projection. Here, each image projection gives a set of linear algebraic equations in the gray-level variable, which, when solved, yields a higher-quality total image estimate.

Image segmentation relies on region and edge detection methods. Region detection is accomplished via simple and more complex “thresholding” methods. Threshold methods define a min/max local gray-scale level, or rate of change of gray level per unit distance, which, when exceeded, is taken to define a new or different image segment. Edges are detected by so-called local contrast gradients, using statistical models of what expected types of edges should look like in terms of gray level. Contrast gradients permit edge definition by computing the second spatial derivative of gray-level values.

Image recovery seeks an even more highly refined image by separating from the net recorded image the effects on the grayscale of object illumination, object reflectivity, and object orientation. In this stage, true image shape can be determined from ray-theoretic shadows and shading, image texture, and model objects and shapes. Gray-level direction gradients resulting from object and surface roughness are used to infer object roughness features.

Pattern recognition, which was traditionally viewed as the final stage of image processing, is integrated throughout the pipeline via deep learning and foundation models. By “pattern” is meant some structure or form present in a given signal data set. Examples of patterns in science and engineering include the waveshape of a seismic or sonar signal in contact with a particular reflecting interface. Pattern recognition in general can be divided into decision-theoretic and structural methods. Decision-theoretic pattern recognition is mainly concerned with the classification of patterns, via algorithmically assigning an observed signal or image pattern to an already established class (or classes) to which it is most likely to belong. If the patterns in question have a more complex structure, such as contour lines or geometric surfaces, then, in addition to pattern classification, pattern analysis is needed. Here the pattern is decomposed into sub-patterns, sub-patterns into sub-subpatterns, and so on, until a stable set of elementary patterns is achieved.

Pattern recognition within images frequently employs Fourier transform analysis, matched filtering, and, more recently, (multi)fractal methods. Fourier series and fast Fourier transform (FFT) analysis permit decomposing and selectively reconstructing complex images as a function of spatial wavelength and angular orientation. Matched filtering is a technique based on least-squares matching of a mathematical function to a given image component. Fractals are mathematical models of one-, two-, or three-dimensional curves, shapes, and other physical boundaries, which assume that a given geometric pattern does not change with positive or negative magnification (scale-invariant).

Many simpler patterns can be detected by comparing the observed image with a prior template or model using the cross-correlation function, by subtractive differencing, stereo-imaging, or triangulation. In pattern recognition, an object is taken to be a structure or arrangement of parts whose relations or properties satisfy different geometrical, topological, and other constraints. Pattern and feature detection typically begins seeking spatial pattern types in the observed image with a collection of templates based on computing first/second spatial derivatives, autocorrelation, or spectral functions. Image segmentation further identifies other distinct (sub)regions and properties of pixels within and connecting the above basic image units.

These labeled patterns and segments can then be regrouped and partitioned within and connect to the above basic image sub-patterns.

In so-called syntactic pattern recognition, patterns are identified by detecting various elements or basic image components satisfying a number of constraints, based on a priori knowledge of what a given class of objects is expected to look like. Since the 2010s and especially in the 2020s, image analysis has been transformed by deep learning, including convolutional neural networks, vision transformers, and multimodal foundation models trained on very large datasets.

Applications

The practical possibilities and utility of many image-processing operations depend on the ease or difficulty with which digital computers can perform binary operations in general and Fourier transform-type computations in particular.

Examples of applied image processing and analysis include medical imaging, such as analysis of X-ray photography, scintigrams, ultrasonography, MRI (magnetic resonance imaging), and CT (computed tomography) scans; related (forensic) applications include fingerprint, voice, and profile analysis. Since the 2020s, medical image analysis has been shaped by the regulation of artificial intelligence–enabled medical devices, especially software used for detection, classification, and decision support.

In the realm of classical and quantum physics, uses of digital image processing include electron microscopy and various types of optical and acoustical interferometry. The basic products of meteorology, contoured maps of surface temperature, humidity, and barometric pressure, are also heavily dependent upon image processing and display. Many geophysical image applications—such as gravity and magnetic contour maps, seismic and acoustic wavefields, and remote-sensing satellite-telemetered visual, radar, and microwave band images—exploit methods to translate two-dimensional images into a series of parallel one-dimensional signals. In astronomy, image processing allows scientists to extract information from celestial bodies that may not be accessible otherwise. Astronomers enhance the details of astronomical images to create meaning out of the raw data by making objects too faint or obscure visible. This allows astronomers to study the universe with precision and greater clarity.

Tomography, another area of digital image processing, is the reconstruction or imaging of a three-dimensional object or volume by using measurements of waves passing through the object. In X-ray and seismic tomography, for example, the estimates of tissue density and earth layer velocities are reconstructed from measurements of X-ray absorption and seismic travel time. In several high-resolution forms of scanning microscopy, images of surfaces are obtained by applying a special scanning signal to x- and y-dimension transducers; the resulting image in x, y, and z space is a combination of material geometry and atomic structure. More generally, digital imaging microscopy, widely used in almost all the above applications and representative of many imaging approaches, is notably improved by permitting routine detection of very low light levels and of a quantized image intensity.

Background noise, sensor nonlinearity, and other distortions are algorithmically removed before focusing, and both contrast stretching and thresholding are subsequently employed to isolate and emphasize particular image features.

Despite the common use of the grayscale or color to measure and define image intensity, the features that most images map are not primarily shadow, color, or topographic surface height, but rather a contoured surface of an object’s physical parameters, such as temperature, chemical reactivity, energy emission in one or more wavelength-specific bands, and more generally the presence or absence of a given physical, chemical, and biological feature.

When the imaged object is sufficiently small and/or simple, its surfaces of measured object parameters may closely follow object shape, although X-ray and other high-resolution microscopy techniques can also detail an object’s fundamental molecular and atomic structure at scales below which many physical parameter measurements have no clear meaning. In this wider definition, digital image processing can radically extend the variety, accuracy, and intelligibility of an ever-growing number of scientific and engineering physical measurements.

Context

The use of computers for processing two-dimensional and three-dimensional signals and image data had its most conspicuous origins in the late 1950s through the unmanned planetary science probes and, independently, in the Massachusetts Institute of Technology’s Geophysical Analysis Group’s efforts (later continued by the oil companies) at two- and three-dimensional imaging of the subsurface. Particularly at the Jet Propulsion Laboratory in Pasadena, California, the Surveyor series of space probes returned hundreds of images of the lunar surface, subsequently redisplayed and analyzed in detail to evaluate landing sites for later manned missions.

Although radar and radio astronomy provided the impetus for laying the theoretical groundwork for Fourier analysis and aperture synthesis in the early to mid-1950s, it was particularly following publication of the Cooley-Tukey fast Fourier transform algorithm in the mid-1960s that digital image processing was further developed and applied. An excellent example of digital image processing advanced by both hardware and software developments was the Landsat series of multispectral Earth-orbiting satellite imagery systems.

In the medical sciences, digital image processing was first applied to X-ray images in the late 1960s, expanding in the early 1970s, and shortly thereafter to medical imaging, including X-ray image analysis and automatic classification of genetic chromosomes. In later years, CT scans became an important medical imaging modality that permitted high-speed, real-time monitoring of human organ functioning.

In oil exploration, the development and implementation of “migration” subsurface imaging techniques to correct for irregularities in the earth’s topography and subsurface velocity have been responsible for a number of deconvolutional and focusing image-processing techniques. Here, image processing developed as the outcome of two-dimensional signal processing.

Many factors in the development of digital image processing include the declining costs and improved capabilities of serial and parallel computer processing and bulk storage devices, as well as increased availability of specialized hardware for image digitizing and graphic display. Many new image-processing developments have accompanied the development of hardware, such as computer array processors specifically designed to process arrays of numerical gray-scale data. The most flexible emergent image-processing systems are interactive computer technologies requiring a minimum of human attention to the underlying operations while allowing rapid human selection and display of the effects of changing processing parameters. The use of artificial intelligence, such as knowledge-based and expert systems, allows the human machine image-processing network to “learn from its past,” in turn leading to greater speed and predictability in processing. Expert systems are particularly useful image-processing enhancers when the class of objects imaged represents objects with a number of more or less similar features.

Particularly important in present and future image processing is the continuing development of specialized computer architectures. Because of the two- and three-dimensional nature of massively parallel arrays, so-called multiple subarray architectures significantly speed up parallel-configured signal processors by employing multiple subarrays for simultaneously processing different data streams. In cellular array architectures, all or some of the data flow is extracted from memory and distributed within the array of signal-processing elements. As the total number of memory bits available to each processing element increases, more sophisticated and faster operations and architectures become feasible, and with them more complex and comprehensive image-processing tasks.

Image processing represents a diversified technological interaction between signal processing, artificial intelligence, statistics and probability, and a number of mathematical specialties such as topology and morphometric analysis. Likewise, image processing has found an increasingly wide adaptation and use in almost every branch of the physical, engineering, and natural sciences, as an extension of both measurement and graphic display techniques.

Principal terms

ADAPTIVE: data processing in which processing parameters are varied as the measurement of data statistics changes

ENHANCEMENT: the improvement of one or more features of an image

INFORMATION: generally, data that have been recorded, classified, and analyzed within a given statistical framework

INTERACTIVE: computer data processing in which the user can modify the processing operations while observing the output results

ITERATIVE: data processing that continues until some condition, such as accuracy or error-tolerance, is satisfied

QUANTIZATION: the restriction of a variable to a discrete number of possible values

RECOVERY: estimating the orientation of an image surface from shades and/or texture

RECURSIVE: refers to processing for which the output depends on previous outputs, as well as the input and intrinsic system response

RESTORATION: an image-processing sequence to correct for effects of known errors and noise

SEGMENTATION: the division of an image into distinct, identifiable regions

Bibliography

“Artificial Intelligence-Enabled Medical Devices.” U.S. Food and Drug Administration, 4 Mar. 2026, www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices. Accessed 16 Apr. 2026.

Burger, Wilhelm, and Mark J. Burge. Principles of Digital Image Processing: Advanced Methods. Springer London, 2013.

Bushberg, Jerold T., et al. The Essential Physics of Medical Imaging. Wolters Kluwer Health, 1994.

“Computed Tomography (CT).” National Institute of Biomedical Imaging and Bioengineering, July 2025, www.nibib.nih.gov/science-education/science-topics/computed-tomography-ct. Accessed 16 Apr. 2026.

Gonzalez, Rafael C., and Richard E. Woods. Digital Image Processing, Global Edition. Pearson Education, 2018.

“Image Processing.” Encyclopaedia Britannica, 23 May 2025, www.britannica.com/technology/image-processing. Accessed 16 Apr. 2026.

“JPEG AI.” Joint Photographic Experts Group, jpeg.org/jpegai/. Accessed 16 Apr. 2026.

“Magnetic Resonance Imaging (MRI).” National Institute of Biomedical Imaging and Bioengineering, www.nibib.nih.gov/science-education/science-topics/magnetic-resonance-imaging-mri. Accessed 16 Apr. 2026.

“Multispectral Scanner System (MSS).” NASA, 2 Dec. 2025, science.nasa.gov/mission/landsat/mss/. Accessed 16 Apr. 2026.

Russ, John C. Computer-Assisted Microscopy: The Measurement and Analysis of Images. Springer US, 1988.

Solomon, Chris, and Toby Breckon. Fundamentals of Digital Image Processing: A Practical Approach with Examples in Matlab. Wiley, 2011.

Starck, Jean-Luc, and Fionn Murtagh. Astronomical Image and Data Analysis. Springer Berlin Heidelberg, 2013.

Trends and Advancements of Image Processing and Its Applications. Springer International Publishing, 2021.

Yang, Yixin, et al. “Application of Artificial Intelligence in Medical Imaging: Current Status and Future Directions.” iRadiology, vol. 3, no. 1, 2025, doi:10.1002/ird3.70008. Accessed 16 Apr. 2026.