Multidimensional Convolution Operation with Synthetic Frequency Dimensions in Photonics

The convolution operation is widely used in signal and image processing and represents the most computationally intensive step in convolutional neural networks. We introduce a scheme to achieve arbitrary convolution kernels in the synthetic frequency dimension with a simple setup consisting of a ring resonator incorporating a phase and an amplitude modulator. This scheme can be used to perform multidimensional convolutions. We provide an analytic approach that determines the required modulation profile for any convolution kernel. Our work points to a direction of using optical computing to remove the computational bottleneck in traditional electronic circuits and may be useful in improving machine-learning hardware in artificial-intelligence applications.

Figure 1

(a) A schematic of the modulated ring with simultaneous modulation in amplitude and phase at the frequency of free spectral range $\mathrm{\Omega }/2\pi$ and its integer multiples. The ring supports resonant modes $\{a_n\}$ and has an input-output coupling rate ${\gamma }_e$ and an intrinsic decay rate ${\gamma }_0$. The scattering matrix $\mathbf{S}$ from the modulated ring resonator converts the input ${\mathbf{c}}_{\mathrm{in}}$ to the output ${\mathbf{c}}_{\mathrm{out}}$. (b) A one-dimensional (1D) convolution operation with the kernels $s_{-1}$, $s_0$ and $s_1$ maps the input ${\mathbf{c}}_{\mathrm{in}}$ to the output ${\mathbf{c}}_{\mathrm{out}}$, which can be completed with a scattering matrix $\mathbf{S}$ of translational symmetry.

Figure 2

(a) A schematic of convolution for a two-dimensional (2D) kernel $\mathbf{F}$ of size $3\times 3$ with the zero-padded input matrix $\mathbf{X}$ of size $4\times 5$ and the output matrix $\mathbf{Y}$ of size $2\times 3$. (b) The input matrix $\mathbf{X}$ is vectorized into a 1D vector ${\mathbf{c}}_{\mathrm{in}}$ as a frequency comb. (c) The scattering matrix generated by the modulated ring resonator maintains translational symmetry among frequency sites, which is equivalent to a convolution operation. (d) The output vector ${\mathbf{c}}_{\mathrm{out}}$ after multiplication between the scattering matrix in (c) and the input vector in (b), which recovers the convolution output matrix $\mathbf{Y}$.

Figure 3

A demonstration of image convolution for a symmetric Gaussian blurring kernel $\mathbf{G}$ and a Laplacian kernel $\mathbf{L}$. (a) The original 2D image data from MNIST representing the digit 2 with a size of $22\times 24$. (b),(c) The modulation magnitude for the phase and amplitude modulation for $\mathbf{G}$ and $\mathbf{L}$, respectively. (d),(e) The time-dependent transmission factors for $\mathbf{G}$ and $\mathbf{L}$, respectively. $j\ln (T_{\mathrm{Ph}})$ and $\ln (T_{\mathrm{Am}})$, the respective logarithm of transmission factors due to the phase and amplitude modulation, are plotted. (f),(g) Generated scattering matrices with elements $S_{m,n}$, which are unitless, that correspond to $\mathbf{G}$ and $\mathbf{L}$. (h),(i) The convolution output images from $\mathbf{G}$ and $\mathbf{L}$ kernels generated from the modulated ring system, where the image is blurred and highlighted with edges, respectively. In (a), (h), and (i), the color map represents unitless pixel values.

Figure 4

A demonstration of image convolution for an asymmetric Sobel $x$ kernel ${\mathbf{I}}_x$ and the Sobel $y$ kernel ${\mathbf{I}}_y$. (a) The original 2D image data from MNIST [the same as in Fig. 3]. (b),(c) The magnitudes for the phase and amplitude modulation of ${\mathbf{I}}_x$ and ${\mathbf{I}}_y$, respectively. (d),(e) The time-dependent transmission factors for ${\mathbf{I}}_x$ and ${\mathbf{I}}_y$, respectively. $j\ln (T_{\mathrm{Ph}})$ and $\ln (T_{\mathrm{Am}})$, the respective logarithms of transmission factors due to the phase and amplitude modulation, are plotted. (f),(g) Generated scattering matrices with elements $S_{m,n}$, which are unitless, that correspond to ${\mathbf{I}}_x$ and ${\mathbf{I}}_y$. (h),(i) The convolution output images from ${\mathbf{I}}_x$ and ${\mathbf{I}}_y$ kernels generated from the modulated ring system, where the image is highlighted with the horizontal and vertical edges. In (a), (h), and (i), the color map represents unitless pixel values.

Figure 5

(a) A schematic for large-scale 2D convolution, where the input is sliced using the bandwidth-saving technique to efficiently utilize the modulator strength. (b) An example input image with a size of $64\times 64$ pixels. (c) Output images from the original input with cutoff modulation orders at $500\mathrm{\Omega }$. (d) An output image from the sliced input as generated with cutoff modulation orders at $50\mathrm{\Omega }$. (e) An output image from the original input as generated by the modulation $B_{m,o}$ with cutoff modulation orders at $50\mathrm{\Omega }$. (b)–(e) The color maps represent unitless pixel values. (f) The magnitudes of the amplitude modulation $B_m$ required to generate the kernel using the original input. (g) The magnitudes of the amplitude modulation $B_m$ required to generate the kernel using the sliced inputs.

Figure 6

(a) A schematic for multidimensional convolution, where the input has multiple channels. (b) The modulation magnitude to generate the 3D Laplacian kernel. (c) The time-dependent transmission factors, $j\ln (T_{\mathrm{Ph}})$ and $\ln (T_{\mathrm{Am}})$, due to the phase and amplitude modulation. (d) Input data consisting of an array of images at different temporal frames to represent a person waving his or her arms. (e) The output convolution image, consisting of an array of images at different temporal frames that highlight the arm motion. In (d) and (e), the color maps represent unitless pixel values.