The Intel neuromorphic DNS challenge

One of the first prominent neuromorphic challenge problems was image classification on the N-MNIST or N-Caltech101 datasets [8]. N-MNIST and N-Caltech101 are neuromrophic versions of the classic MNIST [2] and Caltech101 [9] datasets: the neuromorphic datasets were captured using an event-based camera moving in a precise saccadic motion while pointed at a computer monitor displaying an MNIST or Caltech101 static image. While N-MNIST and N-Caltech101 were instrumental in advancing neuromorphic vision research and provided common datasets to compare various neuromorphic algorithms, the inherent source of information is a static image and lacks spatiotemporal information content [10], especially once the saccadic motion is compensated. Thus these datasets are generally not ideal for showcasing the full potential of neuromorphic computational models which aim to exploit neuronal dynamics inspired by biological neurons for efficient temporal signal processing (section 3).

Another popular neuromorphic vision challenge problem is gesture recognition on the DVS Gesture dataset [11]. The DVS Gesture dataset is naturally matched to neuromorphic computing—the sparse, event-based, and spatiotemporal nature of dynamic vision sensor data naturally lends itself to neuromorphic processors that also possess these attributes. Evaluated as a neuromorphic challenge problem, however, DVS Gesture uses specialized event-based sensor data which limits widespread applicability, and the dataset is small (1342 instances). Neuromorphic solutions on DVS Gesture achieve a latency of 104 ms [11] on TrueNorth and 12.5 ms on Loihi [1] processing at 1 ms per step. Further study shows that the accuracy on the task improves with a coarser timestep of up to 25 ms [12]. This indicates that the fine-grained temporal information in the DVS Gesture dataset may not be vital in this task; intuitively, common gestures are likely slow enough to be sufficiently captured on a slower timescale.

A popular neuromorphic audio challenge problem is keyword spotting on the Spiking Heidelberg Datasets [7]. The Spiking Heidelberg Datasets target the widely-applicable task of keyword spotting, and importantly, audio is pre-processed with a neuroscience-inspired cochlea model. This provides a consistent neuromorphic encoding to spikes upon which researchers can build their keyword spotting algorithms, thus facilitating simple and fair task performance comparisons across different spiking neural network (SNN) algorithms. However, the cochlear encoding of the Spiking Heidelberg Datasets presents some critical shortcomings when viewed from the greater context of more general and more difficult audio processing tasks. Firstly, the information preservation of the cochlear encoding is unquantified, thus this encoding could artificially bottleneck the performance of keyword spotting, and perhaps severely bottleneck performance for more sophisticated audio processing tasks. Secondly, the power cost of computing the cochlear encoding is also unquantified, yet power is an important factor in real-world low-power audio processing systems. Indeed, how to encode an audio signal efficiently and faithfully for processing in a neuromorphic system is an open research question which plays an important role in our definition of the Intel Neuromorphic Deep Noise Suppression Challenge (Intel N-DNS Challenge).

Other neuromorphic benchmarks have been proposed that target applications that are also well-matched to the spatiotemporal event-based neuromorphic computing style, such as Braille letter reading [13] and gesture recognition using electromyograph and dynamic vision sensor fusion [14]. However, these benchmarks involve niche sensors and applications, limiting their real-world impact and interest compared to more mainstream AI problems dealing with images, video, text, or audio.

In this work, we identify audio denoising as an excellent neuromorphic challenge task. As detailed in subsequent sections, audio denoising has ubiquitous real-world applicability and plays to the strengths of neuromorphic computing. We have developed the Intel N-DNS Challenge to make the task easily accessible, free to all, unsaturated, and designed specifically to make it easy to compare solutions over a comprehensive set of metrics.

The Intel N-DNS Challenge is inspired by the Microsoft DNS Challenge, an audio denoising challenge that has been running since 2020 [15–18]. At a basic level, the Microsoft DNS Challenge has focused on improving speech denoising solutions as measured by human perceptual audio quality metrics and the Challenge included a track with the constraint that solutions must run in real-time on an Intel i5 or equivalent processor; essentially, the goal was to obtain the highest audio quality possible under the compute architecture constraint. In contrast, in the Intel N-DNS Challenge, we are changing this architecture constraint and taking a more holistic approach to evaluating solutions by defining metrics for power and latency in addition to audio quality metrics.

The spirit of the Intel N-DNS Challenge is to achieve production-level (near-SOTA) denoising performance in a system with at least an order of magnitude reduction in power, while also reducing latency, compared to real-time denoising solutions on conventional architectures. Our belief is that the neuromorphic computing features of Intel's Loihi 2 chip—representative of future commercial neuromorphic devices—will enable the realization of such gains. Thus in the Intel N-DNS Challenge we define one track focusing on evaluating solutions on existing neuromorphic hardware (Loihi 2) and another track focusing on neuromorphic algorithm development, which may motivate new features in future neuromorphic hardware.

The Intel N-DNS Challenge has a 1 year timeline, but we invite the community to continue using the Intel N-DNS Challenge as a benchmark after the challenge ends. More broadly, we view the N-DNS challenge as a single iteration in a continuing effort to develop challenge problems that help to advance neuromorphic computing to commercial maturity.

We define the audio denoising task in section 2, discuss neuromorphic computing as it pertains to this work in section 3, overview the Intel Neuromorphic DNS Challenge in section 4, describe the data in section 5, specify evaluation criteria in section 6, describe our baseline solution in section 7, address additional clarifications in section 8, and summarize our contributions in section 9. We make our code publicly available for obtaining the challenge data, evaluation pipeline, and the example baseline solution in the Intel N-DNS Challenge Github Repository (https://github.com/IntelLabs/IntelNeuromorphicDNSChallenge) with a permissive MIT license.

Recently neural networks (NN) based algorithms have been extensively applied to audio denoising problems. Initial solutions focused exclusively on denoising quality and used large models to solve the problem with great breakthroughs in accuracy. However, as the models become more and more accurate, the focus has shifted to real-time denoising performance. In fact, the most recent Microsoft DNS Challenges have dropped the non-real-time track [15, 17, 18].

Non-real-time solutions focus purely on the quality of denoising and are typically non-causal. Non-real-time solutions from the speech enhancement and source separation literature include attention architectures [28], temporal convolutional networks (TCN) [29], the Convolutional Time-domain audio separation Network (Conv TasNet) [24], convolutional phase and amplitude processing (PHASEN) [30], and audio source separation with nested depthwise convolutional downsampling (SuDoRM-RF) [31]. For the denoising task in speech enhancement, the desired enhancement is the removal of noise, and in source separation, the desired separation is between speech and noise. Real-time solutions focus on making the network lightweight and causal while maintaining denoising performance. Some examples include causal forms of TCN, Conv TasNet [29], and recurrent topologies with stacked LSTM or GRU [32].

The most common encoding-decoding method of choice is STFT-ISTFT [28, 30, 32] or its similar spectrogram transformation like DCT [33], while methods like SuDoRM-RF [31] directly process the raw audio samples. There are different approaches for processing the complex STFT input in the literature. Some methods only make use of the magnitude information [24, 29, 32], some process the magnitude and phase separately and combine them [34], while some process the complex spectrum directly using complex convolutional filters [28].

The majority of the solutions use backpropagation-based supervised training. However, a wide variety of losses have been used in different works. The most common ones are the mean-square error of the resulting spectrum or maximization of the signal-to-noise ratio. A survey of various loss metrics used in audio denoising with their benefits is described in [35]. Some solutions even prioritize speech over suppression with an additional loss penalty term [34]. In addition, unsupervised or semi-supervised training methods have also been investigated to achieve a general solution even on out-of-distribution datasets. A particularly interesting method is the teacher-student training method proposed in RemixIT [36] where a teacher network trained on out-of-distribution data is used to bootstrap the noisy signals to multiply the variety of in-distribution data samples.

It is evident that noise suppression with deep neural networks is an active area of research with new methods being introduced regularly. Recent efforts have not only focused on the quality of denoising but also on the size of models and satisfying real-time requirements. There is a vast body of research from which to borrow for neuromorphic audio denoising.

Neuromorphic computing aims to apply fundamental principles of the brain's information processing mechanisms to engineered computing devices. The brain consumes a mere 20 watts of power yet can execute remarkable feats of perception, planning, control, and learning while operating in real time processing sequential data streams. In contrast, our conventional computer systems today struggle to emulate even a narrow subset of such feats with much larger power budgets, even though they have the advantage of precisely engineered ultra-fast nanoscale transistors as a computational substrate [38]. Indeed, biological inspiration is compelling. However, when computer architects go about designing neuromorphic systems, they face a fundamental question: What biology-inspired computational strategies unlock neuromorphic performance advantages versus conventional architectures?

Neuromorphic researchers have identified several promising strategies, such as analog computation, sparse connectivity, spike-based communication, in-memory computation, local synaptic learning rules, recurrent feedback, and stateful, dynamic neuron models [39]. Subsets of these computational strategies are being implemented in hardware, e.g. novel analog devices [40], analog computation in conventional circuits [41–44], digital processing with spike-based communication [45–49], and many others.

In the Intel Neuromorphic Computing Lab, we focus on designing all-digital neuromorphic processors that can be manufactured in state-of-the-art semiconductor process technology. The SOTA process enables direct comparisons to SOTA conventional architectures, and the all-digital character allows a broad range of architectural features to be rapidly prototyped with fully deterministic and repeatable execution. While the all-digital character sacrifices some efficiency benefits of analog computation, we believe it is most important to first rapidly explore the architecture-algorithm co-design space before undertaking the more difficult, slower, and currently less area-efficient path of analog circuit design and novel device engineering. We believe the subset of neuromorphic computational principles supported by our latest chip, Loihi 2, are sufficient to show significant gains in power and latency compared to conventional computer architectures, and that this will motivate further optimizations via more nascent neuromorphic computing principles.

Loihi 2 is a state-of-the-art neuromorphic chip designed to efficiently compute temporal dynamics in sparse networks using sparse, event-based communication [50]. Like its predecessor [46], Loihi 2 consists of neuron cores that compute the temporal dynamics of stateful neural models and a communication mesh optimized for spike-based communication. Loihi 2 implements a number of generalizations and optimizations motivated by the learnings and pain points of its predecessor. These include microcode-programmed neuron models, which enable a much wider variety of neurons as seen in the brain [51] as well as in novel neuromorphic algorithms [52] and promising computational benefits in heterogeneous networks [53]. Loihi 2 also features graded spikes, i.e. spikes that carry an integer value, rather than binary spikes. While not biologically motivated, graded spikes are only marginally more costly to support than binary spikes in digital neuromorphic hardware and offer straightforward gains in algorithmic precision and processing speed. Loihi 2 also enhances Loihi 1's learning support so arbitrary local modulating factors ('third factors') may be computed by postsynaptic neuron microcode. We believe Loihi 2's rich feature set is sufficient to unveil significant performance gains in tasks well-suited to temporal dynamics processing, hence the spirit of using Loihi 2 as a model for neuromorphic processing in the Intel N-DNS Challenge.

The computational model implemented by neuromorphic processors such as Loihi 2 is that of a discretized dynamical system. Unlike conventional artificial neurons from machine learning, the state variables of a dynamical system evolve and process inputs in time—i.e. time is a fundamental ingredient of the computation. Thus we expect neuromorphic processors to naturally excel in temporal processing tasks, such as audio processing. Indeed, precisely-timed spiking codes are well-known to underlie audio processing in the brain [54–56], and cochleas perform sophisticated transformations to encode incoming audio for effective processing [55, 56]. These insights from neuroscience provide clear hope for the feasibility and success of neuromorphic audio processing, and recent progress on tasks such as keyword spotting provide some evidence thereof [7, 52, 57, 58].

One can immediately ascertain three critical research questions when designing a neuromorphic audio processing system: (1) How to efficiently represent an audio waveform with high fidelity in the neuromorphic domain? (2) How to efficiently perform the desired audio processing (denoising) on this neuromorphic representation? and (3) How to efficiently invert the neuromorphic representation to yield an output (waveform)?

A natural place to start answering these questions is to start with the first: how to efficiently represent a waveform in the neuromorphic domain. There exist a variety of possibilities for representing data neuromorphicly—e.g. binary spikes, graded spikes, population codes, sparse distributed codes, and phase codes—and a variety of encoding algorithms—e.g. biology-inspired cochleogram models [7, 52], Short-Time Fourier Transforms (STFTs) [59], and Mel-frequency cepstral coefficients [60]. Taking inspiration from biology, in developing our baseline solution for the Intel N-DNS Challenge, we initiated our study of neuromorphic audio encodings on cochleogram models, which can provide sparse representations in binary spikes, high sensitivity, frequency selectivity, large dynamic range, pitch-shifting, and self-peak normalization [52, 56, 61, 62]. However, we quickly realized that cochleogram models such as [7, 63] are generally computationally expensive to invert with high fidelity, which is prohibitive for a low-power denoising system. As an alternative, we developed our initial baseline solution for the Intel N-DNS Challenge using a more conventional audio encoding, the STFT [59], which is easy to invert and has perfect fidelity (aside from quantization and numerical error); furthermore, the STFT encoding can take advantage of graded spikes which are supported on Loihi 2.

While we select an STFT encoding for our baseline, we emphasize that new solutions to the Intel N-DNS Challenge have a wide range of encoding strategies to explore, e.g. designing invertible bio-inspired cochleogram models, utilizing sparse STFTs [52], or even encoding schemes that depend on feedback from other portions of the neuromorphic denoising system, much like the recurrent feedback connections from deeper areas of the brain to more low-level sensory encoding areas. Importantly, the encoding used in a neuromorphic audio processing system must be co-designed with the task for efficient operation; indeed, such synergistic design is observed in biology [64, 65].

Secondly, after audio is encoded, the actual execution of the audio processing in the neuromorphic domain is a very open research opportunity. Neuromorphic audio processing systems can employ a wide variety of strategies to perform processing in the neuromorphic domain, such as simplistic DNN conversion [66], using a network of feedforward or recurrent leaky integrate-and-fire neurons [7, 67, 68], a network of complex resonate-and-fire neurons [52], or a sigma-delta neural network (SDNN) as we describe in the following subsection for our baseline solution. Methodologies inspired by conventional deep learning, e.g. multi-timescale networks [29, 31, 36] or attention [28], if mapped efficiently to the neuromorphic domain, could be promising directions as well. And finally for completeness—to address the third question posed above—decoding the output of the neuromorphically-processed audio again depends on the processing used and must be tailored appropriately to operate in an efficient manner.

Thus we see much opportunity for innovation throughout a neuromorphic processing pipeline—encoding, processing, and decoding. Furthermore, the audio denoising task represents just one potential audio processing task that opens the door to tackling many others with methods that are transferable to other signal processing domains such as wireless, biosensors, and control.

We have developed a simple baseline neuromorphic solution to the audio denoising task, and we already begin to see evidence of significant energy efficiency gains from using neuromorphic features. The baseline solution uses a SDNN, an adaptation of a conventional feedforward ReLU neural network architecture that exploits sparse message passing with graded spikes and stateful neurons—computational strategies that can be implemented efficiently in neuromorphic architectures and that are supported by Loihi 2 in particular. The SDNN baseline solution achieves similar audio quality to a conventional baseline solution NsNet2 from the Micrsoft DNS Challenge 2022, but with an order of magnitude fewer operations and less than half its latency. We provide a more detailed overview of the baseline solution architecture and its performance in section 7.

Importantly, our SDNN baseline solution is a very basic feedforward architecture, and does not exploit several of the aforementioned neuromorphic features that perform well on Loihi 2 (table 1). As new solutions incorporate more of these features, such as recurrent and sparse connectivity, we anticipate further significant improvements in power and model size.

6.1.1. SI-SNR metric

Task performance in the N-DNS Challenge is measured as the output audio quality; we use the Scale-Invariant Source-to-Noise Ratio (SI-SNR)—SI-SNR is a common metric in the audio processing literature (e.g. [69, 70]). SI-SNR measures how clear the human speech is above the noise in the output of the N-DNS system, similar to a Source-to-Noise Ratio (SNR) [70]. But importantly, SI-SNR is also scale-invariant—i.e. changing the overall magnitude (volume) of the output does not change the SI-SNR; intuitively, we do not wish to favor solutions over others' that simply increase the output volume.

For a single input waveform, a real-valued zero-mean vector s, and the corresponding output waveform from the N-DNS system $\hat{s}$, the SI-SNR is defined as

$$\begin{equation} \textrm{SI-SNR} : = 10 \log_{10} \frac{|| s_\textrm{target} ||^2 }{||e_\textrm{noise} ||^2}, \end{equation} \tag{ 3 }$$

where $s_\textrm{target} : = \frac{\langle \hat{s}, s \rangle s}{||s||^2}$ and $e_\textrm{noise} : = \hat{s} - s_\textrm{target}$.

We choose SI-SNR as one of our metrics for its simplicity and generality, rather than more complicated audio quality metrics, such as speech-to-text word accuracy used in the Microsoft DNS Challenge [18]. The focus of the N-DNS challenge is on neuromorphic algorithm innovation; this in itself constitutes a sufficiently challenging task. Moreover, we view the audio denoising task as a representative of a general audio processing workload, and some commercial applications may not specifically prioritize human-listener perceptual quality. Finally, the SI-SNR can be conveniently used as a loss function for machine learning approaches.

The mean $\textrm{SI-SNR}$ on the test set will be used to compare solutions. A script for computing mean SI-SNR is provided in the Intel N-DNS Challenge Github Repository.

6.1.2. Minimum SI-SNR improvement

Since solutions in the Intel N-DNS Challenge are evaluated holistically, solutions may target high audio quality by using a large amount of power, or lower audio quality using a smaller amount of power, or any audio quality-power point in between. However, to ensure that the audio denoising task is being solved to some significant extent, we require solutions to achieve a minimum audio quality improvement over the noisy input audio quailty. Moreover, per our emphasis on neuromorphic computing, we require that the neuromorphic component of the N-DNS system be responsible for a significant portion of the audio quality improvement; a solution may optionally perform some denoising in the encoder and decoder, but the spirit of the Intel N-DNS Challenge is in performing neuromorphic denoising.

Therefore, we define two measures of audio quality (SI-SNR) improvement (i) relative to (1) the noisy data ($\textrm{SI-SNRi}_{\textrm{data}}$) and (2) the encode+decode processing ($\textrm{SI-SNRi}_{\textrm{encode+decode}}$), expressed by the following inequalities:

$$\begin{align} \textrm{SI-SNRi}_{\textrm{data}} &> 3~\textrm{dB} \end{align} \tag{ 4 }$$

$$\begin{align} \textrm{SI-SNRi}_{\textrm{enc+dec}} &> 3~\textrm{dB}, \end{align} \tag{ 5 }$$

where

• $\textrm{SI-SNRi}_\textrm{data} = \textrm{SI-SNR}_{\textrm{full system}} - \textrm{SI-SNR}_{\textrm{data}}$,
• $\textrm{SI-SNRi}_\textrm{enc+dec} = \textrm{SI-SNR}_{\textrm{full system}} - \textrm{SI-SNR}_{\textrm{enc+dec}}$,
• $\textrm{SI-SNR}_{\textrm{full system}}$ is the mean test-set SI-SNR from the full N-DNS system (input audio $\rightarrow$ encode $\rightarrow$ neuromorphic denoiser $\rightarrow$ decode $\rightarrow$ output audio),
• $\textrm{SI-SNR}_{\textrm{enc+dec}}$ is the mean test-set SI-SNR from running only encoder and decoder (input audio $\rightarrow$ encode $\rightarrow$ decode $\rightarrow$ output audio), and
• $\textrm{SI-SNR}_{\textrm{data}}$ is the mean test-set SI-SNR on the noisy input audio (no transformations).

Equation (4) ensures that the solution achieves a minimum audio quality improvement, and equation (5) ensures that the neuromorphic denoiser itself is responsible for a minimum audio quality improvement. These definitions allow for some amount of denoising to occur in the encoder and decoder, but critically, adding the neuromrophic denoiser must further improve audio quality. Similarly, additional pre/post-processing could be performed within the neuromorphic denoiser itself, to reduce the amount of computation in the encoder and decoder. But importantly, the computations allocated to the encoder/decoder or the neuromorphic denoiser are accounted for differently in the computational resource and chip usage metrics, as described in later sections.

6.1.3. DNSMOS metric

Computational resource cost is evaluated in terms of power, latency, number of parameters, and model size. To qualify as a real-time solution, the end-to-end latency must not be greater than 40 ms. We measure power and latency on neuromorphic hardware in Track 2, but for Track 1, we introduce proxy metrics.

6.2.1. Latency

An audio denoising system takes some amount of time to process input audio as the audio streams into the system; this results in the output human speech being delayed relative to the input human speech. This delay is the latency of the denoising system. For the denoising system to be considered real-time, the latency must be less than some human perceptual threshold, which in our case we choose to be 40 ms.

We define latency as the maximum time difference between any corresponding segment of audio in the input and output of the N-DNS system. Intuitively, the longest delay in any segment of audio is the overall delay the output must be presented at in order to not introduce playback speed fluctuations in the output audio.

Latency should be calculated by considering a real-time input propagating through an entire N-DNS system (figure 3). This includes data buffer latency, encoder-decoder latency, and network latency (N-DNS latency):

• 1)  
  Data buffer latency is the time required to collect the audio stream to process one discrete timestep, however that may be defined for a given encoding scheme. For the STFT encoder in our SDNN baseline solution, the data buffer latency is equal to the STFT window length.
• 2)  
  Encoder-decoder latency is the wall clock processing time to encode one discrete timestep-worth of the audio data, to be processed by the N-DNS network, and decode it back.
• 3)  
  Network latency (N-DNS latency) is the latency introduced by the neuromorphic denoising network. It is measured by the maximum cross-correlation between the clean target audio and the denoised audio from the network.

Zoom In Zoom Out Reset image size

In Track 1, notably, the (CPU) processing time for the neuromorphic denoiser (N-DNS) portion of the solution is not included in the latency calculation. We assume that the neuromorphic processing time will be small relative to the real-time timestep due to the high degree of parallelization in neuromorphic algorithms and hardware. In the case of the baseline SDNN, for example, the network must process a new STFT frame every 8 ms , whereas Loihi 2 circuits typically complete all spike processing and neuron evaluations for a timestep within microseconds. We provide an example Track 1 latency calculation in section 7.

For Track 2, latency is simply measured on a reference CPU + Loihi 2 system. The measurement methodology and an example will be provided in the Intel N-DNS Challenge Github Repository later in the challenge.

6.2.2. Power

For Track 1, we calculate a power proxy by estimating the effective number of synaptic operations per second:

$$\begin{equation} P_\textrm{proxy} = \textrm{Effective SynOPS} = \textrm{SynOPS} + 10 \times \textrm{NeuronOPS}, \end{equation} \tag{ 6 }$$

where SynOPS and NeuronOPS are the mean number of synaptic operations and mean number of neuron updates, respectively, per second of audio processed in the N-DNS stage. Synaptic operations and neuron operations can be considered the computational primitives of a neuromorphic system, and energy usage is roughly proportional to their number, with the approximate weighting of the energy of one neuron operation being equal to that of about ten synaptic operations in our experience with the Loihi architecture [46]. While P_proxy gives only a crude power estimate, it provides a simple and sufficiently reliable assessment of a neuromorphic power advantage without needing to run on neuromorphic hardware.

The power consumption of the encoder and decoder is not taken into account in Track 1. We make this choice for simplicity, in expectation of the neuromorphic power dominating in realistic solutions. Note that the real-time requirement implicitly bounds the amount of computation that can be performed in the encoder and decoder.

In Track 2, the encoder and decoder are implemented on a CPU and the N-DNS stage is implemented on a Loihi 2 system. The power is simply measured on a reference CPU and Loihi 2 system. Note that since both CPU and Loihi 2 power components will be measured, any attempt to implement a disproportionate amount of the denoising functionality inside the encoding/decoding CPU stages will result in a very high power result. Details for measuring power on a reference system will be provided in the Intel N-DNS Challenge Github Repository.

6.2.3. Power delay product

The power delay product (PDP) metric combines both latency and power efficiency in one number that allows comparing between different solutions that make different tradeoffs between running faster at higher power versus running slower at lower power. For Track 1, a proxy PDP measure is given by

$$\begin{equation} \mathrm{PDP}_{\textrm{proxy}} = P_{\textrm{proxy}} \times L, \end{equation} \tag{ 7 }$$

which is in units of Ops because P_proxy (equation (6)) has units of Ops/s and the latency, L, has units of seconds.

For Track 2, PDP is directly calculated from the measured power as

$$\begin{equation} \mathrm{PDP} = P_\textrm{Track 2} \times L. \end{equation} \tag{ 8 }$$

6.2.4. Chip resources

The proposed neuromorphic solution is a simple feedforward sigma-delta ReLU neural network (SDNN). The solution makes use of two neuromorphic computation ideologies: sparse message passing using sigma-delta neuron and temporal computation using axonal delays.

The delta encoding exploits the temporal similarity in the data. It sparsifies the data communicated to the next layer by sending only a change that is higher in magnitude than a certain threshold. The sigma encoding, on the other hand, reconstructs the original signal at the receiving end. A combination of sigma and delta units wrapped around a dynamics or a non-linearity (ReLU in this case) is a sigma-delta neuron [76]. Sigma-delta neurons make use of the sparse messaging paradigm in neuromorphic hardware and result in a significant reduction in synaptic computations.

The axonal delays endow the network with a short-term memory capability that allows the interaction of audio/features originating at different points in time. Learnable axonal delays have been shown to increase the expressivity and performance of networks, particularly for applications with spatio-temporal features [68, 77]. Audio denoising is one such application.

The structure of the SDNN baseline solution is illustrated in figure 4, and we describe the solution in the following.

Zoom In Zoom Out Reset image size

Encoder: The encoder is a straightforward STFT [59] of the noisy audio waveform followed by delta encoding of the STFT magnitude. The STFT uses a window length of 512 with a hop length of 128 ($^1\!/\!_4$ window length), leading to 8 ms per time-step, as the signal is at 16 kHz. These parameters are user-configurable. The delta encoding sparsifies the STFT magnitude which is then fed to the N-DNS network.

N-DNS: The neuromorphic denoiser (N-DNS) network is a three-layer feedforward sigma-delta ReLU network with axonal delays. The sigma-delta layer efficiently performs denoising in the sparse domain. The axonal delays provide the network with short term memory which can be used to incorporate previous temporal patterns during denoising. The N-DNS network predicts a multiplicative mask at some delay which is then used to mask the STFT magnitude. The STFT phase and magnitude from the encoder need to be delayed accordingly during the decoding phase.

Decoder: The decoder combines the multiplicative mask predicted by the N-DNS network with the delayed STFT phase and magnitude of the noisy audio signal and performs inverse STFT with the same window length and hop length as the encoder. The resulting output is the clean reconstruction (denoised) audio waveform.

The SDNN baseline network was trained with Lava-dl ⁴ , which includes the extended version of the SNN backpropagation training tool SLAYER [77]. Lava-dl SLAYER uses a surrogate gradient method (e.g. see [78]) to address the critical challenge in training spiking neural networks—the non-differentiability of spikes. The baseline network was trained with Loihi 2's fixed precision computation in mind and trained with appropriate quantization for synapse and neuron dynamics. We used a combination of negative SI-SNR and a mean-square error measuring the STFT magnitude reconstruction quality as the minimization loss and a RADAM optimizer for training. The detailed training procedure, as well as Lava ⁵ evaluation of the baseline network, are available in Intel N-DNS Challenge Github Repository.

We evaluated the SDNN baseline solution, Microsoft NsNet2 (the baseline network for Microsoft DNS 2022), and Intel DNS network using Track 1 metrics on the validation set. The metrics are summarized in table 2. All three networks use STFT encoding and ISTFT decoding.

Table 2. Evaluation metrics comparison.

Network	SI-SNR dB	SI-SNRi		DNSMOS b			Latency		Power proxy M-Ops s−1	PDP proxy M-Ops	Param count $\times 10^3$	Model size KB
		data dB	enc+dec dB	OVRL	SIG	BAK	enc+dec a ms	total ms
Microsoft NsNet2	11.89	4.26	4.26	2.95	3.27	3.94	0.024	20.024	136.13	2.72	2681	10 500
Intel DNS network	12.71	5.09	5.09	3.09	3.35	4.08	0.036	32.036	—	—	1901	3802
SDNN baseline	12.50	4.88	4.88	2.71	3.21	3.46	0.036	32.036	14.54	0.44	525	465
Validation set (noisy)	7.62	—	—	2.45	3.19	2.72	—	—	—	—	—	—
	Higher is better ($\uparrow$)						Lower is better ($\downarrow$)

^a Latency results measured on a system with Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz and 32 GB RAM as of February 2023 and may not reflect all publicly available security updates. Results may vary. ^b Please note that the DNSMOS scores in this table are not directly comparable to the DNSMOS scores presented in the results of the Microsoft DNS Challenge due to differing composition of validation/test sets.

Intel DNS network is an Intel proprietary network used in production. The model is causal, operates in real-time, and is built from LSTM and 2D convolution layers. Power metrics for this network are not available. The network was trained using proprietary datasets and augmentation techniques, and as such we view its audio quality results as upper-bound aspirational targets for challenge submissions.

The audio quality metrics include DNSMOS scores, SI-SNR, and improvement in SI-SNR (SI-SNRi). The encoder and decoder for all three networks perform lossless transformation using STFT and ISTFT. As a result, relative performance differences across models in SI-SNR and SI-SNRi are equal.

The latency was calculated by summing data buffer latency, encoder-decoder latency, and network latency (N-DNS latency), as described in section 6.

Power proxy and PDP proxy metrics provide some measure of the relative power and power-delay-product across the three networks suitable for Track 1 comparisons. For the SDNN baseline, these are calculated according to equations (6) and (7), respectively. For the conventional Microsoft NsNet2 network, Ops refer to multiply–accumulate operations without considering the negligible cost of per-neuron ReLU evaluation.

We see that our SDNN baseline is a promising neuromorphic solution to the audio denoising problem. In terms of audio quality, the SDNN baseline has a higher SI-SNR relative to the NsNet2 baseline solution from the Microsoft DNS Challenge 2022, and lower relative DNSMOS scores. Notably, our baseline solution training targeted an SI-SNR loss, thus better relative SI-SNR performance may be expected. Nonetheless, it is encouraging to see substantial DNSMOS improvement over the unprocessed noisy input in a system not trained specifically for perceptual quality. And importantly, the SDNN solution is an order of magnitude more efficient than the NsNet2 baseline in terms of the power proxy even though it processes data at a throughput 1.25 × higher than the NsNet2 baseline, and it uses 5 × fewer parameters. The quantization-aware training of the baseline SDNN solution further reduces the model size by 22 × compared to NsNet2.

Naturally, the NsNet2 solution is a baseline and does not represent state-of-the-art for audio denoising today. For example, the Intel production DNS model (Intel DNS network) achieves higher SI-SNR and DNSMOS than both NsNet2 and the SDNN baseline solution (table 2). Given the simplicity of our SDNN baseline solution as a starting point for neuromorphic audio denoising, we believe it will be possible to significantly improve its denoising quality while also reducing its computational resources with further algorithmic innovations in the Intel N-DNS Challenge.

Notably, the sigma-delta approach in our baseline solution is quite general. Sigma-delta sparsification can be applied to any conventional ReLU-like nonlinearity as well as to the dynamics present in typical neuromorphic neuron models such as leaky integrators and resonators. Furthermore, sigma-delta sparsification represents just one neuromorphic feature available of many to exploit by participants in the challenge. We see a wide space of uncharted waters to explore for the Intel N-DNS Challenge. Our baseline solution represents just a first step, and we find it encouraging that it already provides promising results.