ALSA: pcm: Set per-card upper limit of PCM buffer allocations

ALSA: pcm: Set per-card upper limit of PCM buffer allocations

Currently, the available buffer allocation size for a PCM stream
depends on the preallocated size; when a buffer has been preallocated,
the max buffer size is set to that size, so that application won't
re-allocate too much memory.  OTOH, when no preallocation is done,
each substream may allocate arbitrary size of buffers as long as
snd_pcm_hardware.buffer_bytes_max allows -- which can be quite high,
HD-audio sets 1GB there.

It means that the system may consume a high amount of pages for PCM
buffers, and they are pinned and never swapped out.  This can lead to
OOM easily.

For avoiding such a situation, this patch adds the upper limit per
card.  Each snd_pcm_lib_malloc_pages() and _free_pages() calls are
tracked and it will return an error if the total amount of buffers
goes over the defined upper limit.  The default value is set to 32MB,
which should be really large enough for usual operations.

If larger buffers are needed for any specific usage, it can be
adjusted (also dynamically) via snd_pcm.max_alloc_per_card option.
Setting zero there means no chceck is performed, and again, unlimited
amount of buffers are allowed.

Link: https://lore.kernel.org/r/20200120124423.11862-1-tiwai@suse.de
Signed-off-by: Takashi Iwai <tiwai@suse.de>