源码

git clone https://webrtc.googlesource.com/src

ADM: audio device module模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

webrtc

WebRtcVoiceEngine::Init() 的时候初始化 
    adm_ = webrtc::AudioDeviceModule::Create(
        webrtc::AudioDeviceModule::kPlatformDefaultAudio, task_queue_factory_);

class AudioDeviceModuleIOS sdk/objc/native/src/audio/audio_device_module_ios.mm
    继承 class AudioDeviceModuleIOS : public AudioDeviceModule
    这里使用 std::unique_ptr<AudioDeviceIOS> audio_device_;

    还有其他的 AndroidAudioDeviceModule WindowsAudioDeviceModule 都继承public AudioDeviceModule

class AudioDeviceIOS sdk/objc/native/src/audio/audio_device_ios.mm
    这里使用  std::unique_ptr<VoiceProcessingAudioUnit> audio_unit_;

class VoiceProcessingAudioUnit sdk/objc/native/src/audio/voice_processing_audio_unit.mm
    主要在这里 AudioUnitSetProperty

可以参考源码中的文档
modules/audio_device/g3doc/audio_device_module.md
ADM: audio device module

另外一些分析ADM文章：https://blog.piasy.com/2018/09/14/WebRTC-ADM/index.html

一些refercence

• Official web site: http://www.webrtc.org
• Master source code repo: https://webrtc.googlesource.com/src
• Samples and reference apps: https://github.com/webrtc
• Mailing list: http://groups.google.com/group/discuss-webrtc
• Continuous build: https://ci.chromium.org/p/webrtc/g/ci/console
• Coding style guide
• Code of conduct
• Reporting bugs

Welcome to Hexo! This is your very first post. Check documentation for more info. If you get any problems when using Hexo, you can find the answer in troubleshooting or you can ask me on GitHub.

Quick Start

Create a new post

1

$ hexo new "My New Post"

More info: Writing

Run server

1

$ hexo server

More info: Server

Generate static files

1

$ hexo generate

More info: Generating

Deploy to remote sites

1

$ hexo deploy

More info: Deployment

uint8_t 无符号1个字节的整型

uint16_t 无符号2个字节的整型

uint32_t 无符号4个字节的整型

uint64_t 无符号8个字节的整型

注：一个字节有8位。

unsigned char是无符号字节型，char类型变量的大小通常为1个字节（1字节=8个位），且属于整型。整型的每一种都有无符号（unsigned）和有符号（signed）两种类型（float和double总是带符号的），在默认情况下声明的整型变量都是有符号的类型（char有点特别），如果需声明无符号类型的话就需要在类型前加上unsigned。

create patch

git format-patch HEAD^

check patch

git apply --check ~/Code/ilinkvoip_client/0001-tmp-compile-done.patch

apply patch

git apply ~/Code/ilinkvoip_client/0001-tmp-compile-done.patch

git am xxx.patch

1
2
3
4

// 最近一次的patch
git format-patch -1
// 最近两次的patch
git format-patch -2

ssh 添加服务器信任

1
2

ssh-copy-id -i ~/.ssh/id_rsa.pub -p 36000 root@xxxxx

官方文档

The Audio Unit
IO audio unit

iOS Play audio data through Audio Unit

好奇怪的文章，图片都是中文的
地下链接了一些文章

简书

简书Audio Unit详解

audio unit是一个插件
audio unit可以有多个
audio unit iOS根据功能提供了七种audio unit，分为四类: effect, mixing, i/o, format conversion
I/O输出unit，直接连接输出硬件。
VPIO和RemoteIO是属于IO audio unit提供的一种功能

1. 七种四类audio unit

3.1 Effect Unit
iOS4提供了一个effect unit，即iPod Equalizer，与内置iPod应用程序使用的均衡器相同。要查看该audio unit的iPod应用程序用户界面，请转至Setting> iPod> EQ。 使用此audio unit时，您必须提供自己的UI。 该audio unit提供一组预设均衡曲线，如低音增强器，流行音乐和口语

3.2 Mixer Units
iOS提供两个mixer units。3D Mixer unit是OpenAL构建的基础。在大多数情况下，如果需要3D Mixer unit的功能，最佳选择是使用OpenAL，它提供更适合游戏应用的更高级API。

Multichannel Mixer unit可以为任意数量的单声道或立体声流提供混音，并带有立体声输出。你可以打开或关闭每个输入，设置其输入增益，并设置其立体声平移位置。

3.3 I/O Units
iOS提供三个I/O unit。Remote I/O是最常用的。它连接输入和输出音频硬件，并为你提供对各个传入和传出音频样品值的低延迟访问。它提供硬件音频格式和应用程序音频格式之间的格式转换，通过Format Converter unit进行格式转换。

Voice-Processing I/O unit通过添加声学回声消除来扩展远程I/O单元，以用于VoIP或语音聊天应用程序。它还提供自动增益校正，语音处理质量和静音。

Generic Output unit不连接到音频硬件，而是提供将处理链的输出发送到应用程序。通常使用Generic Output unit 进行离线音频处理

3.4 Format Converter Unit
iOS4提供，通常通过I/O单元间接使用

2. AudioComponentDescription

要在运行时查找audio unit(找到7种的某一个audio unit)，需要为AudioComponentDescription数据结构指定type，subtype，manufacturer。无论是audio unit还是audio process graph API。
其实就是为了找到哪一种audio unit(比如vpio和remoteio)，先定义AudioComponentDescription 然后通过AudioComponentFindNext找到对应component，最后初始化audiounit

1
2
3
4
5
6
7
8
9
10

AudioComponentDescription audioDesc = {0};
audioDesc.componentType = kAudioUnitType_Output;
audioDesc.componentSubType = kAudioUnitSubType_RemoteIO;
audioDesc.componentManufacturer = kAudioUnitManufacturer_Apple;


// find component
AudioComponent compoent = AudioComponentFindNext(NULL, &audioDesc);
// create new instance
AudioComponentInstanceNew(compoent, &audioUnit);

3.使用scope和elements来指定audio units的各个部分

Audio unit的各个部分组织成scopes and element，如下图所示。当调用函数去配置和控制audio unit时，你可以指定scope和element以标识函数的特点目标。

官方文档
scope是audio unit内的编程上下文。虽然global scope可能暗示，但这些上下文从不嵌套。一般是使用一个常量值来自AudioUnitScope枚举。

element是嵌套在audio unit scope中的编程上下文。当element是输入或输出scope的一部分时，它类似于物理音频设备中的信号总线，因此有时成为总线。

global scope适用于整个audio unit，不与任何特定音频流相关联。它只有一个元素，即0。某些属性，如每个切片最大帧数（kAudioUnitProperty_MaximumFramesPerSlice），仅适用于global scope

输入和输出scopes直接参与通过audio unit移动一个或多个音频流。正如你所期待的那样，音频输入scope进入从输出scope离开。属性或参数可以作为整体应用于输入scope或输出scope，某些属性只能应用于scope特定的element

One input element: element 1. One output element: element 0.

By default, the input element is disabled and the output element is enabled. If you need to change this, refer to the description of the kAudioOutputUnitProperty_EnableIO property.

The outward-facing sides of the Remote I/O unit acquire their formats from the audio hardware as follows:

The input element (element 1) input scope gets its stream format from the currently-active audio input hardware.

The output element (element 0) output scope gets its stream format from the currently-active output audio hardware.

Set your application format on the output scope of the input element. The input element performs format conversion between its input and output scopes as needed. Use the hardware sample rate for your application stream format.

If the input scope of the output element is fed by an audio unit connection, it acquires its stream format from that connection. If, however, it is fed by a render callback function, set your application format on it.

audio enters at the input scope and leaves at the output scope.

The input element is disabled by default

总结一下：

element即bus，input：1（采集）， output：0（播放）。

每个element有input scope也有output scope。

input scope 和 output scope可以有多个element

element 1的input scope直接连接input 硬件，input scope的stream format是由硬件设置。

element 0的output scope直接连接output 硬件，output scope的stream format是由硬件设置。

即，app需要设置的是element1的output scope 和 element0的input scope的stream format流格式（AudioStreamBasicDescription，就是ASBD）。

另外enableIO的时候，是enable hardware，即element1的input scope 和 element0的output scope

还有一些配置是在global scope上设置的

每个AudioUnit都有Input, Output 和 Global 三个域。
input输入域是音频流进入unit的入口，output输出域是音频流离开unit的出口，global全局域则代表整个unit。
输入域和输出域都有若干个bus/element，比如说mixer unit有多个输入bus，只有一个输出bus；而splitter unit则有一个输入bus，有多个输出的bus。

1
2
3
4
5
6

对于I/O Unit来说，Scope和bus的语义如下：
Scope          bus                 语义                                          权限
Input          1           Input from hardware to I/O unit                     read-only
Output         1           Output from I/O unit to program or other units      R / W
Input          0           Input to I/O unit from program or other units       R / W
Output         0           Output from I/O unit to hardware                    read-only

什么是render callback

Each request for a set of data is known as a render call or, informally, as a pull. The figure represents render calls as gray “control flow” arrows. The data requested by a render call is more properly known as a set of audio sample frames (see frame).

In turn, a set of audio sample frames provided in response to a render call is known as a slice. (See slice.) The code that provides the slice is known as a render callback function, described in Render Callback Functions Feed Audio to Audio Units.

To provide audio from disk or memory to an audio unit input bus, convey it using a render callback function that conforms to the AURenderCallback prototype. The audio unit input invokes your callback when it needs another slice of sample frames,

The process of writing a render callback function is perhaps the most creative aspect of designing and building an audio unit application. It’s your opportunity to generate or alter sound in any way you can imagine and code

At the same time, render callbacks have a strict performance requirement that you must adhere to. A render callback lives on a real-time priority thread on which subsequent render calls arrive asynchronously. The work you do in the body of a render callback takes place in this time-constrained environment. If your callback is still producing sample frames in response to the previous render call when the next render call arrives, you get a gap in the sound. For this reason you must not take locks, allocate memory, access the file system or a network connection, or otherwise perform time-consuming tasks in the body of a render callback function.

This is the prototype for a function callback Proc that is used both with the
AudioUnit render notification API and the render input callback. See
kAudioUnitProperty_SetRenderCallback property or AudioUnitAddRenderNotify.
This callback is part of the process of a call to AudioUnitRender. As a
notification it is called either before or after the audio unit’s render
operations. As a render input callback, it is called to provide input data for
the particular input bus the callback is attached too.

总结一下

• request data的过程叫做render call 或者 pull。如下图所示的灰色箭头。

• request的data叫做slice(slice就是一些audio frame)

• render callback function就是提供slice的函数，从内存或者磁盘送到audio unit的input（这里应该就是input bus）就需要用AURenderCallback这个函数， 每次audio input 需要slice的时候就会call render callback function， 原型是AURenderCallback官方文档，【所以代码是在GetCapturedDataToSend里面调用AudioUnitRender，这里的 AURenderCallback和AudioUnitRender不同！！】

• render callback function 不要做太耗时的操作

kAudioUnitProperty_SetRenderCallback是audio unit需要数据，向Host(APP)请求数据；

kAudioOutputUnitProperty_SetInputCallback是audio unit通知Host(APP)数据已经就绪，可以通过AudioUnitRender拉取数据；

AudioUnitRender的解释是：Initiates a rendering cycle for an audio unit.
下图阐释了AudioUnit是如何通过AudioUnitRender去Pull音频流数据

其中，kAudioUnitProperty_SetRenderCallback 是 General Audio Unit Properties中的一个,

kAudioOutputUnitProperty_SetInputCallback 是 Properties for Apple I/O audio units (sometimes called output units)的一个

kAudioOutputUnitProperty_SetInputCallback 是

A read/write AURenderCallbackStruct data structure valid on the audio unit global scope. When an output unit has been enabled for input operation, this callback can be used to provide a single callback to the host application from the input I/O proc, in order to notify the host that input is available and may be obtained by calling the AudioUnitRender function. Note that the inputProc will always receive a NULL AudioBufferList in ioData. You must call AudioUnitRender in order to obtain the audio.

AudioUnitRender函数官网文档

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

/*!
	@function		AudioUnitRender
	@abstract		the render operation where ioData will contain the results of the audio unit's
					render operations
	@discussion		an audio unit will render the amount of audio data described by 
					inNumberOfFrames and the results of that render will be contained within 
					ioData. The caller should provide audio time stamps where at least the sample 
					time is valid and it is incrementing sequentially from its previous call 
					(so, the next time stamp will be the current time stamp + inNumberFrames) 
					If the sample time is not incrementing sequentially, the audio unit will infer
					that there is some discontinuity with the timeline it is rendering for
	
					The caller must provide a valid ioData AudioBufferList that matches the 
					expected topology for the current audio format for the given bus. The buffer 
					list can be of two variants:
					(1) If the mData pointers are non-null then the audio unit will render its 
					output into those buffers. These buffers should be aligned to 16 byte 
					boundaries (which is normally what malloc will return).
					(2) If the mData pointers are null, then the audio unit can provide pointers 
					to its own buffers. In this case the audio unit is required to keep those
					buffers valid for the duration of the calling thread's I/O cycle
					 
	@param			inUnit
					the audio unit
	@param			ioActionFlags
					any appropriate action flags for the render operation
	@param			inTimeStamp
					the time stamp that applies to this particular render operation. when 
					rendering for multiple output buses the time stamp will generally be the same 
					for each output bus, so the audio unit is able to determine without doubt that 
					this the same render operation
	@param			inOutputBusNumber
					the output bus to render for
	@param			inNumberFrames
					the number of sample frames to render
	@param			ioData
					the audio buffer list that the audio unit is to render into.
	
	@result			noErr, or an audio unit render error
*/

1
2
3
4
5
6
7
8

[字段含义](https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/AudioUnitHostingGuide_iOS/AudioUnitHostingFundamentals/AudioUnitHostingFundamentals.html#//apple_ref/doc/uid/TP40009492-CH3-SW11)
The inTimeStamp parameter represents the time at which the callback was invoked. It contains an AudioTimeStamp structure, whose mSampleTime field is a sample-frame counter. On each invocation of the callback, the value of the mSampleTime field increments by the number in the inNumberFrames parameter. If your app is a sequencer or a drum machine, for example, you can use the mSampleTime value for scheduling sounds.

The inBusNumber parameter indicates the audio unit bus that invoked the callback, allowing you to branch within the callback depending on this value. In addition, when attaching a callback to an audio unit, you can specify a different context (inRefCon) for each bus.

The inNumberFrames parameter indicates the number of audio sample frames that the callback is being asked to provide on the current invocation. You provide those frames to the buffers in the ioData parameter.

The ioData parameter points to the audio data buffers that the callback must fill when it is invoked. The audio you place into these buffers must conform to the audio stream format of the bus that invoked the callback.

采集回调里面会调用 AudioUnitRender(), 但是播放回调里面不会有，这个函数是干啥的
stackoverflow问题：https://stackoverflow.com/questions/20443187/whats-means-audiounitrender/20450434

Core Audio works on a “pull” model, where the output unit starts the process off by asking for audio samples from the unit connected to its input bus. Likewise, the unit connected to the output unit asks for samples connected to its input bus. Each of those “asks” is rendering cycle.

AudioUnitRender() typically passes in a buffer of samples that your audio unit can optionally process in some way. That buffer is the last argument in the function, ioData. inNumberFrames are the number of frames being passed in by ioData. 1 is the output element or ‘bus’ to render for (this could change depending on your configuration). rioUnit is the audio unit in question that is doing the processing.

Apple’s Audio Unit Hosting Guide contains a section on rendering which I’ve found helpful.

什么时候需要设置ASBD

A key feature of an audio unit connection, as shown in Figure 1-8, is that the connection propagates the audio data stream format from the output of its source audio unit to the input of its destination audio unit. This is a critical point so it bears emphasizing: Stream format propagation takes place by way of an audio unit connection and in one direction only—from the output of a source audio unit to an input of a destination audio unit.

通过audio unit connection 的 format 传播，可以节省设置ASBD

1. AudioStreamBasicDescription Structure

   1 2 3 4 5 6 7 8 9 10 11 12 13 14

   struct AudioStreamBasicDescription { Float64 mSampleRate; UInt32 mFormatID; UInt32 mFormatFlags; UInt32 mBytesPerPacket; UInt32 mFramesPerPacket; UInt32 mBytesPerFrame; UInt32 mChannelsPerFrame; UInt32 mBitsPerChannel; UInt32 mReserved; }; typedef struct AudioStreamBasicDescription AudioStreamBasicDescription; Because the name AudioStreamBasicDescription is long, it’s often abbreviated in conversation and documentation as ASBD. To define values for the fields of an ASBD, write code similar to that shown in Listing 1-7.

2. 交叉格式

An AudioBuffer structure holds a single buffer of audio data in its mData field. The buffer can represent two different sorts of audio:

• A single, monophonic, noninterleaved channel of audio

• Interleaved audio with any number of channels—as designated by the mNumberChannels field

Noninterleaved formats are used primarily by audio units and audio converters.

一些设计模式

官方文档

其实这里有点弄不清楚 什么是 audio unit connection 和 render callback

• I/O Pass Through

I/O pass-through模式将传入的音频直接发送到硬件输出，中间没有处理音频数据。

I/O.png

• I/O Without a Render Callback Function

在Remote I/O unit的elements之间添加一个或多个其它audio unit，可以构建出更有用的应用，因为没有回调函数，则无法直接操作音频，限制了实用性。

• I/O_No_Callback.png
  I/O with a Render Callback Function
  在Remote I/O unit之间添加一个回调函数，可以在传入音频到达输出硬件之前对其进行操作。比如使用渲染回调函数调整输出音量，添加颤音，铃声调制，回声或其他效果。通过使用Accelerate框架中提供的傅立叶和卷积函数，你的可能性是无穷尽的

render_callback.png

• Output-Only with a Render Callback Function
  适用于游戏和音乐合成app，将渲染回调函数直接连接到Remote I/O的Output element的input scope

only_output.png

• complex pattern
  complex.png

stackoverflow

问题：https://stackoverflow.com/questions/31973110/getting-mic-input-and-speaker-output-using-core-audio/32414034
aurioTouch

再参考一个demo

https://github.com/eaglesue/EARecorder_AU

• xposed

算法

• Git: TheAlgorithms/Python Tables

  • can check Project Euler

• labuladong 的算法小抄

• 小浩算法是一套针对小白的完整的算法训练流程

前端

• microsoft/Web-Dev-For-Beginners

• 工作日每天一道大厂前端面试题 Daily-Interview-Question

• 前端面试每日3+1

python

• Complete-Python-3-Bootcamp

github collections

learn-to-code

road map

developer-roadmap

virtualxposed

无须root使用xposed插件

ps 微信64位不支持

xposed插件demo

github xposed demo

我是我平淡无奇生活最后的目击者