session-android/app/src/main/java/org/thoughtcrime/securesms/webrtc/PeerConnectionWrapper.kt

335 lines
11 KiB
Kotlin
Raw Normal View History

Add one on one calls over clearnet (#864) * feat: adding basic webrtc deps and test activity * more testing code * feat: add protos and bump version * feat: added basic call functionality * feat: adding UI and flipping cameras * feat: add stats and starting call bottom sheet * feat: hanging up and bottom sheet behaviors should work now * feat: add call stats report on frontend * feat: add relay toggle for answer and offer * fix: add keep screen on and more end call message on back pressed / on finish * refactor: removing and replacing dagger 1 dep with android hilt * feat: include latest proto * feat: update to utilise call ID * feat: add stun and turn * refactor: playing around with deps and transport types * feat: adding call service functionality and permissions for calls * feat: add call manager and more static intent building functions for WebRtcCallService.kt * feat: adding ringers and more audio boilerplate * feat: audio manager call service boilerplate * feat: update kotlin and add in call view model and more management functions * refactor: moving call code around to service and viewmodel interactions * feat: plugging CallManager.kt into view model and service, fixing up dependencies * feat: implementing more WebRtcCallService.kt functions and handlers for actions as well as lifecycle * feat: adding more lifecycle vm and callmanager / call service functionality * feat: adding more command handlers in WebRtcCallService.kt * feat: more commands handled, adding lock manager and bluetooth permissions * feat: adding remainder of basic functionality to services and CallManager.kt * feat: hooking up calls and fixing broken dependencies and compile errors * fix: add timestamp to incoming call * feat: some connection and service launching / ring lifecycle * feat: call establishing and displaying * fix: fixing call connect flows * feat: ringers and better state handling * feat: updating call layout * feat: add fixes to bluetooth and begin the network renegotiation * feat: add call related permissions and more network handover tests * fix: don't display call option in conversation and don't show notification if option not enabled * fix: incoming ringer fix on receiving call, call notification priorities and notification channel update * build: update build number for testing * fix: bluetooth auto-connection and re-connection fixes, removing finished todos, allowing self-send call messages for deduping answers * feat: add pre-offer information and action handling in web rtc call service * refactor: discard offer messages from non-matching pre-offers we are already expecting * build: build numbers and version name update * feat: handle discarding pending calls from linked devices * feat: add signing props to release config build * docs: fix comment on time being 300s (5m) instead of 30s * feat: adding call messages for incoming/outgoing/missed * refactor: handle in-thread call notifications better and replace deny button intent with denyCallIntent instead of hangup * feat: add a hangup via data channel message * feat: process microphone enabled events and remove debuggable from build.gradle * feat: add first call notification * refactor: set the buttons to match iOS in terms of enable disable and colours * refactor: change the call logos in control messages * refactor: more bluetooth improvements * refactor: move start ringer and init of audio manager to CallManager.kt and string fix up * build: remove debuggable for release build * refactor: replace call icons * feat: adding a call time display * refactor: change the call time to update every second * refactor: testing out the full screen intents * refactor: wrapper use corrected session description, set title to recipient displayName, indicate session calls * fix: crash on view with a parent already attached * refactor: aspect ratio fit preserved * refactor: add wantsToAnswer ability in pre-init for fullscreenintent * refactor: prevent calls from non hasSent participants * build: update gradle code * refactor: replace timeout schedule with a seconds count * fix: various bug fixes for calls * fix: remove end call from busy * refactor: use answerCall instead of manual intent building again * build: new version * feat: add silenced notifications for call notification builder. check pre-offer and connecting state for pending connection * build: update build number * fix: text color uses overridden style value * fix: remove wrap content for renderers and look more at recovering from network switches * build: update build number * refactor: remove whitespace * build: update build number * refactor: used shared number for BatchMessageReceiveJob.kt parameter across pollers * fix: glide in update crash * fix: bug fixes for self-send answer / hangup messages * build: update build number * build: update build.gradle number * refactor: compile errors and refactoring to view binding * fix: set the content to binding.root view * build: increase build number * build: update build numbers * feat: adding base for rotation and picking random subset of turn servers * feat: starting the screen rotation processing * feat: setting up rotation for the remote render view * refactor: applying rotation and mirroring based on front / rear cameras that wraps nicely, only scale reworking needed * refactor: calls video stretching but consistent * refactor: state machine and tests for the transition events * feat: new call state processing * refactor: adding reconnecting logic and visuals * feat: state machine reconnect logic wip * feat: add reconnecting and merge fixes * feat: check new session based off current state * feat: reconnection logic works correctly now * refactor: reduce TIMEOUT_SECONDS to 30 from 90 * feat: reset peer connection on DC to prevent ICE messages from old connection or stale state in reconnecting * refactor: add null case * fix: set approved on new outgoing threads, use approved more deeply and invalidate the options menu on recipient modified. Add approvedMe flag toggles for visible message receive * fix: add name update in action bar on modified, change where approvedMe is set * build: increment build number * build: update build number * fix: merge compile errors and increment build number * refactor: remove negotiation based on which party dropped connection * refactor: call reconnection improvement tested cross platform to re-establish * refactor: failed and disconnect events only handled if either the reconnect or the timeout runnables are not set * build: update version number * fix: reduce timeout * fix: fixes the incoming hangup logic for linked devices * refactor: match iOS styling for call activity closer * chore: upgrade build numbers * feat: add in call settings dialog for if calls is disabled in conversation * feat: add a first call missed control message and info popup with link to privacy settings * fix: looking at crash for specific large transaction in NotificationManager * refactor: removing the people in case transaction size reduces to fix notif crash * fix: comment out the entire send multiple to see if it fixes the issue * refactor: revert to including the full notification process in a try/catch to handle weird responses from NotificationManager * fix: add in notification settings prompt for calls and try to fall back to dirty full screen intent / start activity if we're allowed * build: upgrade build number
2022-04-19 06:25:40 +02:00
package org.thoughtcrime.securesms.webrtc
import android.content.Context
import org.session.libsignal.utilities.Log
import org.session.libsignal.utilities.SettableFuture
import org.thoughtcrime.securesms.webrtc.video.Camera
import org.thoughtcrime.securesms.webrtc.video.CameraEventListener
import org.thoughtcrime.securesms.webrtc.video.CameraState
import org.thoughtcrime.securesms.webrtc.video.RotationVideoSink
import org.webrtc.AudioSource
import org.webrtc.AudioTrack
import org.webrtc.DataChannel
import org.webrtc.EglBase
import org.webrtc.IceCandidate
import org.webrtc.MediaConstraints
import org.webrtc.MediaStream
import org.webrtc.PeerConnection
import org.webrtc.PeerConnectionFactory
import org.webrtc.SdpObserver
import org.webrtc.SessionDescription
import org.webrtc.SurfaceTextureHelper
import org.webrtc.VideoSink
import org.webrtc.VideoSource
import org.webrtc.VideoTrack
import java.security.SecureRandom
import java.util.concurrent.ExecutionException
import kotlin.random.asKotlinRandom
class PeerConnectionWrapper(private val context: Context,
private val factory: PeerConnectionFactory,
private val observer: PeerConnection.Observer,
private val localRenderer: VideoSink,
private val cameraEventListener: CameraEventListener,
private val eglBase: EglBase,
private val relay: Boolean = false): CameraEventListener {
private var peerConnection: PeerConnection? = null
private val audioTrack: AudioTrack
private val audioSource: AudioSource
private val camera: Camera
private val mediaStream: MediaStream
private val videoSource: VideoSource?
private val videoTrack: VideoTrack?
private val rotationVideoSink = RotationVideoSink()
val readyForIce
get() = peerConnection?.localDescription != null && peerConnection?.remoteDescription != null
private var isInitiator = false
private fun initPeerConnection() {
val random = SecureRandom().asKotlinRandom()
val iceServers = listOf("freyr","angus","hereford","holstein", "brahman").shuffled(random).take(2).map { sub ->
Add one on one calls over clearnet (#864) * feat: adding basic webrtc deps and test activity * more testing code * feat: add protos and bump version * feat: added basic call functionality * feat: adding UI and flipping cameras * feat: add stats and starting call bottom sheet * feat: hanging up and bottom sheet behaviors should work now * feat: add call stats report on frontend * feat: add relay toggle for answer and offer * fix: add keep screen on and more end call message on back pressed / on finish * refactor: removing and replacing dagger 1 dep with android hilt * feat: include latest proto * feat: update to utilise call ID * feat: add stun and turn * refactor: playing around with deps and transport types * feat: adding call service functionality and permissions for calls * feat: add call manager and more static intent building functions for WebRtcCallService.kt * feat: adding ringers and more audio boilerplate * feat: audio manager call service boilerplate * feat: update kotlin and add in call view model and more management functions * refactor: moving call code around to service and viewmodel interactions * feat: plugging CallManager.kt into view model and service, fixing up dependencies * feat: implementing more WebRtcCallService.kt functions and handlers for actions as well as lifecycle * feat: adding more lifecycle vm and callmanager / call service functionality * feat: adding more command handlers in WebRtcCallService.kt * feat: more commands handled, adding lock manager and bluetooth permissions * feat: adding remainder of basic functionality to services and CallManager.kt * feat: hooking up calls and fixing broken dependencies and compile errors * fix: add timestamp to incoming call * feat: some connection and service launching / ring lifecycle * feat: call establishing and displaying * fix: fixing call connect flows * feat: ringers and better state handling * feat: updating call layout * feat: add fixes to bluetooth and begin the network renegotiation * feat: add call related permissions and more network handover tests * fix: don't display call option in conversation and don't show notification if option not enabled * fix: incoming ringer fix on receiving call, call notification priorities and notification channel update * build: update build number for testing * fix: bluetooth auto-connection and re-connection fixes, removing finished todos, allowing self-send call messages for deduping answers * feat: add pre-offer information and action handling in web rtc call service * refactor: discard offer messages from non-matching pre-offers we are already expecting * build: build numbers and version name update * feat: handle discarding pending calls from linked devices * feat: add signing props to release config build * docs: fix comment on time being 300s (5m) instead of 30s * feat: adding call messages for incoming/outgoing/missed * refactor: handle in-thread call notifications better and replace deny button intent with denyCallIntent instead of hangup * feat: add a hangup via data channel message * feat: process microphone enabled events and remove debuggable from build.gradle * feat: add first call notification * refactor: set the buttons to match iOS in terms of enable disable and colours * refactor: change the call logos in control messages * refactor: more bluetooth improvements * refactor: move start ringer and init of audio manager to CallManager.kt and string fix up * build: remove debuggable for release build * refactor: replace call icons * feat: adding a call time display * refactor: change the call time to update every second * refactor: testing out the full screen intents * refactor: wrapper use corrected session description, set title to recipient displayName, indicate session calls * fix: crash on view with a parent already attached * refactor: aspect ratio fit preserved * refactor: add wantsToAnswer ability in pre-init for fullscreenintent * refactor: prevent calls from non hasSent participants * build: update gradle code * refactor: replace timeout schedule with a seconds count * fix: various bug fixes for calls * fix: remove end call from busy * refactor: use answerCall instead of manual intent building again * build: new version * feat: add silenced notifications for call notification builder. check pre-offer and connecting state for pending connection * build: update build number * fix: text color uses overridden style value * fix: remove wrap content for renderers and look more at recovering from network switches * build: update build number * refactor: remove whitespace * build: update build number * refactor: used shared number for BatchMessageReceiveJob.kt parameter across pollers * fix: glide in update crash * fix: bug fixes for self-send answer / hangup messages * build: update build number * build: update build.gradle number * refactor: compile errors and refactoring to view binding * fix: set the content to binding.root view * build: increase build number * build: update build numbers * feat: adding base for rotation and picking random subset of turn servers * feat: starting the screen rotation processing * feat: setting up rotation for the remote render view * refactor: applying rotation and mirroring based on front / rear cameras that wraps nicely, only scale reworking needed * refactor: calls video stretching but consistent * refactor: state machine and tests for the transition events * feat: new call state processing * refactor: adding reconnecting logic and visuals * feat: state machine reconnect logic wip * feat: add reconnecting and merge fixes * feat: check new session based off current state * feat: reconnection logic works correctly now * refactor: reduce TIMEOUT_SECONDS to 30 from 90 * feat: reset peer connection on DC to prevent ICE messages from old connection or stale state in reconnecting * refactor: add null case * fix: set approved on new outgoing threads, use approved more deeply and invalidate the options menu on recipient modified. Add approvedMe flag toggles for visible message receive * fix: add name update in action bar on modified, change where approvedMe is set * build: increment build number * build: update build number * fix: merge compile errors and increment build number * refactor: remove negotiation based on which party dropped connection * refactor: call reconnection improvement tested cross platform to re-establish * refactor: failed and disconnect events only handled if either the reconnect or the timeout runnables are not set * build: update version number * fix: reduce timeout * fix: fixes the incoming hangup logic for linked devices * refactor: match iOS styling for call activity closer * chore: upgrade build numbers * feat: add in call settings dialog for if calls is disabled in conversation * feat: add a first call missed control message and info popup with link to privacy settings * fix: looking at crash for specific large transaction in NotificationManager * refactor: removing the people in case transaction size reduces to fix notif crash * fix: comment out the entire send multiple to see if it fixes the issue * refactor: revert to including the full notification process in a try/catch to handle weird responses from NotificationManager * fix: add in notification settings prompt for calls and try to fall back to dirty full screen intent / start activity if we're allowed * build: upgrade build number
2022-04-19 06:25:40 +02:00
PeerConnection.IceServer.builder("turn:$sub.getsession.org")
.setUsername("session202111")
.setPassword("053c268164bc7bd7")
.createIceServer()
}
val constraints = MediaConstraints().apply {
optional.add(MediaConstraints.KeyValuePair("DtlsSrtpKeyAgreement", "true"))
}
val configuration = PeerConnection.RTCConfiguration(iceServers).apply {
bundlePolicy = PeerConnection.BundlePolicy.MAXBUNDLE
rtcpMuxPolicy = PeerConnection.RtcpMuxPolicy.REQUIRE
if (relay) {
iceTransportsType = PeerConnection.IceTransportsType.RELAY
}
}
val newPeerConnection = factory.createPeerConnection(configuration, constraints, observer)!!
peerConnection = newPeerConnection
newPeerConnection.setAudioPlayout(true)
newPeerConnection.setAudioRecording(true)
newPeerConnection.addStream(mediaStream)
}
init {
val audioConstraints = MediaConstraints().apply {
optional.add(MediaConstraints.KeyValuePair("DtlsSrtpKeyAgreement", "true"))
}
mediaStream = factory.createLocalMediaStream("ARDAMS")
audioSource = factory.createAudioSource(audioConstraints)
audioTrack = factory.createAudioTrack("ARDAMSa0", audioSource)
audioTrack.setEnabled(true)
mediaStream.addTrack(audioTrack)
val newCamera = Camera(context, this)
camera = newCamera
if (newCamera.capturer != null) {
val newVideoSource = factory.createVideoSource(false)
videoSource = newVideoSource
val newVideoTrack = factory.createVideoTrack("ARDAMSv0", newVideoSource)
videoTrack = newVideoTrack
rotationVideoSink.setObserver(newVideoSource.capturerObserver)
newCamera.capturer.initialize(
SurfaceTextureHelper.create("WebRTC-SurfaceTextureHelper", eglBase.eglBaseContext),
context,
rotationVideoSink
)
rotationVideoSink.mirrored = newCamera.activeDirection == CameraState.Direction.FRONT
rotationVideoSink.setSink(localRenderer)
newVideoTrack.setEnabled(false)
mediaStream.addTrack(newVideoTrack)
} else {
videoSource = null
videoTrack = null
}
initPeerConnection()
}
fun getCameraState(): CameraState {
return CameraState(camera.activeDirection, camera.cameraCount)
}
fun createDataChannel(channelName: String): DataChannel {
val dataChannelConfiguration = DataChannel.Init().apply {
ordered = true
negotiated = true
id = 548
}
return peerConnection!!.createDataChannel(channelName, dataChannelConfiguration)
}
fun addIceCandidate(candidate: IceCandidate) {
// TODO: filter logic based on known servers
peerConnection!!.addIceCandidate(candidate)
}
fun dispose() {
camera.dispose()
videoSource?.dispose()
audioSource.dispose()
peerConnection?.close()
peerConnection?.dispose()
}
fun setNewRemoteDescription(description: SessionDescription) {
val future = SettableFuture<Boolean>()
peerConnection!!.setRemoteDescription(object: SdpObserver {
override fun onCreateSuccess(p0: SessionDescription?) {
throw AssertionError()
}
override fun onCreateFailure(p0: String?) {
throw AssertionError()
}
override fun onSetSuccess() {
future.set(true)
}
override fun onSetFailure(error: String?) {
future.setException(PeerConnectionException(error))
}
}, description)
try {
future.get()
} catch (e: InterruptedException) {
throw AssertionError(e)
} catch (e: ExecutionException) {
throw PeerConnectionException(e)
}
}
fun setRemoteDescription(description: SessionDescription) {
val future = SettableFuture<Boolean>()
peerConnection!!.setRemoteDescription(object: SdpObserver {
override fun onCreateSuccess(p0: SessionDescription?) {
throw AssertionError()
}
override fun onCreateFailure(p0: String?) {
throw AssertionError()
}
override fun onSetSuccess() {
future.set(true)
}
override fun onSetFailure(error: String?) {
future.setException(PeerConnectionException(error))
}
}, description)
try {
future.get()
} catch (e: InterruptedException) {
throw AssertionError(e)
} catch (e: ExecutionException) {
throw PeerConnectionException(e)
}
}
fun createAnswer(mediaConstraints: MediaConstraints) : SessionDescription {
val future = SettableFuture<SessionDescription>()
peerConnection!!.createAnswer(object:SdpObserver {
override fun onCreateSuccess(sdp: SessionDescription?) {
future.set(sdp)
}
override fun onSetSuccess() {
throw AssertionError()
}
override fun onCreateFailure(p0: String?) {
future.setException(PeerConnectionException(p0))
}
override fun onSetFailure(p0: String?) {
throw AssertionError()
}
}, mediaConstraints)
try {
return correctSessionDescription(future.get())
} catch (e: InterruptedException) {
throw AssertionError()
} catch (e: ExecutionException) {
throw PeerConnectionException(e)
}
}
private fun correctSessionDescription(sessionDescription: SessionDescription): SessionDescription {
val updatedSdp = sessionDescription.description.replace("(a=fmtp:111 ((?!cbr=).)*)\r?\n".toRegex(), "$1;cbr=1\r\n")
.replace(".+urn:ietf:params:rtp-hdrext:ssrc-audio-level.*\r?\n".toRegex(), "")
return SessionDescription(sessionDescription.type, updatedSdp)
}
fun createOffer(mediaConstraints: MediaConstraints): SessionDescription {
val future = SettableFuture<SessionDescription>()
peerConnection!!.createOffer(object:SdpObserver {
override fun onCreateSuccess(sdp: SessionDescription?) {
future.set(sdp)
}
override fun onSetSuccess() {
throw AssertionError()
}
override fun onCreateFailure(p0: String?) {
future.setException(PeerConnectionException(p0))
}
override fun onSetFailure(p0: String?) {
throw AssertionError()
}
}, mediaConstraints)
try {
isInitiator = true
return correctSessionDescription(future.get())
} catch (e: InterruptedException) {
throw AssertionError()
} catch (e: ExecutionException) {
throw PeerConnectionException(e)
}
}
fun setLocalDescription(sdp: SessionDescription) {
val future = SettableFuture<Boolean>()
peerConnection!!.setLocalDescription(object: SdpObserver {
override fun onCreateSuccess(p0: SessionDescription?) {
}
override fun onSetSuccess() {
future.set(true)
}
override fun onCreateFailure(p0: String?) {}
override fun onSetFailure(error: String?) {
future.setException(PeerConnectionException(error))
}
}, sdp)
try {
future.get()
} catch(e: InterruptedException) {
throw AssertionError(e)
} catch(e: ExecutionException) {
throw PeerConnectionException(e)
}
}
fun setCommunicationMode() {
peerConnection?.setAudioPlayout(true)
peerConnection?.setAudioRecording(true)
}
fun setAudioEnabled(isEnabled: Boolean) {
audioTrack.setEnabled(isEnabled)
}
fun setDeviceRotation(rotation: Int) {
Log.d("Loki", "rotation: $rotation")
rotationVideoSink.rotation = rotation
}
fun setVideoEnabled(isEnabled: Boolean) {
videoTrack?.let { track ->
track.setEnabled(isEnabled)
camera.enabled = isEnabled
}
}
fun isVideoEnabled() = camera.enabled
fun flipCamera() {
camera.flip()
}
override fun onCameraSwitchCompleted(newCameraState: CameraState) {
// mirror rotation offset
rotationVideoSink.mirrored = newCameraState.activeDirection == CameraState.Direction.FRONT
cameraEventListener.onCameraSwitchCompleted(newCameraState)
}
fun isInitiator(): Boolean = isInitiator
}