this code is from an angular app, it's a service that records audio.
it works fine when streaming the record, but most of the time it returns an empty record file when it goes through the timeoutWith branch of the last function, which is meant to be used as a fallback in case streaming failed because of a network connectivity problem.
why do you think this code has an issue with the fallback branch?
import { Injectable } from '@angular/core' import * as RecordRTC from 'recordrtc' import { Observable, Subject, Subscription, of } from 'rxjs' import * as moment from 'moment' import { resample } from 'wave-resampler' import { Socket } from 'ngx-socket-io' import { debounceTime, filter, map, take, timeoutWith } from 'rxjs/operators' import { UtilityService } from './utility.service' import { TranslateService } from '@ngx-translate/core' import { ActiveUserService } from './active-user.service' const alawmulaw = require('alawmulaw') interface RecordedAudioOutput { blob: Blob title: string } @Injectable({ providedIn: 'root', }) export class RecorderService { private stream: MediaStream private recorder private usingMediaRecorder: boolean = true private interval private startTime // private recordedSubject = new Subject<RecordedAudioOutput>() private recordedFileNameUrl = new Subject<any>() private recordingTimeSubject = new Subject<{ formatted: string seconds: number }>() private recordingFailedSubject = new Subject<string>() private scriptProcessor: ScriptProcessorNode private audioWorklet: AudioWorkletNode private contextSampleRate = 48000 readonly BACKEND_SAMPLE_RATE = 24000 private recordId: number = 0 private streamedAudioFileIgnored: boolean = false private urlSub: Subscription = null readonly MAX_RECORD_TIME: number = 5 * 60 //in seconds private audioData: Uint8Array = new Uint8Array( this.MAX_RECORD_TIME * 1.5 * this.BACKEND_SAMPLE_RATE ) private audioDataIndex = 0 private internalBufferLength:number = 1024 * 8 private int16ConvertArray:Int16Array constructor( private socket: Socket, private utilityService: UtilityService, private translate: TranslateService, private activeUser: ActiveUserService ) { try { const context = new AudioContext() const sampleRate = context.sampleRate console.log('sample rate is', sampleRate) this.contextSampleRate = sampleRate } catch (e) { console.log( 'error finding browser sample rate, falling back to default rate' ) } this.int16ConvertArray = new Int16Array(this.internalBufferLength*this.BACKEND_SAMPLE_RATE/this.contextSampleRate) this.socket .fromEvent('disconnect') .pipe(debounceTime(2000)) .subscribe((ev) => { console.log('soc dcccccccccccccccccccc') if (this.recorder && !this.streamedAudioFileIgnored) { this.utilityService.presentToast( this.translate.instant('POST_QUESTION.recordingInterrupted'), [], 'warning' ) this.streamedAudioFileIgnored = true } }) } // getRecordedBlob(): Observable<RecordedAudioOutput> { // return this.recordedSubject.asObservable() // } getFileNameUrl(): Observable<any> { return this.recordedFileNameUrl.asObservable() } getRecordedTime(): Observable<{ formatted: string; seconds: number }> { return this.recordingTimeSubject.asObservable() } sendRecordStream(msg, v5: boolean = false) { if (v5) { this.socket.emit('messagev5', msg) } else { this.socket.emit('messagev4', msg) } } initializeStream() { this.audioData.fill(0) this.audioDataIndex = 0 this.streamedAudioFileIgnored = false this.recordId = parseInt(''+(new Date().getTime()) + this.utilityService.randomId(0, 10000)) if (this.activeUser.isActive) { this.socket.emit('startGoogleCloudStreamv5', { recordId: this.recordId, userId: this.activeUser.user._id, }) } else { this.socket.emit('startGoogleCloudStreamv5', { recordId: this.recordId }) } console.log('streamingStarted') } endStream() { this.subscribeForStreamedAudioFileUrl(this.audioData.subarray(0, this.audioDataIndex)) this.socket.emit( 'endGoogleCloudStreamv5', this.usingMediaRecorder ? 'MR0.7.3' : '' ) console.log('end emitt') if (this.stream) { // stop the browser microphone const tracks = this.stream.getTracks() tracks.forEach((track) => { track.stop() }) this.stream = null } if (this.scriptProcessor) { // Stop listening the stream from the michrophone this.scriptProcessor.removeEventListener( 'audioprocess', this.streamAudioData ) } if (this.audioWorklet) { this.audioWorklet.port.onmessage = undefined } } streamAudioData = (e?: AudioProcessingEvent, data?: Uint8Array) => { // HERE GOES THE CODE TO SEND THE CHUNKED DATA FROM STREAM // if (!resamp && e.i8) { // const mergedArray = new Uint8Array(this.audioData.length + e.i8.length) // mergedArray.set(this.audioData) // mergedArray.set(e.i8, this.audioData.length) // this.audioData = mergedArray // return this.sendRecordStream(e.i8.buffer, true) // } // var offlineCtx = new OfflineAudioContext( // 1, // e.inputBuffer.duration * this.BACKEND_SAMPLE_RATE, // this.BACKEND_SAMPLE_RATE // ) // var offlineSource = offlineCtx.createBufferSource() // offlineSource.buffer = e.inputBuffer // offlineSource.connect(offlineCtx.destination) // offlineSource.start() // offlineCtx // .startRendering() // .then((resampled) => { // // `resampled` contains an AudioBuffer resampled at 16000Hz. // // use resampled.getChannelData(x) to get an Float32Array for channel x. // const aa = resampled.getChannelData(0) // const a16 = this.convertFloat32ToInt16(aa) // const a8 = alawmulaw.mulaw.encode(a16) // const mergedArray = new Uint8Array(this.audioData.length + a8.length) // mergedArray.set(this.audioData) // mergedArray.set(a8, this.audioData.length) // this.audioData = mergedArray // this.sendRecordStream(a8.buffer) // }) // .catch((e) => { // console.log('error resampling', e) // let floatSamples = e.inputBuffer.getChannelData(0) // const aa = this.downsample( // floatSamples, // this.contextSampleRate, // this.BACKEND_SAMPLE_RATE // ) // const a16 = this.convertFloat32ToInt16(aa) // const a8 = alawmulaw.mulaw.encode(a16) // this.audioData = new Uint8Array([...this.audioData, ...a8]) // const mergedArray = new Uint8Array(this.audioData.length + a8.length) // mergedArray.set(this.audioData) // mergedArray.set(a8, this.audioData.length) // this.audioData = mergedArray // this.sendRecordStream(a8.buffer) // }) if (e?.inputBuffer) { const aa = resample( e.inputBuffer.getChannelData(0), this.contextSampleRate, this.BACKEND_SAMPLE_RATE, { method: 'linear' } ) const a16 = this.convertFloat32ToInt16(aa) const a8 = alawmulaw.mulaw.encode(a16) this.audioData.set(a8, this.audioDataIndex) this.audioDataIndex += a8.length this.sendRecordStream(a8.buffer) } else if (data.length) { const aa = resample( data, this.contextSampleRate, this.BACKEND_SAMPLE_RATE, { method: 'linear' } ) const a16 = this.convertFloat32ToInt16(aa) const a8 = alawmulaw.mulaw.encode(a16) this.audioData.set(a8, this.audioDataIndex) this.audioDataIndex += a8.length this.sendRecordStream(a8.buffer, true) } } downsample(buffer, fromSampleRate, toSampleRate) { // buffer is a Float32Array var sampleRateRatio = Math.round(fromSampleRate / toSampleRate) var newLength = Math.round(buffer.length / sampleRateRatio) var result = new Float32Array(newLength) var offsetResult = 0 var offsetBuffer = 0 while (offsetResult < result.length) { var nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio) var accum = 0, count = 0 for ( var i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i++ ) { accum += buffer[i] count++ } result[offsetResult] = accum / count offsetResult++ offsetBuffer = nextOffsetBuffer } return result } convertFloat32ToInt16(buffer) { let l = buffer.length while (l--) { this.int16ConvertArray[l] = buffer[l] * 0x7fff } return this.int16ConvertArray } getRecordFinalText() { return this.socket .fromEvent('speechData') .pipe(filter((data: any) => data.isFinal)) .pipe(map((data: any) => data.text)) } getRecordPreviewText() { return this.socket .fromEvent('speechData') .pipe(filter((data: any) => !data.isFinal)) .pipe(map((data: any) => data.text)) } recordingFailed(): Observable<string> { return this.recordingFailedSubject.asObservable() } async initializeAudioContext() { if (this.utilityService.otherRecordInstanceActive) { this.utilityService.presentToast( this.translate.instant('POST_QUESTION.anotherRecordInProgress'), [], 'warning' ) return false } this.utilityService.otherRecordInstanceActive = true this.initializeStream() window.AudioContext = window.AudioContext return this.startRecording() } async startRecordingStream(s: MediaStream) { let AudioContext = window.AudioContext || window['webkitAudioContext'] let audioContext: AudioContext = new AudioContext() if (!audioContext) { return } // AudioNode used to control the overall gain (or volume) of the audio graph //const inputPoint = audioContext.createGain(); //const microphone = audioContext.createMediaStreamSource(s); //const analyser = audioContext.createAnalyser(); try { //throw new Error('audio worklet disabled') await audioContext.audioWorklet.addModule( './assets/worklet/worklet-processor.js' ) this.audioWorklet = new AudioWorkletNode( audioContext, 'worklet-processor', { channelCount: 1, processorOptions: { bufferSize: this.internalBufferLength, inputSampleRate: this.contextSampleRate, outputSampleRate: this.BACKEND_SAMPLE_RATE, }, } ) let input = audioContext.createMediaStreamSource(s) input.connect(this.audioWorklet) this.audioWorklet.port.onmessage = (m) => { this.streamAudioData(null, m.data) } console.log('worklet') } catch (e) { console.log('where my audioWorker', e) this.scriptProcessor = audioContext.createScriptProcessor(this.internalBufferLength, 1, 1) this.scriptProcessor.connect(audioContext.destination) console.log('conx', audioContext.sampleRate) let input = audioContext.createMediaStreamSource(s) input.connect(this.scriptProcessor) // microphone.connect(inputPoint); // inputPoint.connect(analyser); // inputPoint.connect(this.scriptProcessor); // This is for registering to the “data” event of audio stream, without overwriting the default scriptProcessor.onAudioProcess function if there is one. this.scriptProcessor.addEventListener( 'audioprocess', this.streamAudioData ) } } async startRecording() { if (this.recorder) { // It means recording is already started or it is already recording something return false } this.recordingTimeSubject.next({ formatted: '00:00', seconds: 0 }) try { const s = await navigator.mediaDevices.getUserMedia({ // audio: true, audio: { echoCancellation: true, sampleRate: this.contextSampleRate, }, }) this.stream = s console.log('stram', s) this.record() this.startRecordingStream(this.stream) return true } catch (error) { console.log('error recording', error) this.recordingFailedSubject.next('permission-error') this.utilityService.otherRecordInstanceActive = false return false } } private record() { try { if (MediaRecorder.isTypeSupported('audio/webm;codecs="opus"')) { this.recorder = new MediaRecorder(this.stream, { mimeType: 'audio/webm;codecs="opus"', }) this.usingMediaRecorder = true } else if (MediaRecorder.isTypeSupported('audio/mp4')) { this.recorder = new MediaRecorder(this.stream, { mimeType: 'audio/mp4', }) this.usingMediaRecorder = true } else { throw new Error('no mediarecord support type found') } // this.recorder.addEventListener('dataavailable', (e)=>{ // let blob=e.data // if (this.startTime) { // const mp3Name = encodeURIComponent( // "audio_" + new Date().getTime() + ".mp3" // ); // this.stopMedia(); // this.recordedSubject.next({ blob, title: mp3Name }); // } // }); this.recorder.start() } catch (e) { console.log('error using mediarecorder', e) this.usingMediaRecorder = false this.recorder = new RecordRTC.StereoAudioRecorder(this.stream, { type: 'audio', mimeType: 'audio/wav', numberOfAudioChannels: 1, // bitsPerSecond: 128000, // audioBitsPerSecond: 128000, // sampleRate: 96000, desiredSampRate: this.contextSampleRate, }) this.recorder.record() } this.startTime = moment() this.interval = setInterval(() => { const currentTime = moment() const diffTime = moment.duration(currentTime.diff(this.startTime)) const time = this.toString(diffTime.minutes()) + ':' + this.toString(diffTime.seconds()) this.recordingTimeSubject.next({ formatted: time, seconds: diffTime.asSeconds(), }) }, 100) } private toString(value) { let val = value if (!value) { val = '00' } if (value < 10) { val = '0' + value } return val } stopRecording(kill: boolean = false) { console.log('record is', this.recorder, this.usingMediaRecorder) if (this.recorder && !this.usingMediaRecorder) { this.recorder.stop( (blob) => { if (this.startTime) { const mp3Name = encodeURIComponent( 'audio_' + new Date().getTime() + '.mp3' ) this.stopMedia() // this.recordedSubject.next({ blob, title: mp3Name }) } }, () => { this.stopMedia() this.recordingFailedSubject.next() } ) this.endStream() } else if (this.recorder) { this.recorder.stop() this.stopMedia() this.endStream() } if (kill) { console.log('killing record id') this.recordId = 0 this.utilityService.otherRecordInstanceActive = false this.audioData.fill(0) this.audioDataIndex = 0 } } private stopMedia() { if (this.recorder) { this.recorder = null clearInterval(this.interval) this.startTime = null if (this.stream) { this.stream.getTracks().forEach((track) => track.stop()) this.stream = null } } } subscribeForStreamedAudioFileUrl(buffer:Uint8Array) { this.urlSub = this.socket .fromEvent('recordUrl') .pipe( filter((data: any) => { console.log('data', data, this.recordId) return !this.streamedAudioFileIgnored&&data && data.id == this.recordId }) ) .pipe( timeoutWith( 10000, of({ timeout: true, buffer, }) ) ) .pipe(take(1)) .subscribe((data: any) => { console.log('got url', data) // if(data.isFinal){ // this.stopRecording() // } this.recordedFileNameUrl.next(data) this.utilityService.otherRecordInstanceActive = false }) return } }
هل تأكدت من تأثير باقي الملفات في المشروع على قيم المتغيرات في الكود؟ . . . . . .. . . . . . . .. . . . .. . .. . . . .. . . . . . .. . .. . . .. . . . . . . .. . .. . . .. . . . .
تم النشر الأربعاء، ٢١ أغسطس ٢٠٢٤
It looks like you are using the timeoutWith operator as a fallback to handle cases when streaming fails, typically due to connectivity issues. If you're receiving an empty record file, here are a few things to check: Buffer Handling : Make sure the buffer you pass in timeoutWith actually contains data. If the buffer is initialized or cleared just before the timeoutWith fallback is triggered, it might be empty. Timeout Duration : Check if the timeout value (10 seconds in your code) is too short for your use case. If network delays are common, consider increasing this timeout duration. Buffer Data : When the buffer is passed due to a timeout, ensure you’re recording and saving the buffer data correctly. Double-check how buffer is constructed and used. Socket Event Handling : Make sure the 'recordUrl' event emits the expected data before the timeout triggers. If there's a delay in receiving this event, verify the socket communication process and improve its reliability. Check for Errors : Log errors or exceptions that might occur just before the fallback to identify potential issues that might cause premature triggering of the timeoutWith . Audio Data Index : Verify the audioDataIndex value to ensure it's correctly managing the audio data size. Start by adding more logging statements at various points in your subscribeForStreamedAudioFileUrl method to gain insights into the data flow and pinpoint where the issue starts. (يرجى ملاحظة أن هذه إجابة آلية بواسطة الذكاء الصناعي ولا تغني عن زيارة الطبيب في الأمور الصحية)
تم النشر الأحد، ١١ أغسطس ٢٠٢٤
تم النشر الأحد، ٧ يوليو ٢٠٢٤
www.google.com
تم النشر الاثنين، ٢٧ نوفمبر ٢٠٢٣
لعرض السؤال في فدني اضغط هنا