Introduction to AV Foundation
•
22 likes•8,160 views
, AV Foundation moves to center stage as the essential media framework on the device, offering support for playing, capturing, and even editing audio and video. Borrowing some of the core ideas from the Mac's QuickTime, while adding many new concepts of its own, AV Foundation offers extraordinary capabilities for application programmers. This talk will offer a high-level overview of what's in AV Foundation, and a taste of what it can do.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Introduction to AV Foundation
Similar to Introduction to AV Foundation (20)
More from Chris Adamson
More from Chris Adamson (16)
Recently uploaded
Recently uploaded (20)
Introduction to AV Foundation
- 1. Introduction to AV Foundation Chris Adamson CocoaHeads Ann Arbor — Aug. 12, 2010
- 2. Media on iPhone / iOS
- 3. iPhone 2 Media Frameworks Core Audio Low-level audio streaming Media Player Full-screen video player Obj-C wrapper for audio file AV Foundation playback (2.2 only)
- 4. iPhone 3 Media Frameworks Core Audio Low-level audio streaming Media Player iPod library search/playback Obj-C wrapper for audio file AV Foundation playback, recording
- 5. iOS 4 Media Frameworks Core Audio Low-level audio streaming Media Player iPod library search/playback Audio/video capture, editing, AV Foundation playback, export… Quartz effects on moving Core Video images Objects for representing media Core Media times, formats, buffers
- 6. AV Foundation in iPhone 2.2
- 7. AV Foundation in iPhone 3.2
- 8. AV Foundation in iOS 4
- 9. Size is relative AV android. QuickTime QT Kit Foundation media for Java Classes 56 40 24 576 Methods 460 280 360 >10,000
- 10. AV Foundation Classes • Capture • Assets and compositions • Playback, editing, and export • Legacy classes
- 11. Media Capture
- 12. Capture: Old and Busted • UIImagePickerController • Takes user out of your UI • Low configurability • No capture-time data access • AVAudioRecorder • Audio only, to file only
- 13. Capture: New Hotness • AV Foundation capture classes • Highly configurable • Live callbacks with capture data • Image/video preview to a CALayer
- 14. Capture Classes Seem Familiar? QT Kit AV Foundation QTCaptureAudioPreviewOutput AVCaptureAudioDataOutput QTCaptureConnection AVCaptureConnection QTCaptureDecompressedAudioOutput AVCaptureDevice QTCaptureDecompressedVideoOutput AVCaptureFileOutput QTCaptureDevice AVCaptureInput QTCaptureDeviceInput AVCaptureMovieFileOutput QTCaptureFileOutput AVCaptureOutput QTCaptureInput AVCaptureSession QTCaptureLayer AVCaptureStillImageOutput QTCaptureMovieFileOutput AVCaptureVideoDataOutput QTCaptureOutput AVCaptureVideoPreviewLayer QTCaptureSession QTCaptureVideoPreviewOutput QTCaptureView
- 15. AVCaptureDevice • Represents an input (camera, microphone) or output (speakers) device • Discover with +[devices], + [devicesWithMediaType], + [defaultDeviceWithMediaType:], … • Flash, white balance, exposure, focus settings for camera devices
- 16. AVCaptureOutput • A destination for captured data • Files: AVCaptureFileOutput, AVCaptureMovieFileOutput • Images: AVCaptureStillImageOutput • Live data: AVCaptureAudioDataOutput, AVCaptureVideoDataOutput
- 17. AVCaptureSession • Coordinates the activity of audio and video capture devices • Allows you to connect/disconnect inputs and outputs • startRunning/stopRunning
- 18. AVCaptureVideoPreview Layer • Subclass of CALayer • +[layerWithSession:]
- 19. Data Output Callbacks • Audio and video data outputs provide -[setSampleBufferDelegate:queue:] • Delegates get -[captureOutput: didOutputSampleBuffer: fromConnection]; • Sample buffer is a CMSampleBufferRef
- 20. Core Media • New in iOS 4 • Core Foundation opaque types for wrapping sample buffers, format descriptions, time structures • Functions convert video samples to CVImageBuffer, audio to Core Audio AudioBufferList
- 21. Core Media Time • CMTime: value, timescale, flags, epoch • Timescale is n-ths of a second • Set timescale to a resolution appropriate to your media (e.g., 44100 for CD audio). QT convention is 600 for video (ask Chris why!) • CMTimeConvertScale()
- 22. WWDC 2010 Session 409 Using the Camera with AV Foundation Overview and best practices Brad Ford iPhone Engineering 2
- 24. “iMovie is built entirely on exactly the same public API in AV Foundation that we’re presenting to you in iPhone 4.”
- 26. “Boom Box” APIs • Simple API for playback, sometimes recording • Little or no support for editing, mixing, metadata, etc. • Example: HTML 5 <audio> tag
- 27. “Streaming” APIs • Use “stream of audio” metaphor • Strong support for mixing, effects, other real-time operations • Example: Core Audio
- 28. “Document” APIs • Use “media document” metaphor • Strong support for editing • Mixing may be a special case of editing • Example: QuickTime and AV Foundation
- 29. Assets and Movies • AVAsset: Collection of tracks representing timed media data • QTMovie: Collection of tracks representing timed media data
- 30. QuickTime movies Movie Track Track Media Media
- 31. AVFoundation assets Movie Track Track
- 32. AVAsset • Superclass of all “movie”-like structures in AVFoundation • Represents traits of all tracks taken together: size, duration • Build your own with AVURLAsset
- 33. AVComposition • Subclass of AVAsset representing a combination of multiple file-based assets • Tracks are AVCompositionTracks • For editing, AVMutableComposition and AVMutableCompositionTracks
- 34. Effects 1 • AVAudioMix, AVMutableAudioMix: set volumes or audio ramps at specific times • AVVideoCompositionInstructions: provide a set of layer-based instructions for performing time- based opacity or affine transform ramps
- 35. Playback • AVPlayer: Playback controller • play, pause, seekToTime, etc. • AVPlayerLayer: CALayer for presenting video from an AVLayer
- 36. Effects 2 • AVSynchronizedLayer: CALayer that synchronizes with a AVPlayerItem’s playback timing • Use for overlays, titles, rendered images, Ken Burns effects, etc.
- 37. Export • AVAssetExportSession • Must be created with a canned preset • -[exportAsynchronouslyWith CompletionHandler:] • Takes a block! • Exporting CA effects is tricky…
- 38. Other stuff • AVAssetStillImageGenerator: used for generating thumbnails • Not suitable for getting individual frames (but no GetMediaSample() equivalent either!) • NSCoder, NSValue additions for wrapping CMTimes, CMTimeRanges
- 39. WWDC 2010 Session 407 Editing Media with AV Foundation Overview and best practices Eric Lee iPhone Engineering 2
- 40. Demo
- 41. What’s Next?
- 42. What’s Next Next? AVAssetExportSession.h: extern NSString *const AVAssetExportPreset1280x720 __OSX_AVAILABLE_STARTING(__MAC_10_7,__IPHONE_4_0); CMTime.h: CM_EXPORT const CFStringRef kCMTimeScaleKey __OSX_AVAILABLE_STARTING(__MAC_10_7,__IPHONE_4_0);
- 43. Contact Info • http://www.subfurther.com/blog • @invalidname • invalidname [at] gmail [dot] com